In this article, we continue to share our experiences in the field of Large Language Models (LLMs), focusing in particular on how Visual Language Models (VLMs) are revolutionizing document pre-processing in RAG systems.<\/p>\n\n\n\n
VLMs are the meeting point between vision and language, between visual content and text.<\/p>\n\n\n\n
They represent the next step in the evolution of AI for document understanding and are an essential component of next-generation RAG systems.<\/p>\n\n\n\n
With VLMs, document pre-processing in RAG systems is no longer a simple technical step, but becomes a phase of intelligent data interpretation<\/strong>.<\/p>\n\n\n\n In recent years, Large Language Models (LLMs)<\/strong> \u2014 such as GPT-4o, Claude 3.5, or Llama 3.1 \u2014 have transformed the way companies manage and interpret textual data.<\/p>\n\n\n\n From the automatic generation of intelligent responses in support systems to the semantic analysis of business reports and logs, LLMs have become essential tools for improving efficiency and decision quality.<\/p>\n\n\n\n However, most business documents are not made up of text alone.<\/p>\n\n\n\n Technical projects, reports, engineering drawings, or functional specifications contain images, diagrams, charts, and tables<\/strong> that convey crucial information but are difficult to interpret with purely linguistic models.<\/p>\n\n\n\n This is where a new generation of models comes into play: Visual Language Models (VLMs)<\/strong>.<\/p>\n\n\n\n VLMs<\/strong> combine the visual capabilities of computer vision models with the semantic understanding typical of LLMs. In other words, a VLM is able to \u201csee\u201d <\/strong>and \u201cread\u201d<\/strong> simultaneously<\/strong>, interpreting images, text, and graphic structure as a single coherent language.<\/p>\n\n\n\n In a Retrieval-Augmented Generation (RAG)<\/strong> architecture, the use of VLMs represents a turning point in the data preparation phase, which includes document pre-processing, chunking, data enrichment, embedding, and indexing in a vector store. More specifically, for the pre-processing phase, a VLM can analyze documents at a visual and semantic level, returning structured representations enriched with metadata.<\/p>\n\n\n\n In practice, while traditional OCR extracts only text, a VLM is capable of understanding diagrams, legends, tables, and visual relationships<\/strong>, providing a more comprehensive database for retrieval and the generation of high-quality responses.<\/p>\n\n\n\n To understand the potential of VLMs in document understanding<\/em>, let’s imagine a pre-processing pipeline designed to process complex documents\u2014for example, PDFs containing technical diagrams, tables, and illustrations. These are the main stages of the pipeline:<\/p>\n\n\n\n Thanks to public benchmarks from platforms such as Hugging Face, it is possible to compare the best-performing VLMs on the market today. Below is an updated summary:<\/p>\n\n\n\n Sources: OpenCompass public benchmarks \u2013 Hugging Face<\/em><\/p>\n\n\n\n In conclusion, Visual Language Models<\/strong> represent the natural evolution of LLMs, paving the way for systems capable of understanding multimodal documents<\/strong> in a truly intelligent way.<\/p>\n\n\n\n Within RAG pipelines, their contribution is decisive:<\/p>\n\n\n\n For organizations that handle large quantities of technical documents, reports, or project diagrams, VLMs offer a concrete advantage: transforming every document \u2014 even the most complex ones \u2014 into knowledge that can be used by artificial intelligence<\/strong>.<\/p>\n\n\n\n In conclusion, Humanativa continues to invest in RAG\/LLM, and Humanativa’s Competence Center will include Preprocessing Modules using VLM in the next version of the core system of LLM solutions, based on feedback from the first advanced RAG projects.<\/p>\n","protected":false},"excerpt":{"rendered":" The new frontier of AI for understanding complex documents<\/p>\n","protected":false},"author":7,"featured_media":417,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[56,2,43],"tags":[68,11,110,112,108,103,102,116,111,105,104,115,114],"class_list":["post-416","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articoli","category-approfondimenti","category-slideshow","tag-ai","tag-artificial-intelligence","tag-generative-ai","tag-generative-artificial-intelligence","tag-gpt","tag-large-language-models","tag-llm","tag-pre-trained-generative-transformer","tag-prompt-engineering","tag-rag","tag-retrieval-augmented-generation","tag-visual-language-models","tag-vlm"],"yoast_head":"\nFrom the era of LLMs to multimodal understanding<\/h3>\n\n\n\n
What are VLMs and why are they a fundamental component of RAG systems<\/h3>\n\n\n\n
An example of a VLM pipeline for pre-processing in RAG systems<\/h3>\n\n\n\n
\n
<\/strong>Each page of the document is converted into an image and sent to an advanced VLM model (such as Gemini 2.5 Pro, GPT-4o, or an open-source model such as LLaVA-NEXT). The model simultaneously interprets text, layout, and visual components, returning a semantic understanding at the \u201cpage\u201d level<\/strong>.
<\/li>\n\n\n\n
<\/strong>The result of the analysis is translated into structured data<\/strong>, such as a JSON file describing text, coordinates, types of visual elements, and spatial relationships. This step provides a unified view of the document, which is useful for subsequent segmentation or intelligent chunking<\/em> operations.
<\/li>\n\n\n\n
<\/strong>In the absence of labeled datasets, the pipeline can generate synthetic data<\/strong> from public documents or controlled internal repositories. This data is used to optimize the model’s behavior through fine-tuning<\/em> or prompt optimization<\/em>, improving its accuracy in recognizing specific patterns.
<\/li>\n\n\n\n
<\/strong>The results are then enriched with metadata and sent to the embedding phase to be indexed in a vector database. In this way, the RAG system can subsequently retrieve both textual and visual information, ensuring more relevant responses based on multimodal<\/em> understanding.
<\/li>\n\n\n\n
<\/strong>The pipeline can be executed asynchronously: for example, a service monitors an S3 bucket or a shared directory, automatically processes each new document, and updates the knowledge base index in real time.<\/li>\n<\/ol>\n\n\n\nThe main Visual Language Models available today<\/h3>\n\n\n\n
Model<\/strong><\/span><\/td> Type<\/strong><\/span><\/td> State of the art<\/strong><\/span><\/td> Production Ready<\/strong><\/span><\/td> Notes<\/strong><\/span><\/td><\/tr><\/thead> Gemini 2.5 Pro<\/strong><\/td> Proprietary (Google DeepMind)<\/td> High<\/td> YES<\/td> Full multimodal (text, images, video). Excellent for technical documents.<\/td><\/tr> GPT-4o<\/strong><\/td> Proprietary (OpenAI)<\/td> High<\/td> YES<\/td> High multimodal performance; already used in production environments.<\/td><\/tr> Claude 3.5 Sonnet<\/strong><\/td> Proprietary (Anthropic)<\/td> High<\/td> YES<\/td> Strong in diagrams and complex visual comprehension.<\/td><\/tr> LLaVA-NEXT<\/strong><\/td> Open Source<\/td> Medium-high<\/td> NO<\/td> Good balance between performance and openness; still evolving.<\/td><\/tr> Qwen-VL-Max<\/strong><\/td> Open Source (Alibaba)<\/td> Medium-high<\/td> NO<\/td> Excellent balance between visual accuracy and speed.<\/td><\/tr> InternVL 2.0<\/strong><\/td> Open Source<\/td> Medium<\/td> NO<\/td> Interesting for PDFs and complex diagrams; experimental phase.<\/td><\/tr> Kosmos-2<\/strong><\/td> Open Source (Microsoft)<\/td> Low<\/td> NO<\/td> Solid multimodal OCR, but less effective in deep semantics.<\/td><\/tr> Fuyu 8B<\/strong><\/td> Open Source (Adept AI)<\/td> Low<\/td> NO<\/td> Excellent speed, ideal for prototyping and testing.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n VLM as the key to document understanding in RAG systems<\/h3>\n\n\n\n
\n