I've been working with RAG for several years, building solutions for different business use cases. With better LLMs and vector search, retrieval keeps improving. But one challenge remains: the quality of the source data.

I've struggled with converting complex PDFs, Word documents, Excel files, and other formats into clean, structured content for embeddings.

There are plenty of tools available, but advanced document processing can get expensive:

• Amazon Textract — Analyze Document (including tables): $0.015/page
• LlamaParse — Agentic parsing for complex layouts: $0.0125/page
• Amazon Bedrock Data Automation — Standard output: $0.010/page

Free libraries like Docling and PyMuPDF are useful, but handling complex layouts, tables, charts, and mixed content can still be challenging.

That's why I built Docparser.net, a document parser designed specifically for RAG pipelines.

• Complex documents: handles text layouts, tables, charts, and mixed layouts.
• Affordable: just $0.007/page.
• Large files: supports files up to 1.5 GB.
• Fast: processes 1,000 pages in under a minute.
• Easy integration: REST API for your RAG pipeline.

I built this to solve a problem I've faced myself, and I'd love to hear feedback from the community.

What tools are you using to prepare documents for RAG?

I've been building RAG over a few thousand YouTube lectures and talks, and for the first month the answers were bad in a way I couldn't explain. Retrieval looked fine on paper, similarity scores were high, but the model kept answering half a question or mixing two topics.

The problem was the chunking. I was doing what every tutorial does, 512 tokens with some overlap. That works on docs because docs have sentences, paragraphs and headings. Auto generated captions have none of that. No punctuation, no paragraph breaks, just a stream of 2-3 second fragments.
So a token window cuts wherever it wants. The speaker asks a question at the end of one chunk and answers it in the next. Retrieval grabs the answer without the question, or the question without the answer. Topic changes land in the middle of a chunk, so one embedding is half about attention heads and half about the coffee break.

What fixed most of it was chunking by time instead of tokens. I keep the segment timestamps from the transcript, group them into windows of roughly 60-90 seconds, and snap the boundaries to pauses, any gap between segments longer than about 1.5 seconds. Speakers pause when they switch thoughts, so the pauses are the closest thing to paragraph breaks you get.

Two smaller things helped too. I prepend the video title and the chapter name (if the creator added chapters) to every chunk before embedding. And I store the start time in metadata, so the answer can link straight to the moment in the video, which made people trust the answers way more.
On my eval set of about 200 questions, top-5 hit rate went from around 40% to a bit under 70%. Nothing else in the pipeline changed.

For the transcripts themselves I pull them with segment-level timestamps from transcriptapi.com (disclosure, I'm connected to the team). Whatever you use, make sure you keep the timestamps, a lot of tools give you one flat text blob and throw them away.
If you're getting better results with semantic chunking on raw captions, I'd like to see the setup, because it fell apart for me

I'm heavily working on RAG and Vector database, sometimes I want to improve the performance by trials and errors. For example, I can adjust top-k or rerank parameters to reduce latency or CPU, but the overall recall@k downgrades. I need to strike a balance between the metrics to meet the goal. Or sometimes updating one or two parameters may cause breaking SLOs, etc.
Does anyone face the same issues?

Write-up on using pre-existing structured data from orgs to build knowledge graphs for quicker agent retrieval. Basically, scale-up graph-RAG, but make it cheaper (minimise model-mediated connections).

Has the standard issues (can go stale, can be expensive) but it still seems interesting IMO.

https://lexifina.com/blog/building-full-firm-knowledge-graphs-to-dramatically-speed-up-agents

We benchmarked 11 document parsers, including two frontier models, on heavily redlined contracts, where a deletion is just a line through the text.

1,500 questions over 94 contracts. Each parser turns the PDF into markdown, the same agent answers from it, and an LLM judge (gpt-6-sol) grades the answers. Every score is 9 runs (3 parses × 3 asks).

Reducto and LlamaParse agentic plus tie on accuracy. One costs 5.6x more

    Parser (text-layer PDFs)    acc%    $ / 1k correct
    --------------------------------------------------
    Extend                      93.1       $40.91
    Reducto (r-1)               84.8       $17.97
    LlamaParse agentic plus     84.7      $101.17
    LlamaParse agentic          83.8       $22.71
    Datalab track changes       80.7       $18.89
    Datalab convert             63.9       $23.86
    Pulse                       48.4       $47.17
    Mistral OCR                 36.9       $16.54
    Unstructured                27.0       $84.67

For scale: dropping every strike scores 25.2%, a perfect parse 94.6%.

On scanned PDFs, GPT-6 Astra leads at 93.4%, but scores 61.6% on the same contracts with a text layer. Extend is the top dedicated parser on scans at 86.7%.

Reducto, LlamaIndex and Extend are partners. They don't see or influence task design, data, labelling or the held-out set. Answers are LLM-judged; the judge prompt is published.

Disclosure: I run Openbenchmarks. We run independent benchmarks and publish the raw artifacts and results for agents. No vendor paid for placement or inclusion.

Started as plain rag over pdfs and internal docs. Then people want it to act: pull context from jira/slack over mcp, draft a reply or email it

Retrieval turns out to be the easg part. Trust is the hard part - one stale chunk and it confidently emails the wrong policy

Right now genie one handles the docs and mcp tools, and drafts land in a lakebase table as pending until someone approves them. It works but it’s clunky and the approval queue piles up fast

Anyone found a middle ground? Auto send the low risk stiff and gate the rest. How do you define low risk?

I've spent the past three and a half years working on RAG systems and LLM training. Lately, I've been thinking about what an agent saves after using retrieved knowledge. An agent might finish a task and save its conclusion, but a later session also needs the source passages and assumptions behind that conclusion.

We're building Agentic GraphRAG (https://github.com/ontogr/agentic-graphrag), an open source project that connects document knowledge, task activity, and decisions in Neo4j. Agents can retrieve previous decisions along with the evidence they used.

For retrieval, we preserve the source text and let agents read around a search result:

1. Document structure stays in the graph. We use Docling for ingestion and Chonkie for chunking, preserving sections, tables, and passage hierarchy. Each passage retains its source version and location, including page positions for PDFs.
2. Passage search works independently of entity extraction. We combine Neo4j's native full-text and vector search rankings with reciprocal rank fusion. A fact doesn't have to appear in the extracted graph for an agent to find the passage containing it.
3. Agents can navigate beyond the initial hit. They can read surrounding sections, neighboring passages, and table rows with their headers. They can also follow entity relationships back to supporting text where evidence is recorded.

The graph is built around a schema you define for your domain. It guides entity and relationship extraction, validation, and entity search. Extraction can run locally with GLiNER, with BAML handling structured LLM extraction for selected chunks.

Agent memory adds two layers alongside that knowledge. Working memory records tasks, progress, messages, and tool results. Decision memory stores conclusions, assumptions, and open questions, linked to passages, task events, or earlier decisions. Our built-in DeepAgents/LangGraph agent records task activity so later sessions can recover previous progress.

For example, an agent reviewing a contract can save a recommendation with the contract clause and policy passage it used. When the policy changes, the recommendation and its explicit dependent decisions are flagged for review. A reviewer can reaffirm or replace the recommendation, while its history and the originally cited text remain available.

The built-in agent also checks that it read the cited passages during the run, that they belong to the allowed documents, and that they match the cited versions. The checks don't establish whether the passages support the conclusion.

We're still working on domain-specific benchmarks and don't yet have results to report. You can use the built-in agent or connect another agent through MCP.

TL;DR

We're connecting RAG evidence to persistent task and decision memory in one graph. Agents can search original passages, navigate document structure, and trace saved decisions back to the source versions they used. Source updates flag affected decisions for review.

What's your current stack for RAG and agent memory, and how is it working for your domain?

Been building a RAG system for document QA and kept running into the same problem-the model gives an answer with a citation,the citation exists,but the source text doesn't actually support what it said,turns out this is well documented: up to 57%of citations are post-hoc rationalizations,the model generates the answer first,then goes back and finds something that looks related

so I built a verification layer that checks three things before an answer goes out:

Is the citation real and verbatim?(string matcch against source)

Does the cited text actually support the claim?(NLI/LLM check)

Is the chunk itself self-contained,or did chunking break the context?

Ran it on 20 real docs(~1400 chunks) - contracts,GB/Tstandards,RFCs,papers,government notices,caught real stuff:622 numbered clauses getting eaten by the parser,11 references dropped form a paper,8 data fragments in a government notice that would've been cited as evidence

The tricky part isn't the happy path,It's when the LLM judgment flips between runs on the same input,or when a chunk is 90% correnct but missing the one condition that makes it wrong,Right now I flag those as "uncertain"instead of pretending they passed

Anyone else dealing with this?Curious how you handle citation faithfuIness in production,manual review,automated checks,or just trustion the model?

Hi everyone,

I’m a beginner learning RAG and AI agents. I’m currently doing a RAG course on DeepLearning.AI, and so far I’ve learned how to create embeddings with sentence-transformers (model.encode()), compare them with cosine similarity, and I understand the basic idea of retrieval.

I’m running everything locally on Windows with VS Code and Python, using small models like all-MiniLM-L6-v2 because my storage and hardware are limited.

I’d love some guidance on the following:

1. What is the right order to learn things? After embeddings and similarity search, what should come next (chunking, vector databases like FAISS or Chroma, prompt building, evaluation)?
2. How do AI agents differ from a RAG system, and at what point should I start learning agents (tool use, planning, memory, frameworks like LangChain or LangGraph)?
3. Which beginner projects would you recommend building that are good for a portfolio? I was thinking of a chatbot over my own PDFs or notes.
4. Which resources really helped you (courses, docs, YouTube channels, GitHub repos)?
5. Common mistakes to avoid with chunking, retrieval quality, and hallucinations?
6. Can I realistically learn this well using only free or low-cost resources and a normal laptop, or do I need a GPU or paid APIs?

Any step-by-step roadmap or advice from your own experience would be really appreciated. Thanks!

Disclosure: I'm building Jylus, an evidence/context API. This is a product demo using two synthetic records.

On Wednesday, a sensor is recorded as Normal. On Friday, a correction says its Wednesday state should have been Warning.

Both records describe the same effective time:

Now ask for Wednesday's status with two different evidence cutoffs:

• Evidence known by Wednesday: Normal.
• Evidence known by Friday: Warning, with the Normal → Warning correction preserved.

I ran both settings in our public trial. The Friday view retained both source proofs. These are compiled evidence results, not generated model answers or a benchmark.

The distinction is the state's effective time versus when the evidence became available. A historical review needs to say which question it is answering: what we knew then, or what later evidence says about then.

We built Jylus to prepare compact Context Packs from changing records, state history and relationships, with source proof IDs and explicit gaps/conflicts for your existing model. The model still writes the final answer.

Run the two-record test — no account required

Select Late correction, keep State effective as of at 2026-09-23, and compare Evidence known as of 2026-09-23 versus 2026-09-25. Inspect the evidence and source proofs in each result.

API documentation

What late-arriving correction would you add to test this in your domain?

Genuine question about chunking strategy.

Fixed-size chunking with overlap seems to be the default everyone starts with and the thing most production systems eventually move away from, toward section-aware splitting, parent-child retrieval, or semantic boundaries.

For those of you running RAG in production: did you ever make the switch, or is the simple default still doing the job? If you switched, what was the failure that pushed you off it: answers split across boundaries, tables getting shredded, something else? And for the people who stayed with fixed-size: was that a deliberate decision after testing, or has nobody found a reason to change it yet?

Curious whether the simple default is secretly fine or just nobody's priority to fix.

Today's Topic — Conformer Architecture

Today, I'll try to explain the basic intuition behind the Conformer architecture.Conformer is an architecture widely used for Automatic Speech Recognition (ASR). It combines Convolutional Neural Networks (CNNs) and Transformer-based self-attention to capture both local and global patterns in speech.First, the raw audio is converted into a Mel spectrogram, which is a time-frequency representation of the audio signal. It shows how the energy of different frequency bands changes over time. Next, the spectrogram is converted into numerical feature vectors and fed into the Conformer encoder. Its convolution module captures local acoustic patterns, such as changes between speech sounds, while self-attention captures relationships between different time steps, including long-range dependencies. By combining these two mechanisms, Conformer learns richer representations of speech, which are then used to recognize the spoken words.

Open to criticism

Can you compare "Langchain + python" vs "langraph", pros and cons of them, when one can have advantage over other especially on hybrid search scenarios?

I’m building a college project: a retrieval system that runs locally on modest hardware, designed to work as an assistant for a small business (think a handful of staff, not an enterprise).

The problem is data. I’ve searched for a while and can’t find a dataset that realistically represents a small business’s documents and is large enough to properly test retrieval. Most public datasets are either enterprise-scale, general Q&A, or too small to be useful.

What I’m looking for:

• Documents a small business would actually have: invoices, receipts, supplier emails, product manuals, policies, FAQs, price lists, etc.
• Enough volume to test retrieval meaningfully
• Preferably openly licensed

Questions:

1. Does anyone know of a dataset that fits, or something close enough to adapt?
2. If not, is generating synthetic documents a reasonable approach for a project like this, and how would you keep it realistic?
3. If you think the small-business angle is a dead end, what other use cases suit a local RAG system on modest hardware?

Any pointers, even partial ones, would be really appreciated. Thanks!

I've been working with the team at Dataville on a problem that doesn't seem to get as much attention as models, prompts, and agent frameworks: getting reliable external data into AI applications.

If you're building an agent that needs SEC filings, Census data, Wikipedia, or academic research, you quickly run into different APIs, schemas, and update schedules.

We're developing Dataville, a common REST API and MCP interface for accessing external datasets, to reduce the need for developers to build and maintain separate integrations for every source.

One thing I've heard from other developers is that the biggest challenge isn't necessarily making the initial API connection. It's normalizing the results and keeping everything current as sources change.

I'm curious how others here are approaching this:

• Are you building custom API tools, using MCP servers, or relying primarily on RAG?
• Is data normalization or freshness the bigger problem in your applications?
• Are there external datasets you wish were easier to integrate?

We're early and interested in learning from people building real applications.

Disclosure: I'm helping Dataville with marketing and developer outreach.

For anyone running an AI assistant over company docs, how do you know it’s stopped using old information after something changes?

Say a policy gets updated, but the old version is still sitting in another document or a previous conversation. Have you run into cases where the assistant keeps giving the old answer, even though it has access to the new one?

I’m curious how people handle this in practice. Do you have tests for it, manually check a few questions after updates, or mostly find out when someone flags a wrong answer? What happens when two documents disagree and it isn’t obvious which one should win?

I’m working on an early-stage project around checking these kinds of answers, so I’d like to hear how much of a problem this actually is. If your team has it mostly figured out, I’d be just as interested in hearing what worked.

Note: At the risk of incurring the wrath of the community here, I do believe this is a discussion that needs to be had and whether the idea is liked or disliked, the approach I feel remains valid for certain scenarios and should be explored by all RAG enthusiasts: Judge me on my results!

By now, we've probably seen or heard of using Jev (TypesafeAI's decision model) as a reranker for traditional document RAG and the short of it is that it actually works really well. So I took the idea further - what if we're not just ranking a subset of page results to a user's question but rather what if we could rank every page in the document from the get go?

Ridiculous? Yes. For this to be a possible, you'd need to be able to see into the future and know every question your users will ever ask! But what if you're in a scenario where you get the user's questions upfront before any document is parsed? This is the actually what happens in typical document RAG and this is the caveat I'm referring to.

Some context: I run an "AI spreadsheet" app called Ragextract which has documents for rows and extraction questions as columns (think airtable, tabular review etc). Users define the columns first (eg. "who is the claimant?") and load in the documents (eg. third party claims) after. The flow is I use multimodal vector embeddings to index every page as an image - to support scanned pdfs - and performing searches over it when it came time to extract.

Introducing JAG (Jev-augmented Generation?)

So my use-case is kinda perfect for this little thought experiment.

What I basically did was run every page through JEV instead and have each column by the questions that Jev decides on: "Is the answer (or near answer) on this page?". The results on accuracy, speed and costs were overall better than vector embeddings. For now, I'm calling this JAG - Judgement-augmented Generation.

On a 300 page scanned PDF x 30 questions, this took ~28s (batched, 4 pages per call, median 0.37s/page) and cost $0.08 ($0.00027/page). Made possible with Jev's image API and compared against Voyage's multimodal embeddings:

• ~58% more likely to find the right page as first result
• ~7% more likely to find the right pages(s) in the top k=6
• ~133% correctly identifying when an answer didn't exist in the doc
• ~59% cheaper retrieval cost

Not to mention, it also means you don't need a vector store or separate rerank model anymore.

Now the cons:

• This approach only really works if you can have the questions upfront - AI spreadsheets, application tracking systems, surveys etc
• You lose your vector search index so you'll need to run through every page for every new question or when existing questions change. This might be a biggest deal breaker.

Conclusion

I have a more detailed write-up, results and sample file I used for this test in my blog (see comments) if you'd like to read more.

So what do you think? The great thing about this technique is that it doesn't need to be specifically Jev (they didn't pay me to write this, though I wish they would! wink wink) but any vision-enabled decision model will probably do - though accuracy, speed and cost will need to be re-verified.

For Ragextract, I've decided to stick with Jev as a reranker as (1) it produces similarly good results (2) I still need vector search and (3) certain data retention/privacy claims are dubious with Jev which I'm hoping to clarify before real use.

A few people here asked me for help learning RAG, so instead of sending links one by one, I'm sharing it with anyone starting to learn RAG.

What it is: an agent called Neethi that answers questions about building RAG systems, using only the material loaded in its office: beginner-to-intermediate YouTube tutorials (RAG from scratch, embeddings, hybrid search, evaluation), papers (the original RAG paper, Microsoft's GraphRAG paper), Speech and Language Processing (Jurafsky & Martin), and a practical guide on chunking, citations and evaluation.

Why it's useful for learning:

• Every claim is checked against its source. Video citations jump to the exact minute.
• If the sources don't cover something, it says so instead of making it up, and those answers don't cost anything.
• It suggests follow-up questions so you can keep going down a topic.

How to access it:

1. Open the link below. You'll land in my office, where several agents work.
2. Select EN at the top right.
3. Click Neethi: the bottom-left desk with the "NEETHI" tag, or his card at the bottom of the screen ("Neethi · Experto en RAG").
4. His conversation opens on the right. Type in "Ask Neethi..." and send.
5. He helps one person at a time. If there's a line, you'll see your place, and after each answer you have 60 seconds to ask the next one.

You get a few free questions without an account, enough to see how it works.

With an account you also get:

• Your conversations saved per agent, so you can come back to your learning path
• Rating answers and leaving comments
• Saving useful answers as notes in your own space
• Attaching images to your questions

Link: https://scrapevantageai.edatasoluciones.com/oficinas/@scraper0001

Some questions to start with:

• "What's the difference between vector RAG and GraphRAG?"
• "How do I split documents by structure instead of fixed size?"
• "How do I evaluate whether my RAG answers are actually supported by their sources?"

Disclosure: it runs on an app I'm building. I'm sharing it for feedback: if an answer is wrong, cites the wrong source or misses something, comment or DM me with the question you asked and what you expected. That's exactly what I use to improve it.

I’ve been building a RAG + agent stack for financial documents and ran it on FinanceBench (Patronus AI’s 150 analyst questions over 368 SEC filings, 54k pages).

All 368 reports are searched together, and the questions are used exactly as published, with no company or period hints. The hardest part is that the same company appears across many years, with near-identical tables holding different numbers.

Retrieval: what moved the needle
On a different corpus, I built a 1,000-question retrieval test designed to break search (paraphrase, distant headings, units stated once, multi-hop chains, cross-language). Top-3 hit rate went from 96.0% to 99.9%. What mattered:

• Parent-child chunking cut along the document’s own headings: small passages matched, their whole section read
• Every passage carries its heading path (“Annual report 2023 › Results › Site B”), so identical paragraphs no longer look identical
• A metadata record per document (title, top headings, opening), with passages credited by how close the question is to their document
• A reranker (Nemotron 1B) that reads each passage after its document’s metadata. Reading the passage alone threw the year away again.
• One context line per passage, written at ingest by a small model: the subject, period or unit the passage leaves unsaid (~$0.80 per 3,000 passages, once)
• A two-step search for multi-hop questions: a quick model call names what the first results point to, and that gets searched too. Two-hop top-3 went from 70% to 100%.

What didn’t work

• An 8B reranker: worse overall and 12× slower under load
• Translating queries: no gain over a multilingual reranker
• Blending reranker order with first-stage order: a French question fell from 1st to 15th
• A vision model re-reading messy pages: one fix, two breaks

Answering
Two independent paths, a pipeline and an agent, with a referee that rejects answers drawn from the wrong year’s report or contradicting their own figures. Arithmetic is done in code. Models: gpt-oss-120b and Qwen 3.6 27B answer, Mistral Large 3 referees, bge-m3 embeds, Qdrant stores. Cost is $20 per full run (13 cents/question).

Caveats

• I graded by hand against the documents. 7 of my decisions differ from the reference answers, and all are flagged.
• FinanceBench scores across write-ups aren’t strictly comparable (different libraries, judges, question edits).

Curious where people think this breaks, and what has worked for you on near-duplicate documents.

I've been thinking about one of the frustrating problems with RAG systems: sometimes the correct information is already in the knowledge base, but the model still produces an incorrect or incomplete answer.

At first, it seems like a retrieval problem. But the more you look at the pipeline, the more places you find where things can go wrong.

For example:

• Chunking: The relevant information gets split from the context needed to understand it.
• Retrieval: The system finds semantically similar passages instead of the exact passages that answer the question.
• Ranking: The right chunk is retrieved but pushed below less relevant results.
• Context assembly: Important details get lost when multiple chunks are combined or the context window fills up.
• Generation: The LLM receives the correct evidence but ignores it, misinterprets it, or fills in missing details with assumptions.
• Evaluation: The system appears to work well on simple test questions but fails on ambiguous, multi-part, or real-world queries.

What makes this tricky is that improving one stage doesn't necessarily fix the whole pipeline. A better embedding model might improve retrieval, for example, but it won't automatically solve poor chunk boundaries or a model that doesn't follow the retrieved evidence.

For those who have built or debugged RAG systems, what has made the biggest difference in your experience?

Was it better chunking, hybrid search, reranking, metadata filtering, query rewriting, better prompts, or something else entirely?

I'm especially interested in cases where the correct information was definitely in the knowledge base, but the system still couldn't produce the right answer.

One thing worth understanding about Retrieval-Augmented Generation (RAG) is that getting a chatbot to answer questions is the easy part. Making sure it retrieves the right information, respects document permissions, and doesn't confidently return outdated or unsupported answers is much harder.

Imagine an employee asking an internal AI assistant about a leave policy or an API configuration. The answer needs to reflect the organisation's actual documentation, not just what an LLM generates from its training.

Here's how I'd approach an end-to-end enterprise document RAG project.

1. Start with document ingestion

Enterprise information is usually spread across PDFs, Word documents, internal web pages, technical manuals, and knowledge repositories.

Tools such as PyMuPDF, python-docx, and Beautiful Soup can help extract text from these sources.

I'd preserve metadata from the beginning, including document titles, versions, departments, update timestamps, and access permissions. Without it, identifying authoritative information and restricting access becomes much harder.

2. Get chunking right

Extracted text needs to be divided into smaller passages before indexing.

Fixed-size chunking is straightforward, but it can separate important instructions from their headings. Recursive or structure-aware chunking can help preserve related sections.

There's no universally ideal chunk size. Small chunks may lose context, while large chunks can introduce irrelevant information. I'd test different sizes and overlap settings against realistic questions instead of relying on default values.

3. Combine embeddings with effective retrieval

An embedding model converts each text chunk into a numerical representation, allowing semantically related passages to be retrieved even when their wording differs from the user's question.

Tools such as FAISS and Qdrant can support vector search.

However, semantic search isn't always enough. Exact error codes, product identifiers, and policy numbers may be better handled by keyword search. Combining both approaches through hybrid retrieval, followed by reranking when useful, can improve the relevance of retrieved passages.

4. Generate answers grounded in evidence

A typical query pipeline looks like this:

User question → Retrieval → Reranking → Prompt construction → LLM response → Source citations

The LLM receives the question alongside relevant document passages. The prompt should instruct it to use the supplied evidence and acknowledge when the available information is insufficient.

For example, if an internal API manual explains authentication but doesn't specify rate limits, the assistant shouldn't invent a rate limit.

Frameworks such as LangChain and LlamaIndex can help organise these workflows, but they don't automatically guarantee accurate answers.

5. Treat security and document updates as core features

Suppose an organisation replaces its old leave policy with a revised version. If the outdated document remains indexed, the assistant might continue returning obsolete instructions.

Document version tracking, update detection, re-indexing, and deletion workflows help prevent this problem.

Access control matters just as much. Permissions should be enforced before restricted content reaches the LLM. Retrieved documents can also contain malicious instructions, so applications need safeguards against prompt injection.

Source citations should identify the documents, pages, or sections supporting an answer. Ideally, the system should also verify that those references support the claims being made.

6. Evaluate the complete pipeline

A fluent response doesn't necessarily mean the system retrieved the right information.

I'd evaluate:

• Retrieval quality: Did it find the relevant passages?
• Groundedness: Are the claims supported by evidence?
• Citation accuracy: Do the references support the answer?
• Access control: Can users retrieve only authorised information?
• Performance: What are the response latency and token costs?

A small test set of realistic business questions provides a useful starting point for comparing chunking strategies, embedding models, and retrieval configurations.

A practical project to build

I'd start with a company policy assistant that supports document uploads, text extraction, vector retrieval, source citations, and metadata filtering. Once that works, I'd add document updates, access permissions, and an evaluation suite.

This project connects several practical skills: Python, data processing, search, LLM integration, APIs, security, and evaluation. It's also a useful way to understand why production RAG requires more than connecting a vector database to a chatbot.

I write about data engineering, analytics, and AI career development as part of my work with IntelliBI, a data and AI training institute in Pune. I'm particularly interested in how these technologies move from demos into practical applications.

For those who've built RAG systems, what caused the biggest challenge in your projects: chunking, retrieval quality, outdated documents, evaluation, or access control?

I'm learning about Agentic AI and RAG, and I'm trying to understand how to design a system that works reliably in real-world scenarios.

One thing I've been questioning is the common RAG pipeline:

Load documents → Split into chunks → Generate embeddings → Retrieve similar chunks → Send them to an LLM → Generate an answer.

This works for some use cases, but I feel that it misses some important architectural questions.

1. How should we handle different types of data?

Real-world information doesn't come only from PDFs or text files. It can come from Word documents, spreadsheets, SQL databases, websites, images, videos, and structured or unstructured sources.

Should we process and classify these sources before indexing them? For example, extract structured information, identify document types, preserve metadata, and assign traceable IDs to sources and chunks.

Not every question should be answered through semantic search. If someone asks for total sales, querying a database might be more appropriate than retrieving text chunks.

2. How can an agent decide whether a question needs RAG at all?

Users can ask almost anything. We cannot predict every possible question in advance.

For example:

"What is a Transformer?" — This may only require the LLM's general knowledge.

"What is our company's refund policy?" — This requires retrieving internal documents.

"What were our total sales last month?" — This may require a SQL query.

"What is the latest version of this library?" — This may require searching up-to-date external sources.

"Why did our sales decrease last month?" — This may require combining database queries, reports, and reasoning.

How do you build a reliable query router that understands the user's intent and selects the appropriate tools or knowledge sources? Should this be handled by an LLM, explicit rules, a classifier, or a combination of these?

3. How do we make retrieval source-aware rather than just similarity-based?

Suppose several documents discuss the same topic but contain different information because of different versions, dates, or contexts.

I think a better approach would be to:

Classify and normalize information before indexing it.

Preserve source IDs, document versions, timestamps, page numbers, and other metadata.

Combine semantic search with keyword search, metadata filtering, and reranking.

Compare retrieved information to identify duplicates, updates, and genuine contradictions.

Keep every chunk linked to its original source.

Generate answers that distinguish between sources and cite the evidence supporting each claim.

In other words, the goal should not simply be to retrieve text that resembles the question. It should be to retrieve the right evidence and understand what each source actually says.

4. What happens when the system cannot find enough evidence?

A reliable system should not force an answer when its knowledge base does not contain the required information.

How do you evaluate retrieval quality, detect missing evidence, decide whether to search again, use another tool, or ask the user for clarification?

My main question

If you were building an Agentic RAG system from scratch today, what architecture would you choose to handle heterogeneous data, dynamically route questions to the right information source, and generate answers with reliable source attribution?

I'm especially interested in practical architectures, design patterns, research papers, and real-world implementations rather than just choosing a framework such as LangGraph, CrewAI, or another agent library.

I'd appreciate explanations of the reasoning behind your design choices, including the trade-offs and failure cases you've encountered.

Thanks!

I've been working on catching wrong sentences in legal answers while the model is still writing, instead of checking the whole answer at the end.

The usual way is to let the model finish, check the answer, and regenerate all of it if one sentence is wrong. You pay for every token of the bad answer and then throw it away.

So I tried checking one sentence at a time. Here's how it works:

1. The model writes a sentence.
2. A second model, the judge, checks it against the source text. For me that's UK statutes.
3. If it passes - it stays. If not - I cut it and add the relevant bit of the statute as a new message right where the cut is and the model tries the sentence again.

It runs on SGLang. While a model reads text it builds up a working memory of it called the KV cache. SGLang keeps that memory and reuses it when the next request starts with the same text so a retry only has to process the new part. On Qwen 2.5 7B on one L4, a retry took 117 ms. Processing the whole prompt again took 270 ms.

I tested it on 240 questions about UK statutes in October 2026. The judge was Qwen 3.8 27B, quantized to 4 bits (not finetuned) so it fits on one L40S. In the 27B runs it also wrote the answers and checked its own work.

• When the model answered from memory with no source in the prompt, answers with at least one wrong sentence went from 132 to 23 for Qwen 27B. For Qwen 2.5 7B (Qwen 27B as judge) they went from 198 to 28.
• I also compared retry methods on 95 real mistakes from the 7B. The default retry stops the model from starting the sentence the same way and it got it wrong again 71 times. Adding the statute text as a message at the cut got it wrong again 16 times.
• With the source already in the prompt, the loop still helped the 7B from 25 wrong answers down to 13. For the 27B it did nothing. It had 4 wrong answers with the loop and 4 without and each answer took 0.9 s longer. On the same questions Gemini 3.8 Flash had 3 and Claude Opus 4.6 had 8.

So it's worth it when the model answers from memory or when the model is small. If you already put the source in the prompt and use a strong model it mostly costs you time.

A few things didn't work.

• My first judge was DeBERTa-v3, a small model that scores whether a sentence follows from a piece of text. On a test set where I planted the errors myself, 89% of the sentences it rejected were really wrong. On real answers about statutes that fell to 39%, so most of what it flagged was fine. Qwen 27B as the judge got 90.1% of 151 unseen sentences right. DeBERTa got 58.3%.
• I tried writing the statute text straight into the KV cache at the cut instead of adding it as a message. It was no better.
• I tried reading the model's own attention and token probabilities to spot wrong sentences. They didn't separate right from wrong.

Each setup ran once with greedy decoding, so no randomness. Claude and Gemini models graded most of the sentences and I graded the ones they disagreed on or weren't sure about myself.

The code and every result file are on GitHub: https://github.com/azterizm/legal-rag-verifier

I wrote it up in more detail here: https://memonsystems.com/journal/in-flight-rollback-on-sglang-measured-against-gemini-and-claude-opus/

Happy to answer questions. I'd also like to hear from anyone who got the KV cache approach to work better than plain text.

For anyone running a local model to check what another agent reads, have you tried putting a smaller model in front of tool responses or retrieved documents?

I’m wondering how you handle text that quotes an attack for a legitimate reason, like a security article. Blocking every “ignore previous instructions” would get annoying fast.

What’s been harder in your setup: missed injections or too many false alarms?