RAG and agents: giving the model your documents and tools

How retrieval grounds an answer in your protocol, and what changes when the AI can take actions.

10 min

Retrieval-augmented generation (RAG) fixes part of the hallucination problem by searching a trusted set of documents first and handing the matching passages to the model with the question. Tools such as OpenEvidence, and “chat with this PDF” features, work this way.

1. Question

Resident asks: "When does our protocol say to repeat the lactate?"

2. Search

The question is turned into numbers (an embedding) and compared with every chunk of the hospital documents.

3. Retrieve

The closest chunks come back with similarity scores.

4. Ground

The chunks are pasted into the prompt with an instruction: answer only from these, cite them, say "not found" if they are silent.

5. Answer

Answer: Repeat the lactate within 2 hours if the first value is above 2 mmol/L [2]. Every claim points to a source you can open.

The protocol text here is made up for the demo.

RAG is only as good as its library and its search. If the right passage is not retrieved, the model may fill the gap from memory. Good systems show their sources so you can click through, and say “not found” rather than guess.

An agent goes a step further: the model can call tools (search, a calculator, a database, an order-entry system), look at the result and decide what to do next, in a loop. That makes it more useful and more dangerous. A wrong answer in a chat window is read by a doctor; a wrong action by an agent may happen before anyone reads it.

Embedding: a list of numbers that captures the meaning of a piece of text, so similar meanings sit close together.

Vector database: a store of embeddings that can quickly find the passages closest in meaning to a question.

Human in the loop: a person approves each output before it takes effect. “Human on the loop” means a person supervises and can intervene, but outputs flow without approval.

MCP: Model Context Protocol, an open standard for connecting AI models to tools and data sources.

Check yourself

1. The main purpose of RAG is to:

2. A RAG system fails to retrieve the right passage. The answer may:

3. What makes an agent riskier than a chatbot?

Educational material, not clinical advice. Interactive numbers are teaching examples unless a source is named.