Project · Python · FastAPI · Postgres + pgvector
citeline
A retrieval augmented question answering service over the US National Primary Drinking Water Regulations. It answers with a citation to the exact regulation section, or it tells you it does not know. There is no third option, and that is enforced in code rather than requested in a prompt.
The problem
A retrieval demo that answers every question looks impressive and is nearly worthless. The interesting question is not whether a system can answer when the answer is present. It is what the system does when the answer is absent, because that is the case that occurs constantly in real use and it is the case where a language model is most confident and most wrong.
I picked federal regulations as the corpus for exactly that reason. Regulatory text is dense with numeric thresholds, deadlines, defined terms and cross references. A confidently wrong contaminant limit is not an amusing error; it is the kind of error that gets a compliance tool thrown out. So the requirement is not "answer well". It is never assert anything that is not in the retrieved text, and prove that property rather than claim it.
Why it is harder than it looks
Vector search alone quietly fails on regulations
Ask for the limit on arsenic and a dense index returns several passages about arsenic monitoring, sampling schedules and treatment technology. They are all genuinely about arsenic and semantically very close to the question. None of them contains the number. Embeddings blur exactly the tokens that matter here: numeric thresholds, units, and section identifiers.
So retrieval runs two ways at once. A dense vector search over pgvector handles paraphrase, and a Postgres full text search pins the exact terms. Their results are merged with reciprocal rank fusion, which combines by rank rather than score and therefore needs no normalisation between two retrievers whose numbers have nothing to do with each other.
Chunking on a fixed size destroys the answer
The contaminant limits live in tables. Slice those on a character count and you get a chunk holding a contaminant name whose number landed in the next chunk, and a chunk holding a number with nothing to say what it measures. Both look like fine retrieval results and both are useless. The chunker splits on paragraph boundaries first, falls back to sentence boundaries only when a single paragraph exceeds budget, and prefixes every chunk with its own section number and heading so a retrieved fragment is always attributable on its own.
A rank score is not a confidence score
This one cost me a rewrite, and it is the most useful thing on this page. My first version gated abstention on the fused retrieval score. That is wrong twice over.
First, arithmetically: with a fusion constant of 60 and two retrievers, the maximum
achievable fused score is 2/61 = 0.0328. I had set the threshold to
0.045, which no result can ever reach, so the service would have
abstained on every question ever asked.
Second, and worse, the approach could not have worked even with a reachable number.
Reciprocal rank fusion is a rank statistic. A dense retriever returns its k
nearest rows for any input whatsoever, so whatever is closest still lands at rank 1
and still scores 1/61. The top fused score for
what is the capital of France looks almost identical to the top fused score
for a question the corpus genuinely answers. It carries essentially no information
about coverage.
The gate now tests absolute cosine similarity, which does move between those two cases, plus an independent requirement that the sparse retriever matched something at all. Fusion still decides the ordering. It just no longer pretends to measure confidence.
Architecture
Two paths through one system: a batch pipeline that builds the index, and a request path that serves it.
0.015 mg/L and 141.80.
Fusing them means a passage has to satisfy only one of the two to survive, and the ones both
agree on rise to the top.
The three gates
gate 1 · before the model
If no passage is semantically close enough, or no passage contains the question's terms, the language model is never invoked. This is the cheap gate and it catches the common case. Its threshold was chosen by sweeping the labelled set, not by judgement.
gate 2 · the prompt
The model is instructed to reply with a fixed marker when the supplied excerpts do not answer the question. This is the weakest of the three, because it is the only one that depends on instruction following, which is why it is not the only one.
gate 3 · after the model
Every citation the model emitted is checked against the excerpts it was actually handed. A citation to an excerpt that was never supplied, or a substantive sentence carrying none, fails the whole answer. It is discarded, not served with a warning attached.
Engineering trade-offs
The decisions that shaped this, and what each one cost. None of them is the only defensible choice; these are the reasons for the ones I made.
| Decision | Chosen | Instead of | Why, and what it costs |
|---|---|---|---|
| Fusion | Reciprocal rank fusion | A cross-encoder re-ranker | RRF combines ranks, so it needs no shared score scale between a cosine distance and a lexical relevance score, and no model. A re-ranker would likely order the top few better, but it adds a second model to the request path on a host that already cannot afford the first one. The cost is real: RRF cannot tell that the third result is much better than the fourth, only that it ranked higher. |
| Abstention threshold | Cosine 0.62, measured | A tuned guess | Swept against the eval set and chosen where out-of-scope questions were blocked without starting to block answerable ones. The first attempt gated on the fused RRF score, which was wrong: the highest possible fused score was lower than the threshold being asked for, so nothing could ever pass. Raw cosine is comparable between questions; a fused rank score is not. |
| Chunking | On section boundaries | Fixed character windows | A citation is only useful if it names something a reader can look up. Fixed windows split a limit away from the contaminant it applies to and produce citations that point at nothing. The cost is uneven chunk sizes and a few sections too long for one chunk. |
| Embeddings | Local, nomic-embed-text | A hosted embedding API | No key to leak, no per token cost, and no corpus text leaving the host, which is the posture a regulated customer will ask about. The cost is quality: a larger hosted model would almost certainly retrieve better. |
| Generation on demand | Off, answers precomputed | Generating per request | Measured on this host, one grounded answer ran past four minutes. The CPU is a virtual one with no AVX or SSE4.2, so llama.cpp falls back to scalar arithmetic and no amount of tuning closes that gap. Retrieval stays live because it is fast and it is the part worth showing. |
Query it
This is the running service, not a recording. Try a question the regulations answer, then try one they do not, and watch it decline.
The dashed ones are outside the corpus. The last two are the interesting ones: benzene is in these regulations with a drinking water limit, so asking for its air standard is a near miss, and PFOA is a real contaminant that this edition of the corpus genuinely does not cover. A keyword filter is fooled by both.
Retrieval alone
What any RAG service has at this point
With the gate and the citation contract
What this service does with it
Evaluation
A labelled set of 28 questions: 20 the regulations answer, with the section that answers each one, and 8 they do not. Scoring asserts specific strings rather than asking a model to grade, so the result is reproducible and cannot drift with a grader's mood.
Evaluation figures are generated by eval/run_eval.py and inserted here after each run.
What is measured, and why both halves matter
Retrieval quality
- recall@k: did the correct section appear in the top k at all
- hit@1: was it ranked first
- MRR: mean reciprocal rank of the correct section
- latency: p50 and p95, measured end to end
Answer integrity
- accuracy: answered, contained the required fact, cited the right section
- abstention recall: share of unanswerable questions correctly refused
- false answer rate: unanswerable questions that got an answer anyway. This is the hallucination rate, and it is the number the whole design exists to hold down
Run it yourself
The repository is self contained. Docker brings up Postgres with pgvector; the models run locally through Ollama, so there is no API key and nothing leaves the machine.
git clone https://github.com/kenbin64
cd citeline
docker compose up -d # postgres 16 + pgvector
ollama pull nomic-embed-text # embeddings, 768-d
ollama pull llama3.2:3b # generation
make install # venv + dependencies
make migrate # apply the schema
make ingest # fetch, gate, chunk, embed, index
make check # ruff + mypy + 31 unit tests
make eval # regenerate the numbers above
citeline search "MCL for arsenic" # retrieval only
citeline ask "MCL for arsenic" # full pipeline
make ingest is idempotent. Run it twice and the second run writes
nothing, reporting every chunk as skipped, because each document is compared by
content hash before any work is done.
Limits
Things this does not do, stated plainly so nobody has to discover them.
- No reranker. A cross encoder would likely improve ranking, but it cannot run on this CPU for the reason above. The interface for it exists; the model does not.
- The eval set is small. 28 questions written by one person. It is enough to catch a broken gate and to set a threshold, and not enough to claim a general accuracy figure. Treat the numbers as a regression baseline, not a benchmark.
- One corpus, one part. Part 141 only. Nothing about the design is specific to it, but no other body of text has been tested, so no claim is made about them.
- Gate 3 verifies grounding, not correctness. It proves every claim carries a citation to a passage that was actually supplied. It does not prove the model read that passage correctly. That is a real gap, and it is why the answer always shows its sources for a human to check.
- Abstention has a cost. A tighter gate refuses some questions the corpus genuinely answers. That trade is deliberate and the sweep table shows exactly what it costs at each threshold.