SAMPLE interface fixtures. The answers and marks on this trial page are for testing.
Answers beside the page they cite
A small office keeps its rules in contracts, price lists and a policy handbook, and somebody has to find the clause. We build assistants that answer from those files on the office's own computer and quote the page for every claim. This page records one run: six SAMPLE documents, sixteen questions, every answer as it came back.
The office and its documents
Ternwick Properties is a fictional brokerage in Dubai. Its people, its buildings and its six documents were written for this run: 27 pages, each marked SAMPLE. No person, building or price in them is real.
Sixteen recorded questions
This is a recording. Nothing on this page is asked live. Each answer appears exactly as the pipeline returned it; select a numbered marker to see the quoted words on the page.
Scores
Every answer was marked against its cited pages by an automated reviewer: a second language model, separate from the pipeline, that built none of it and did not see its code. No person has re-marked the sixteen. For the thirteen answers that cite a page, the page is shown beside the answer so that the mark can be checked by eye; the two declines and the rejected draft cite no page. A pass is an answer that is correct and supported by its quoted words, or a correct decline. We publish a run only when at least six of the eight held-out answers pass and no answer is wrong while showing a citation.
Visible set: 8 of 8
Held-out set: 7 of 8
| ID | Question | Status | Mark |
|---|
How the run was made
- Reading. Text, word positions and ruled tables were extracted from each of the 27 pages. Repeated page headers and footers were identified by their position and repetition and kept out of the passages.
- Passages. The text was divided at clause, paragraph and table-row boundaries into 297 passages. Each passage carries its document, page and heading; each table row carries its header row.
- Versions. Statements of validity and replacement were recorded at this stage, so that a withdrawn document is identified as withdrawn before any question is asked.
- Evidence. For each question the 297 passages were ranked by word overlap. In a first call, a language model running locally on one studio laptop read all of them, with the top-ranked ones marked, and selected up to twelve. Its selection, the highest-ranked passages of two searches and the validity statements of the documents concerned went to a second call: 32 to 47 passages per question in this run.
- Answer. In the second call the same model wrote one sentence per claim from those passages and attached passage labels to it. A label resolves to a document, a page and the exact words: the model does not retype a quotation.
- Check. The pipeline searched each quoted span on its cited page. It then compared the numerals, written dates, currency codes and reference codes of each claim with that claim's own quotations. A number written as a word, such as "one working day", is outside this comparison. One mismatch rejected the whole answer. No answer was corrected.
The visible set was used while the pipeline was built. The held-out set was written from the documents alone by a separate writer, fixed by a hash before any pipeline code existed, and run once on the finished pipeline. One held-out question and one visible question turned out to ask, in different words, about the same price and the same replacement notice.
Across the two runs: 13 answered, 2 declined, 1 rejected by the pipeline's own check. Median time per question was 4.33 s and the longest was 7.53 s, on one laptop graphics card. The network guard counted 0 connection attempts to any address other than the computer itself.
Limits
- Six documents and 27 pages. Retrieval on an archive of thousands of pages is a separate test.
- Sixteen questions. The scores are counts on these questions, not a rate for other documents.
- The PDFs hold selectable text and ruled tables. Scanned pages and tables without rules were not tested.
- English only.
- No question in either set needs a figure to be computed, so arithmetic was not tested.
- A matching quotation shows where the words are. Whether they answer the question was judged by the automated reviewer, and the marks are its marks.