Confidentiality: INTERNAL · Status: DRAFT - REVIEWED BY AI

Extraction accuracy - investigation & recommendations

Date: 2026-06-12 · Scope: three pipeline changes evaluated against ground truth across the test-data set, headlined on Document-combined.pdf.

Summary

We investigated three changes aimed at lifting extraction accuracy - semantic hints, LLM figure recovery, and a dual (Textract + BDA) pass. Bottom line:

ship LLM figure recovery + semantic hints give a real, near-free lift in figure detection (Textract 56% → ~88%) with no regression on clean documents - adopt as the default.
limited The dual pass shows some success but the improvement is relatively minor - and not worth its extra cost and latency. Keep it only as an optional, confidence-gated fallback for clearly figure-poor papers.
explore An LLM “layout pre-pass” (profile 2–3 pages first) is the most promising next step and could replace the hand-coded hints.

The three changes - how each helps

1. Semantic hints

A deterministic pre-Claude pass that tags headers/footers/cover pages, detects question numbers and shared resources, adds a “numbering restarts per section” note for Challenge-style papers, and flags axis/graph references - injected into the enrichment prompt.
Helps by nudging Claude on messy, multi-source layouts: keeping orphaned axis/label text attached to the right question and warning that repeated question numbers are expected.
Cost: none (no extra model call). Advisory - most useful for weaker models / messy documents, marginal on strong runs, no harm on clean ones.

2. LLM figure recovery (the `LLM_FIGURE` / prompt change)

The enricher may now add diagrams the extractor missed (line-only grids, charts, geometric figures) with an estimated bounding box, tagged LLM_FIGURE, with guardrails (valid box, minimum size, per-page cap).
Helps by closing the biggest extraction gap: Textract figure recall 56% → ~88% on Document-combined; scanned docs 0% → 62%; clean text docs stay 100%.
Cost: negligible - same enrichment call. Boxes are approximate, but detection is the win. Works with or without semantic hints (it lives in the base prompt).

3. Dual pass (Textract + BDA, merged)

An operator-triggered rescan gated by a new confidence score: reuse the cached Textract extraction, add a BDA extraction, merge (Textract text + BDA figures/extra text), enrich once.
Helps by recovering some figures and text a single extractor misses, so it has some success on weak inputs. But the gain is modest, and the raw figure-recall metric overstates it: the merge boxes many regions separately (e.g. each pictogram), inflating the figure count without a matching accuracy benefit.
But on an already-strong Textract+LLM result the question accuracy was mixed/down (F1 0.354 → 0.299, text precision 52% → 43%) at ~+19% cost and +35% latency.

Page 18 Textract vs Dual figure overlay — Dual detects more figure regions but **over-segments**: page 18 jumps from 3 boxes (Textract, teal) to 14 (Dual, green) as each pictogram is boxed separately - which inflates the figure metric more than it reflects a real accuracy gain.

Recommendation

Ship LLM figure recovery + semantic hints together as the default. Effectively free (no extra pass), a real measurable lift in figure recall, and no regression on clean documents.
Don’t adopt the dual pass as a routine step. It has some success on weak inputs, but the accuracy gain is minor and not worth the extra extraction cost and latency - and its headline figure-recall improvement is partly inflated by over-segmentation rather than a genuine gain. Keep it only as an optional, confidence-gated fallback for clearly figure-poor papers, with modest expectations.
Worth prototyping - an LLM “layout pre-pass”. Before the main run, send just 2–3 representative pages to an LLM to profile the paper: numbering scheme (e.g. section/Challenge restarts), shared-resource patterns, answer-box conventions, figure types, headers/footers (the “gotchas”). Feed that document-specific brief into the enrichment prompt. This could replace the hand-coded semantic hints with a tailored guide and is the most promising path to a significant accuracy gain - for one small extra call, far cheaper than a full dual extraction. Build it as an experiment and measure against the current sweep.

Supporting data

Full cross-document figure-recovery results (OLD vs NEW)

Headline - Document-combined.pdf

56% → 89%

Textract figure sensitivity (Sonnet) - missing diagrams recovered

36 → 9

Missed figures (of ~76)

99%

BDA figure sensitivity (was 93%)

Document-combined page 1 before vs after figure recovery — Document-combined, page 1 (Textract · Sonnet). **Before**: 0 figures. **After**: the coordinate grid recovered as a green `LLM_FIGURE`.

Effect across the other test-data types

Figure sensitivity and missed-figure count, OLD → NEW. Clean text documents are unaffected; scanned and diagram-heavy documents gain the most.

Textract

Document	Model	Fig sens OLD	Fig sens NEW	Δ fig sens	Missed OLD→NEW	Δ F1
Document-combined (36-page multi-source)	`haiku-4.5`	56%	90%	▲ +34%	36 → 8	▼ -0.006
Document-combined (36-page multi-source)	`sonnet-4.6`	56%	89%	▲ +33%	36 → 9	▼ -0.051
11+ SPaG (text-only)	`haiku-4.5`	100%	100%	no change	– → –	no change
11+ SPaG (text-only)	`sonnet-4.6`	100%	100%	no change	– → –	▼ -0.084
12-page maths	`haiku-4.5`	70%	75%	▲ +5%	6 → 5	▲ +0.007
12-page maths	`sonnet-4.6`	70%	75%	▲ +5%	6 → 5	▲ +0.003
1-page maths	`haiku-4.5`	100%	100%	no change	0 → 0	no change
1-page maths	`sonnet-4.6`	100%	100%	no change	0 → 0	▼ -0.040
rescanned maths (scan)	`haiku-4.5`	62%	62%	no change	3 → 3	▼ -0.012
rescanned maths (scan)	`sonnet-4.6`	62%	62%	no change	3 → 3	▼ -0.187
rescanned phone maths (scan)	`haiku-4.5`	0%	62%	▲ +62%	– → 6	▲ +0.038
rescanned phone maths (scan)	`sonnet-4.6`	0%	38%	▲ +38%	– → 10	▲ +0.025
scanned phone maths (scan)	`haiku-4.5`	30%	53%	▲ +23%	33 → 22	▲ +0.040
scanned phone maths (scan)	`sonnet-4.6`	30%	51%	▲ +21%	33 → 23	▲ +0.026

BDA

Document	Model	Fig sens OLD	Fig sens NEW	Δ fig sens	Missed OLD→NEW	Δ F1
Document-combined (36-page multi-source)	`haiku-4.5`	93%	99%	▲ +6%	6 → 1	▲ +0.012
Document-combined (36-page multi-source)	`sonnet-4.6`	93%	98%	▲ +5%	6 → 2	▼ -0.026
11+ SPaG (text-only)	`haiku-4.5`	100%	100%	no change	– → –	no change
11+ SPaG (text-only)	`sonnet-4.6`	100%	100%	no change	– → –	no change
12-page maths	`haiku-4.5`	70%	75%	▲ +5%	6 → 5	▲ +0.003
12-page maths	`sonnet-4.6`	70%	75%	▲ +5%	6 → 5	▲ +0.003
1-page maths	`haiku-4.5`	100%	100%	no change	0 → 0	no change
1-page maths	`sonnet-4.6`	100%	100%	no change	0 → 0	no change
rescanned maths (scan)	`haiku-4.5`	38%	75%	▲ +38%	5 → 2	▲ +0.039
rescanned maths (scan)	`sonnet-4.6`	38%	62%	▲ +25%	5 → 3	▼ -0.020
rescanned phone maths (scan)	`haiku-4.5`	38%	62%	▲ +25%	10 → 6	▼ -0.006
rescanned phone maths (scan)	`sonnet-4.6`	38%	56%	▲ +19%	10 → 7	▲ +0.010

rescanned phone maths before vs after figure recovery — Other-type example - **rescanned phone maths** (a phone scan): figure sensitivity 0% → 62% as diagrams are recovered as `LLM_FIGURE`s.

Reading the results

Big wins where diagrams were being lost: Document-combined (Textract 56%→~90%, missed 36→~8), rescanned phone maths (0%→62%), scanned phone maths (30%→~52%), BDA rescanned maths (38%→75%).
No harm on clean documents: 11+ SPaG (text-only) and 1-page maths stay at 100%.
Question F1 mostly flat, with a few Sonnet regressions (recovered figures sometimes carry sub-part labels that add zero-scoring classes to the macro average).
Bounding boxes remain approximate (mean IoU ~0.34–0.41) - detection improved markedly, geometry is still an estimate.