Intelligent Legal Document Processing
AUD 220k saved against vendor quotes: 10,000+ contracts and 130k+ pages processed for about AUD 5k of infrastructure
An automated extraction pipeline pulling 20+ structured fields from more than 10,000 heterogeneous contract documents, built to support a legal team contract cleanup that had been quoted as an outsourced project.
- AWS (S3, Textract, Lambda, Step Functions)
- Azure OpenAI GPT-4o
- LLM-as-a-judge
- Fuzzy scoring
- Apttus & Salesforce REST APIs
- Prompt engineering
Problem
A legal team needed to clean up a contract repository held in Apttus: over 10,000 documents, more than 130,000 pages, in a mix of plain text, Word and PDF, including scans. Each document needed 20+ structured fields extracted before any cleanup could happen.
An external vendor had quoted the work. The question was whether it could be built instead.
Approach
Format heterogeneity was the first real obstacle, not the extraction itself. Text-layer PDFs, Word documents and scanned images each need different handling before a model sees them, so OCR ran as a distinct stage with its own quality check rather than being folded into extraction.
For the extraction pass:
- Structured outputs per field, so a downstream consumer never had to parse prose back into data.
- Two-track validation: automated LLM-as-a-judge scoring across the corpus, plus fuzzy scoring against a human-checked sample. Agreement between the two was itself a signal; where they diverged, the field needed prompt work.
- Confidence surfaced per field, so the legal team could review the uncertain minority rather than spot-checking at random.
Contract extraction is exactly the setting where a confident wrong answer is worse than an abstention, so the pipeline was tuned to flag rather than guess.
Result
Over 20 million tokens processed, at an infrastructure cost of roughly AUD 5k, about AUD 220k less than the quotes to have the work done externally. Confidence was established through both manual and automated validation before the output was handed over.
What I’d do differently
The OCR stage deserved its own quality gate from the beginning. A handful of poorly scanned documents produced plausible-looking extractions from garbled text, and those were caught by sampling rather than by the pipeline. A readability score on the OCR output would have caught them automatically.