Large-Scale Unstructured Notes Processing
1M+ free-text notes and 1B+ tokens turned into a structured data space during a platform migration
A cloud LLM pipeline that summarised and extracted implicit and explicit fields from over a million unstructured member-conversation notes, built to feed a structured data model during a company-wide platform transition.
- AWS Bedrock
- Step Functions
- Lambda
- S3
- Claude 3.5 Sonnet
- Fuzzy scoring
- CI/CD
Problem
Several years of urgent notes from member conversations, over a million of them, existed only as free text. A platform migration needed that history as structured data, and the fields it needed were not consistently written down: some were stated explicitly, others were implied by the way a note was phrased.
Manual extraction across a corpus that size was never a realistic option.
Approach
The pipeline is straightforward per-record work; the engineering was in making a million of them finish reliably and affordably.
- Throughput. Threading, batch processing and requests spread across parallel Bedrock accounts, so provider rate limits stopped being the binding constraint on wall-clock time.
- Accuracy. Fuzzy scoring against a manually validated sample, so quality was a measured number per field rather than a general impression. Fields that scored badly got prompt work; fields that scored well were left alone.
- Cost. Model choice and prompt size were tuned per field type. Extraction of an explicit field does not need the same budget as inferring an implicit one.
The failure mode I designed against was silent partial success: a batch that looks complete but quietly dropped records. Step Functions orchestration with per-stage state made the pipeline resumable and made gaps visible instead of invisible.
Result
Over 1 billion tokens processed, with per-field accuracy measured and validated rather than assumed. The structured output fed the new platform’s data space on the migration timeline.
What I’d do differently
Validation sampling started too late. Building the fuzzy-scoring harness before the first full run, rather than after, would have caught two prompt-level misreadings while reprocessing was still cheap.