Statements QA Automation
350+ automated checks replaced brittle Excel macro QA
- 350+ checks
- 2 validation tracks
A document intelligence pipeline that extracts key fields from year-end financial statements and cross-validates them against member database records.
Problem
Year-end statements needed their key fields checked against member database records before anything could be signed off. The existing process ran on Excel macros and hardwired validation logic, which created three separate problems: it was brittle against format drift, so a small change in how a statement was laid out could break a check silently; it was opaque when something failed, reporting that a statement hadn’t passed without saying which field or why; and it was expensive to extend, because every new field or edge case meant editing macro logic by hand.
The underlying difficulty is that not every check is the same kind of check. Some are genuinely deterministic — a figure either reconciles or it doesn’t. Others are judgement calls about whether two differently-worded records describe the same thing, which a fixed rule can’t cleanly express. The old process forced both into the same rigid mechanism.
Approach
- Field extraction from the statements themselves, decoupled from the validation logic that follows, so a change to one doesn’t require touching the other. Format drift becomes an extraction fix rather than a rewrite of the rule set.
- Deterministic rule sets for the checks that genuinely are rule-based.
- LLM-as-a-judge scoring across 350+ checks, for the judgement calls a fixed rule can’t express, with fuzzy scoring where a near-match is meaningful.
- Cross-validation against the member database, so a discrepancy surfaces as a specific, named mismatch rather than a generic failure.
The deterministic and LLM-based checks were kept as separate, visible tracks rather than merged into a single score. A reviewer can tell at a glance whether a flagged statement failed a hard rule or a softer judgement check — which is the difference between a correction and a conversation.
Outcomes
Replaced the Excel macro and hardwired validation process with an automated pipeline covering 350+ checks.
- Finance reviewers signing off year-end statements see a named mismatch on a specific field instead of an opaque failure, so the question becomes what to do about it rather than what went wrong.
- The team maintaining it can add a check without editing macro logic, and can tell which of the two tracks a new check belongs in.
- Error rate and rework both dropped, because discrepancies surface during validation rather than after sign-off, when correcting them costs most.