Agentic RFP & Vendor Evaluation
Replaced a ~3-month manual tender review with an agentic workflow; ~4 hours of reviewer time saved per vendor response
A three-stage agentic workflow for procurement: completeness checking, rubric-based scoring and side-by-side vendor comparison, with human approval gates at every decision point.
- Airia.ai
- LLM-as-a-judge
- Human-in-the-loop
- Guardrails
- Prompt engineering
- SharePoint integration
Problem
Tender evaluation ran on document libraries, email and spreadsheets. Each year brought 10–15 tenders, each with 5–8 vendor responses covering modern slavery, environmental and standard questionnaires. A reviewer spent roughly four hours per response checking that questions had actually been answered before any assessment could begin, and a full tender took around three months end to end.
The expensive part was not judgement; it was the reading. Scoring was also inconsistent between reviewers, because the rubric lived in people’s heads rather than in the process.
Approach
I split the work into three stages, so that each one could be evaluated and trusted independently rather than as a single opaque agent:
- Completeness checking: does this response actually address every question, including the ones answered obliquely or in an attachment?
- Rubric-based scoring: an LLM-as-a-judge pass against explicit criteria, producing a score with the supporting quote attached, not a bare number.
- Comparison: a dashboard putting vendors side by side on the same axes.
The design decision that mattered most was keeping a human approval gate between stages. Full autonomy was achievable, but procurement decisions are contestable and need an audit trail, so the system is built to produce a recommendation a person signs off on rather than a decision it makes alone. Guardrails and structured outputs keep each stage’s result machine-checkable.
I deliberately did not build a custom orchestration layer. Airia.ai, a low-code agentic platform, covered the workflow, SharePoint and email integrations, and approval routing, which kept the delivery focused on prompt and rubric quality, the part that actually determined whether the output was usable.
Result
Three stages live in production, progressed from proof of concept through to production use. Modern slavery compliance checks are now standardised across the vendor base (43 completed in FY25 and 27 so far in FY26) where previously each was a manual read.
Reviewers still make every call. What changed is that they start from a structured, evidenced summary instead of a stack of PDFs.
What I’d do differently
The scoring rubric went through more iterations than it should have because early versions asked the model for a judgement without asking it to cite the evidence first. Requiring the quote before the score, rather than alongside it, made scores both more accurate and far easier for a reviewer to check. That should have been the starting design.