02 · Contract Intelligence
Turning contract language into product infrastructure.
Two applied-LLM systems transformed difficult aviation documents into portfolio intelligence for sales and visible trust signals for marketplace customers.
- Program documents
- 1,100+
- Operator policies
- 532
- Badge-ready without review
- 84.7%
System view
From unstructured PDFs to a visible product decision.
The architecture used deterministic extraction where possible, language models where interpretation was necessary, and human review where the source itself remained ambiguous.
Two production paths
One capability, applied to two different user needs.
01
Chamomile · Program intelligence
From hackathon prototype to portfolio-level contract intelligence.
The project began as a second-place individual data-science hackathon entry. Regex handled deterministic extraction cheaply; a self-hosted Llama 3.1 8B model on SageMaker cleaned and summarized the result without sending sensitive contracts to a frontier API.
1,100+PDFs
→RegexFirst pass
→Llama 8BClean + summarize
→SnowflakePortfolio analysis
02
Cancellation policy badging
Route easy cases cheaply. Escalate ambiguity deliberately.
For 532 third-party operator contracts, Haiku handled the fast, inexpensive first pass and Sonnet received ambiguous cases. Structured outputs became flexibility tiers and refund-eligibility windows defined with product and surfaced in the live XO marketplace.
532 operators
→Haiku · first passSonnet · escalation84.7% badge-ready
Architecture evolution
The pipeline changed as the document problem changed.
Chamomile did not begin as a fashionable RAG system. It began with the simplest pattern that fit the task, then evolved when scale and document variability made that pattern insufficient.
- 01PrototypeWhole contract in context
A single agreement fit in the prompt, making direct extraction the fastest way to validate usefulness.
- 02Portfolio scaleChunking + retrieval
At 1,100+ documents, the system moved to retrieval across the contract corpus, built with engineering.
- 03Document variabilityOCR + page-image parsing
Later generations addressed scanned and non-standard contracts with OCR and stronger API-based multimodal parsing.
Evaluation & fallback
Uncertainty became a routing decision, not a hidden failure.
Manual spot-checks and sales feedback formed the early 2023 evaluation loop before mature LLM-eval tooling.
Use the smaller model for clear policies and spend more reasoning only on genuinely ambiguous cases.
The remaining 15.3% mostly reflected unclear source language; human review was the correct fallback, not a larger-model reflex.
My role
Hands-on applied AI, with the handoff designed in.
- Build
Built the original self-hosted Llama contract pipeline and the later Anthropic-based policy extraction workflow.
- Architect
Combined deterministic extraction, retrieval, model routing, and human fallback according to task risk and cost.
- Collaborate
Worked with engineering on the RAG evolution and with product on customer-readable badge definitions.
- Ship
Moved outputs into Snowflake portfolio analysis and live marketplace trust features.
Public case-study boundary
Enough detail to show the system. No contract content exposed.
This case study describes scale, architecture, evaluation choices, and product outcomes. Proprietary contracts, extracted clauses, prompts, and customer information are intentionally omitted.