Case Studies
Three engagements, each with its model card, including the limitations. Two of them describe a model that lost to a human at something.
Manufacturing
Demand forecasting against a baseline that refused to lose
The situation
A components manufacturer forecast demand with a spreadsheet and an experienced planner. They wanted a model. We warned them the planner would be hard to beat.
What we found
She was. Her implicit seasonal adjustment scored a MAPE of 11.9 percent, and our first three model iterations were worse. The gain only came after we rebuilt the feature pipeline to include promotional calendars and a public holiday table, which the planner had been carrying in her head for eleven years.
What we did
We ended at 8.4 percent MAPE with a confidence interval of 7.6 to 9.2, evaluated on a held out final year the model never saw during development. Crucially, the model performs worse than the planner on newly introduced parts, which is documented in the model card. Those are still routed to her.
The outcome
The forecast is now automated for established parts and escalated to a human for new ones. That split was the actual deliverable. A model that had claimed to beat her everywhere would have been a worse outcome and, on inspection, would have been leakage.
Industrial
A vision model that scored 97 percent, then 61 percent on the shop floor
The situation
A manufacturer had bought a defect detection model that scored 97 percent on the vendor's benchmark. In their plant, operators stopped trusting it within three weeks.
What we found
Two failures compounded. The training images were captured under even studio lighting, and the plant runs a mix of daylight and sodium lamps depending on the hour. Worse, the annotations disagreed with each other: two annotators labelling the same part agreed only 71 percent of the time, so the model had been trained on a moving target.
What we did
We rebuilt the annotation protocol first, with a written defect taxonomy and adjudication of disagreements, lifting inter annotator agreement to 0.93. Only then did we retrain, augmenting for illumination, and deploy to an edge device at 34 milliseconds per part.
The outcome
Recall reached 0.94 with an interval of 0.91 to 0.96 on parts photographed in the plant across a full shift cycle. The model card records that performance below 300 lux is untested, and the line now refuses to run inspection when the light meter drops. Fixing the labels, not the architecture, produced almost all of the gain.
Financial Services
Document extraction, and a hallucination rate we insisted on measuring
The situation
A lender wanted an assistant to read loan files. The proposal on the table was to fine tune a large language model on their document corpus.
What we found
That approach would have taught the model the shape of their documents without teaching it the contents, and it would have invented plausible field values when uncertain. Since a fabricated loan amount is materially worse than no answer, we built retrieval with mandatory source citation and an explicit abstention path instead.
What we did
Before deployment we evaluated against two sets: documents with known field values, and documents where the field was genuinely absent. The second set is the one that matters, and it is the one nobody runs. It measures how often the system invents an answer rather than admitting it has none.
The outcome
Field accuracy reached 98.1 percent with an interval of 97.4 to 98.6. Human entry remains more accurate at 99.3 percent, and we said so. The system's value is speed, not accuracy: four seconds against six minutes, with every extracted field carrying a citation and low confidence fields abstaining to a reviewer. The lender deployed it as a first pass, with human confirmation retained.
Every metric carries its interval
Each figure was measured once, on a test set held out before modelling began. Where a human outperforms the model, we have said so. Client names are withheld under confidentiality, and any number here can be substantiated on request.
How we evaluateSend us your data problem. We will tell you whether it is one.
A working data scientist reads every enquiry. You will get an honest read on whether your data can support the model you have in mind, before anyone quotes you a number.