Case study
Claims stopped going missing overnight.
A US healthtech company of about 25 staff, loading claims through a nightly batch that lost roughly 40 records a week. Three engineers rebuilt ingestion on Kafka and added a reconciliation report; dropped records held at zero across 11 months. The client was then acquired, and the engagement closed early.
The situation
Claims arrived from payer systems as files and were loaded by a job that ran once a night. If a file was malformed, or landed after the job had started, or the job failed halfway through, the records in it were not loaded and nothing anywhere said so. The rate was about 40 records a week.
Finding them meant someone comparing totals by hand the next morning, and the gap between a claim going missing and anyone noticing was a day at best. The client's engineering team was four people, all of them on the product. Nobody owned ingestion.
What we did
Ingestion moved onto Kafka. Each file is read once on arrival and its records published to a topic; the Java/Spring consumers write into PostgreSQL and are idempotent, so a replayed file produces no duplicates. A record that cannot be written is parked with the reason attached instead of disappearing.
The second half of the work was the reconciliation report: a daily count of what each payer sent against what landed in PostgreSQL, with every difference itemised. The client's operations team asked for it in the first scoping call, and it is the reason the zero below is a measured number rather than our assertion. Three engineers on AWS, 19 months.
What changed
Dropped records went from about 40 a week to zero, and stayed at zero across 11 months of daily reconciliation. The count was read by the client's operations team, not by us. Claims land as they arrive rather than waiting for the night's run, and a failure now names the record and the payer instead of showing up as a total that does not match.
The change the client's operations team cared about was not throughput. It was that a record now fails loudly: the 40 a week had been going missing without producing an alert, an error, or a line in any log that someone read.
How it ended
The client was acquired. Claims processing moved onto the acquirer's platform and the pipeline was switched off with it. The engagement closed early, with months of scoped work not done.
We had a few weeks' notice. We wrote down how the system worked, handed that to the acquirer's engineers, answered their questions for a fortnight, and stopped on the agreed date. There is no second act to this one: 19 months of work is not running anywhere now, and the figures above describe a system that no longer exists.
Ingestion is Java/Spring on AWS, with Kafka in front of PostgreSQL. Four of the ten technologies we staff against; no part of this engagement sat outside them. The nightly batch it replaced was a scheduled job writing into the same PostgreSQL database, which is why the cut-over could be done payer by payer.
Ingestion, before and after
A nightly batch nobody owns?
Three engineers, 19 months, Kafka in front of PostgreSQL. The daily reconciliation report that made the zero measurable was asked for in the first scoping call, not bolted on at the end.