Thousands of fibre jobs, tracked in spreadsheets and chat

The situation

Fibre rollout is a volume business with a paperwork problem. Each site needs a survey, then construction, each with its own evidence requirements, each done by a different crew, and all of it has to reconcile against what the customer believes was delivered.

This operation ran it on spreadsheets and messaging apps. Work orders were assigned verbally or in chat, evidence came back as photos in a thread, and month-end reconciliation meant cross-checking two systems by hand. Disagreements with the customer were settled by whoever had kept better notes.

Why nobody had already fixed it

Not because the problem was hard to describe. Every person in that operation could describe it.

The work order tools that exist are built either for a telco’s own field force or for a generic service desk, and a rollout contractor is neither: the evidence rules come from the customer’s contract, not the software vendor’s idea of a job. Configuring a platform to match takes a consultant who understands both, and the licence cost lands before anyone has proved it fits. The quotes that came back were for a programme, not a tool.

So the spreadsheet stayed, because it was the only thing that would bend.

Why the schema came first

Sixty tables and 43 migrations before there was much of an interface. Deliberate.

Survey and construction look similar enough that the tempting move is one generic “work order” with a type flag and conditional logic sprinkled through the code. That holds until the evidence requirements diverge — which they do immediately — and then the rules live in fifteen places and nobody can say with confidence what the system will accept.

Instead: two distinct workflows over a shared spine, with the rules expressed in the data model. It costs more up front, and it is why a six-week build did not need rewriting in week seven.

work order board · phase B + C / this week

order	phase	task	crew	evidence	sla
WO-24-0881	B	Survey — riser access	Crew 04	passed	on time
WO-24-0884	C	Construction — duct	Crew 11	review	on time
WO-24-0887	C	Construction — splice	Crew 04	rejected photo unreadable	at risk
WO-24-0890	B	Survey — façade route	unassigned	none	at risk
WO-24-0893	C	Construction — drop cable	Crew 07	passed	on time

412 open 2 at risk 1 awaiting crew p95 < 1s at 300 users

The part that mattered most happened after it worked

The system worked. That is not the same as knowing it will keep working.

A rollout programme accumulates history indefinitely, so we seeded a synthetic history of roughly half a million orders and 1.6 million tasks, and ran concurrent mixed load against it.

It failed. A p95 of around five seconds at 200 concurrent users and nearly seven at 300, against a two-second target.

The cause was not the database. Concurrent writes were clearing in-flight summary calculations, so several requests each started the same expensive aggregation instead of sharing one result. A classic stampede — invisible at single-user scale, obvious at 300.

The fix keeps in-flight work shared and marks its result expired if a write lands mid-calculation, so the next request rebuilds rather than serving something stale. Historical aggregates survive unrelated writes, work-order writes invalidate them, and the outer summary can never outlive the snapshot inside it.

After that: roughly half a second at 200 users and under a second at 300, with zero request errors and comfortable headroom against the target.

What it runs on

Postgres through Drizzle, with the migrations checked in — 43 of them, which is what a data model that was allowed to change honestly looks like. Clerk for identity. Vercel Blob holds the evidence photos. Leaflet with clustering draws several thousand sites without collapsing. Email goes out through Resend, and browser push through web-push, which together generated 15,000 scheduled notifications in under sixteen seconds while 200 users were live on the system.

The tests run against a real embedded Postgres rather than mocks. Mocked database tests pass while the constraint you actually rely on does not exist.

It says when it is unwell

A rollout system that silently stops reconciling is dangerous rather than merely broken: the numbers keep looking plausible and the disagreement with the customer arrives a month later.

So the system reports on itself — throughput, queue depth and the health of the scheduled work — on a panel the operations lead can read without asking anyone. Evidence that fails review is a visible count, not a silent rejection, and the crew that submitted it is told rather than left to discover it at month end.

Reporting the failing run, not just the passing one

The internal test notes record the failed run in full, with the explicit observation that a latency failure is still a failure even when the test process exits cleanly — and that a favourable comparison between two differently-configured runs is encouraging evidence rather than a controlled measurement.

That is the standard the documentation is written to. A performance report that only records the good run is worth nothing to whoever inherits the system.

The handover is part of the work

Current state, decisions and business rules, workflow and status model, architecture, operations and deployment, testing, a prioritised backlog, and a continuation brief.

Every outside build raises the same question — what happens once we are no longer on it. The answer has to be a real one, written down, before the client thinks to ask.

What actually changed

Before, the answer to “is this job finished, and can we bill it” lived in a photo thread and somebody’s memory, and the customer’s answer was often different.

Now it is a record with evidence attached and a review state, and both sides are looking at the same thing. Month-end stopped being a negotiation about whose notes were better.