On 4 September 2026, Ben Tossell published the findings of Session #5 of Ben's Bites, documenting a massive automated data engineering experiment executed through autonomous multi-agent pipelines. The project successfully extracted, cleaned and structured 107 million rows of municipal expenditure data from local authorities across the United Kingdom. Using an agentic pipeline built around ChatGPT Codex, the system ran largely autonomously for approximately 48 hours to complete the end-to-end transformation. This implementation highlights how generative models can move beyond isolated code snippets to manage prolonged, heavy-duty operational workloads.
The target records comprised official spending disclosures for purchases over 500 pounds, which public bodies in the United Kingdom are required to publish. Tossell directed the system to ingest records across 319 of the country's 339 local councils, dealing with a fragmented landscape of reporting conventions, varying column structures, and inconsistent naming practices. Harmonizing this breadth of municipal accounting data usually represents a tedious data cleaning bottleneck for software engineers and public finance analysts alike. By taking on 319 distinct municipal jurisdictions simultaneously, the experiment set out to test whether autonomous language models could overcome real-world data entropy at scale.
Rather than relying on a monolithic prompt or a single linear script, the execution framework dynamically distributed tasks across 656 subagents functioning within 552 coordinated threads. These subagents operated with significant autonomy, dividing labor between web scraping, raw table extraction, data cleaning, vendor deduplication, and automated consistency verification. When encountering spelling variations or contradictory vendor identities across council records, the subagents determined standardization rules without requiring continuous human oversight. This parallel threading allowed the agent swarm to maintain continuous throughput across hundreds of different sources simultaneously.
Sustaining such a large-scale swarm over a 48-hour continuous runtime came with substantial compute consumption, totaling 8.2 billion tokens across all model calls. Once the subagents verified and cleaned the municipal transaction lines, the pipeline ingested the full structured dataset directly into a local DuckDB database instance. DuckDB was selected to provide high-performance local analytical processing capable of querying the hundred-million-row scale efficiently on developer hardware. The database backend was subsequently connected to a map-based user interface, enabling users to explore local spending patterns through geographical navigation.
Tossell's deployment signals an evolutionary step in autonomous software development, shifting the conversation from synthetic agent benchmarks to real-world data transformation infrastructure. Previous public agent demonstrations often ran into failure loops or required constant corrective interventions when dealing with sprawling web inputs. In this instance, coordinating hundreds of subagents over billions of tokens demonstrated that autonomous agent swarms can act as resilient data pipelines for complex tabular information. The project provides a concrete reference point for how engineering teams might approach large-scale legacy data consolidation using autonomous coding models.