Building a response-ready public health data system

Summary

The challenge

Public health authorities at all levels of government rely on data to understand and address public health challenges. Robust systems are critical at every step of the data journey — from patient to public health and back. But outdated, siloed, and fragile systems often disrupt the flow of data, delaying detection and response.

The COVID-19 pandemic exposed these weaknesses. Many state, tribal, local, and territorial (STLT) health departments struggled to manage the overwhelming volume and variety of incoming healthcare data, with outdated tools buckling under pressure and failing to handle non-standard inputs — such as data from questions like, “Have you lost your sense of taste or smell?”

The stakes extend beyond any single outbreak. Without modern, interoperable data systems, the same bottlenecks that hampered the COVID-19 response will recur with the next public health crisis. The U.S. public health system needs to become “response-ready” — capable of proactively detecting and addressing threats rather than scrambling to catch up after they arrive.

The solution

We partnered with CDC and USDS to design, build, and scale open-source tools that help public health departments process and transform data. The work supports CDC’s Public Health Data Strategy and a broader multi-year modernization initiative focused on pandemic readiness and interoperability. We apply agile principles and user-centered design methods — conducting research with STLT staff across the data life cycle — to make sure the tools we build solve real problems in the field.

The design philosophy behind DIBBs is modularity. The tools are composable, much like building blocks that can be stacked in different configurations. Each DIBBs tool handles a specific function — ingesting data, standardizing formats, enriching records, or surfacing information — and agencies can select and combine the ones they need to create a pipeline tailored to their environment. This approach automates manual data processing, improves data quality, and gives public health departments the flexibility to adapt as requirements change.

Different configurations of DIBBs depending on user needs.

Proving the approach required real-world validation, not just a working prototype. We developed and tested DIBBs through a series of pilots with STLT partners, starting with a prototype pipeline with the Virginia Department of Health that processed incoming COVID-19 data faster, created a single source of truth, and eliminated duplicative manual processes. We then expanded to a production pipeline with Los Angeles County’s Department of Public Health that processed and enriched multiple data streams — including electronic case reporting (eCR) data and electronic laboratory reporting (ELR) — improving downstream analysis and case investigation. Alongside the pipelines, we developed and piloted an intuitive viewer interface that surfaces key information from eCR files to make them more useful for monitoring reportable conditions. And to lower the barrier to adoption, we stood up flexible cloud hosting infrastructure that enables STLTs to deploy and scale DIBBs without building their own environments from scratch.

Each pilot fed back into the product, sharpening the tools and validating the modular approach at increasing scale. The result is a growing set of open-source building blocks that agencies can adopt independently — creating modern public health data infrastructure that works across diseases and conditions and delivers timely, actionable data to decision-makers at every level of government.

The results

• VDH increased data throughput from 5,800 to 20,000 HL7 messages per hour and reduced patient record duplication by 19% through a prototype DIBBs pipeline that proved the modular approach
• LAC Hepatitis case investigators receive eCR data 95% faster (from 20 hours to 1 hour) via a production pipeline that processes and enriches multiple data streams, including eCR and ELR data
• eCR processing reduced from 25 clicks to 5 clicks through an intuitive viewer interface piloted with multiple STLTs to make eCR data more useful for monitoring reportable conditions
• Deployed DIBBs to a centrally-hosted CDC cloud environment and began onboarding four STLTs — Alabama, Idaho, Dallas, and Chicago — to pilot the infrastructure
• Developed API documentation enabling STLTs to explore how to integrate DIBBs products into their existing data environments