Scaling the Use of Real-Time Open Data
October 04, 2021 by Steve Goll
Every day, the life and well-being of a community plays out through the public's interactions with local government.
Permit applications, 911 calls, arrests, pothole reports, sales taxes, COVID-19 cases. Like plot characters coming together, the data behind these daily interactions tell a story about the social and economic health of the community.
Although every community across the country generates similar types of data, the stories they reveal vary. What would government leaders and citizens discover if they could connect and compare all the stories in real time, then go behind the scenes to analyze the underlying data? Would they gain new insights for decision-making?
Seeing the Big Picture With Open Data
All data tells a story. But like the parable of the blind men and the elephant, laying your hands on a small set of data will, at best, lead to a limited understanding, and, at worst, cause you to miss the big picture.
To make fully informed policy and funding decisions, leaders in states, counties, and federal agencies need access to a broad range of timely data from the local level. Local governments, in turn, need access to data from the state and federal levels for their own decision-making purposes.
Yet, government data lives in many different locations, systems, and formats — from a spreadsheet on someone’s desktop to a 40-year-old legacy database system in the federal government. It can be hard to bring this interrelated data together to look for big-picture cause-and-effect insights. Helping to overcome this challenge are three concepts: open data, data consolidation, and data federation.
The concept of open data is grounded in the principle that government information should be easily found, freely accessible, and readily useable. Data can be made open by providing it in useful formats online, perhaps accessible via application programming interfaces (APIs) which allow two software applications to communicate with one another.
Data consolidation and data federation involve accessing data from multiple — oftentimes dissimilar — sources and pushing it back out in a unified form. Think of viewing a single table or visualization of data coming from a desktop spreadsheet from a state agency in Idaho and a mainframe computer from a federal agency in Washington, D.C.
Data consolidation and data federation differ in where data is stored. Data consolidation brings together data from disparate source systems in a data store such as a data warehouse. Whereas data federation lets data stay in the source system but creates unified access through virtualization.
Together, open data, data consolidation, and data federation enable the flow of information across all levels of government.
The Advantage of Federated Data
In terms of timeliness and infrastructure, data federation has an edge over data consolidation.
With federation, the moment data is updated at its source, it becomes available for viewing along with data from other sources. The end user sees data virtually as it exists in its home database, wherever that may be.
With consolidation, data must be pulled from sources and combined in a separate database before being made viewable. Depending on the frequency of the consolidation, the user may be seeing information that is out of date by minutes, days, or perhaps longer.
Data federation removes any doubt about whether the latest data from the source is being viewed. That immediacy becomes critical when real-time decisions must be made.
For example, COVID-19 cases, hospitalizations, deaths, and vaccinations are tracked at the county level. Using data federation enables state-level decision-makers to quickly adjust resource allocations as the situation in counties changes in real time. Citizens, in turn, can make informed decisions to protect their health in the moment.
Scaling the Use of Open Data
Increasing numbers of state and local government leaders are turning to federation to scale the amount of data they can bring to decision-making.
- The State of Colorado: Colorado’s Information Marketplace uses federated data to inform the public about regionally relevant data. For example, Fort Collins’ CARES Act funding data is accessible from the state’s data portal, so other cities can easily find and access it to compare their own region’s utilization of recovery funding.
- The State of Texas: The Texas Open Data Portal features datasets consolidated from 23 state agencies as well as data federated from the cities of Austin and Dallas. For constituents, the portal provides access to open data on demand which reduces time spent on public information requests.
- The State of Maryland: Maryland’s Open Data Portal uses catalog federation to make local data discoverable and accessible from open data programs in counties like Montgomery County, Howard County, and Queen Anne's County.
The past year of the pandemic has clearly shown the value of quickly sharing data from multiple sources. With open data systems in place, communities are basing policies and decisions upon broadly sourced authoritative data, not just isolated tribal knowledge.
Moreover, today’s cloud-based API-enabled open data platforms, such as those from Tyler Technologies, have evolved in ways that can quickly connect data stories from vastly different sources into a unified view right from the start.
Without connections between multiple sources of information, full stories remain hidden from view. Local governments and citizens benefit equally when open data shows them the big picture of their community.