Identifying valuable datasets for turning into APIs

Summary

The challenge

The VA has produced millions of public datasets over the years in support of the veteran community. These resources tell a story about how much demand there is for accessing them. The VA’s API Platform Management team needed that insight in order to help formulate a near-term roadmap of valuable APIs to deliver.

The solution

To address this challenge, we conducted an API landscape analysis that involved:

• Developing and running a spidering script that crawled all the VA web properties and performed a weighted semantic analysis on resources such XML files, CSV files, HTML tables, HTML forms, XLS/XLSX files, and other machine-readable formats
• Conducting user research with veterans and their supporters to learn more about what types of data resources would be most valuable to them if made more available and accessible on the web, mobile devices, and other platforms
• Analyzing and synthesizing the results of the spidering process and the user research
• Preparing a report that detailed our methodology, analysis results, and recommendations for resources to prioritize as APIs

In our report, we also made recommendations for how the VA can improve the quality of their analytical information so that resource ranking can be done more efficiently and effectively in the future.

The results

• Delivered the project under the federal procurement micropurchase threshold of $10,000
• Crawled over 1 million URLs and evaluated over 20,000 data files
• Informed the VA’s near-term API roadmap, which incorporated many of the APIs we recommended for delivery
• Published the report in the open and with no copyright, making it free to anyone who wants to learn from and reuse the content