On August 28, 2015, 18F announced the first round of awardees for the Agile Delivery Services Marketplace (or Agile Blanket Purchase Agreement). For me, it was a bittersweet culmination of nearly nine months of work. On the one hand, I knew I’d just played a central role in architecting and executing a first-of-a-kind acquisition. On the other hand, I knew we’d fallen short of achieving some of our goals, such as maintaining a consistent schedule and running a legitimate best-value procurement.
The past, present, and future of 18F’s Agile Delivery Services Marketplace is a story unto itself. I’ve heard some call it a success (excellent vendor outreach, precedent setter, etc.), and others call it a failure (too many awardees, small order sizes, low order volume, etc.). While the results are no doubt mixed to-date, you can’t overlook the effects: It has influenced the way government thinks about evaluating the technical capabilities of vendors — “Show, don’t tell us.”
As the first agile marketplace of its kind, we relied heavily on 18F’s model during our research and design of the Centers for Medicare and Medicaid Services’ (CMS’) Agile Delivery to Execute Legislative Endeavors (ADELE) contract vehicle. The validated learning that we realized from 18F allowed us to make incremental improvements that have proved critical to ADELE’s ability to deliver agile outcomes at CMS.
Dan Levenson, Former CMS Digital Service Contracting Advisor and Lead Contracting Officer for the Quality Payment Program
Over the past few years, I’ve had time to reflect on my own personal experience running a tech challenge. I’ve also had the opportunity to observe and participate in several others. Given the increasing use of tech challenges in government procurements (for example, the Department of Homeland Security’s Flexible Agile Support for the Homeland and the State of California’s Agile Development Prequalified Vendor Pool), I’d like to share my thoughts on what we, as a community, can do to make even better use of them.
Understand the underlying principle at work
In traditional govtech procurements, vendors write a tome on what they purportedly can do, and whoever’s literature inspires the most confidence often stands a good chance of winning. It’s sort of like commissioning an artist to produce a great work of art based solely on how well their agent says they can paint.
Tech challenges shift the evaluation focus away from being told to being shown. For example, the government could ask vendors to submit an application prototype for evaluation by in-house technical experts. Such an approach is designed to serve as a more precise proxy measurement of a vendor’s overall ability or capacity to deliver the contract successfully.
But tech challenges aren’t the only way to accomplish that. 18F, for example, is beginning to explore evaluating the quality of vendors’ work portfolios — such as product designs and code repositories. This approach has been used for decades in the commercial world. I love the idea, although it may take a few years from now until it’s viable; there haven’t been enough opportunities for vendors working in the govtech space to produce work that’s shareable outside of the contract or firewall.
Once you fully grasp the underlying principle behind the use of tech challenges, there are many creative riffs or alternatives that you can explore. So don’t limit your imagination (or be afraid to use your sound judgement out of fear of protest or violating the Federal Acquisition Regulation). You might just discover something that works even better for you — and others in the procurement community.
Procure with empathy
Most government procurements reflect a fundamental misunderstanding of the professional services business model. That misunderstanding has carried over to tech challenges as well, unfortunately. As a result, I’ve seen several misapplications: a month-long prototype development exercise; a tech challenge exercise plus a traditional narrative response (over 40 pages); and an unpredictable solicitation timeline. These are just different ways of shooting yourself in the foot.
For one, you greatly increase what it costs for vendors to pursue and respond to your procurement. Ultimately, these costs are passed back to government in the form of higher rates, which is bad for taxpayers. Two, you perpetuate the high barriers to entry that government continues to put up, which keep many types of vendors (for example, small or new ones) from participating. And, three, which is related to two, you suboptimize competitive dynamics. Because you make the process so expensive, vendors either self-select out or don’t fully invest in a quality response.
I’ve never heard any government official who transitioned to industry say: “Wow, we didn’t make our procurements unnecessarily burdensome at all. I don’t know what all the fuss is about.” In fact, quite the opposite.
Unfortunately, I can’t flip a switch and make procurement officials more empathetic toward industry. But I can create a 2x2:
With greater empathy, I speculate you’d see government officials design more tech challenges that feature:
- Smart tests that only take a few days for a vendor to respond to and yield the minimum amount of information necessary for the government to conduct a reliable evaluation. (I’ve referred to this elsewhere as the principle of minimally sufficient evaluation criteria over completely exhaustive evaluation criteria.)
- Remunerating vendors for participating in tech challenges that legitimately require significant investment of time and effort, such as what the Health and Human Services’ Buyers Club did.
- Reliable timelines so vendors know when to be prepared, how to be prepared, and for how long in advance.
- Extensive upfront input from industry that shapes how the tech challenge is designed, with the aim of maximizing vendor participation and thus competition.
Don’t waste people’s efforts
Vendors invest a tremendous amount of effort in responding to tech challenges.
The technical artifacts they produce — designs, code, algorithms, etc. — are potentially valuable and should be put to good use, not wasted. Asking vendors to create a conference room scheduler or parking lot reservation prototype isn’t furthering your mission or humanity whatsoever. Instead, use a tech challenge as a means to test out or learn more about an idea that’s relevant and important to your mission. You could emulate what the Veterans Affairs Digital Service team did by having vendors implement user stories and push code to an actual production system. Or you could ask Code for America for ideas for simple apps, web services, or prototypes that might help civic communities regardless of outcome. There are many useful possibilities that can come out of a tech challenge. At the very least, require vendors to commit their responses to the public domain so others (such as an aspiring coder) can either reuse or learn from the artifacts.
Feedback is a must. It’s the least you can do to acknowledge people’s efforts. More importantly, vendors can’t learn and improve without meaningful feedback and evaluation, which is a detriment to everyone.
Continue to experiment and learn
Tech challenges are a vast improvement over traditional approaches. However, we can’t be blind to the possibility that there might be even more effective techniques. Personally, I’d like to encourage government procurement officials to run experiments in the following two areas:
Put greater emphasis on evaluating the technical talent of the vendor’s team. There’s simply no substitute for great talent. The more emphasis that the government places on talent, the more vendors will invest in that area, even if that means fundamental changes to their talent models.
As a thought exercise, imagine that during your next procurement you completely eliminate all evaluation factors except for personnel quality and price. What effect do you think that would have on how vendors prepare a response? Personally, I’d rather have vendors compete primarily on the basis of talent than anything else. To quote Red Adair: “If you think it’s expensive to hire a professional, wait until you hire an amateur.”
So how do you do this? I have some thoughts, but I’ll defer those to (perhaps) a future post. However, I’ll say that I think the emergence of lightweight, skills-based hiring techniques could be translated for use in government procurements. Well-designed oral presentations that function more as interviews could also work, as evidenced by how the General Services Administration ran the recent Key Initiatives Centers of Excellence (Phase 1) procurement.
Find ways to push work to vendors based on actual performance. Tech challenges, work portfolios, oral presentations, past performance information, and so on are all examples of proxy measurements. The best form of measurement, however, is direct measurement. And the only way to do that is through measuring exactly the thing that you’re looking to measure — actual performance of the work (for example, user satisfaction, code quality, or cross-team collaboration).
The reality is that you can’t design an error-proof vendor selection process using proxy measurements. Sometimes a vendor doesn’t work out for whatever reason. What you can do, however, is design an error-tolerant process that gives you the option of seamlessly switching or shifting work based on who is and isn’t performing well. One way to accomplish that is to structure your procurement to include more than one vendor (à la modular contracting), measure their performance, and shift work around accordingly, if necessary.
There are few reasons a $100M or even a $2M contract should be awarded to a single vendor. Sure, it might seem like more work to administer the contract, but the benefits far outweigh costs. Besides, you can find ways to offset the overall effort involved by running a lighter upfront pre-award and award process, for example.
Looking to pioneer the next procurement breakthrough?
We can’t make government IT better without buying better. The Skylight team has pioneered several acquisition breakthroughs, including Microconsulting, 18F’s Agile Delivery Services Marketplace, and 18F’s Agile Acquisition Framework. We’d love to work with you on making the next one happen!
Special thanks to Dan Levenson for his input on this article. Dan architected and ran the Centers for Medicare and Medicaid Services’ Agile Delivery to Execute Legislative Endeavors contract vehicle, and is currently the Vice President of Digital Services at Agile Six.