Phases of an agile procurement

The phases of an agile procurement are flexible, but they typically follow this pattern when starting the procurement process for a new product:

• Defining the goals and objectives of the product
• Understanding the skills required for development
• Drafting the Statement of Objectives (versus the Statement of Work)
• Planning the proposal evaluation process
• Evaluating proposals
• Meeting with unsuccessful vendors

Another approach would be to use these key questions to guide the process:

• What is the scope of the project? What are the key deliverables?
• What are the objectives and milestones? How frequent are they?
• What are the performance metrics defined in the contract? (For example, response time, system uptime, time period to address priority issues)

Defining the goals and objectives

Before thinking about a procurement approach, the government must first understand the needs, goals, mission, and rough scope of the product that needs to be built. Representatives of research, product, engineering, and procurement will need to participate in defining the project plan. This collaborative effort ensures that each community of practice (for example, research, design, engineering) has full context before making recommendations on the specific areas where the community will support the product.

Aligning on goals and strategy

When products involve multiple parties or stakeholders, it’s a good idea to have working sessions early on in the process to ground all parties on the vision, mission, and goals of the product. By getting everyone on the same page at the beginning of the procurement process, the team will avoid costly assumptions down the road.

These working sessions will align stakeholders on:

• What’s being built
• Why it’s being built
• Who it’s being built for

Frameworks such as CATWOE can help identify areas of the development process that could be a source of conflict between stakeholders. Finding these areas of possible conflict early is much like catching software bugs early in development — the cost of fixing is exponentially more the longer a source of conflict isn’t addressed.

Once the team has a good understanding of the goals and objectives of the product or service, the procurement team can tailor a strategy to the product. That might include evaluating the necessity of custom development versus off-the-shelf solutions or determining the performance period timeline to gain a better understanding of the value that’s delivered against the calendar.

For particularly complex procurements, having a RACI helps the whole team have a clear understanding of roles and responsibilities.

Understanding the skills required for delivery

Once the team has a common understanding of the overall picture, the team needs to start thinking about how to build the product. This is where much of the specific procurement work begins.

Research

With all products, there’s a variety of possible solutions. In this phase, the team will explore what the needs are for the product and look into answering basic questions such as:

• Who are the end users of the system and what are their expectations?
• Does there need to be custom development?
• Is there a comparable service already available? For example, we wouldn’t build a point-of-sale system by hand if a service like Square would meet the needs.
• Are there development teams that specialize in the type of development needed?

The basic answers to these questions will help guide any market research needed.

Market research

At the beginning of the procurement process for digital services, it can be difficult to have a good understanding of the large bucket of product development requirements. This is a great time to solicit feedback from the vendor community. When asking for vendor input, the government should be specific about the product goals. This will help the vendor community provide detailed feedback about how they would approach — and ultimately solve — some of the problems that the government will face when building out digital services.

While it’s valuable to see how industry would approach solutions, it’s more important to see what skillsets they would require for their solutions. Ultimately, this is what the government should target for their procurement — the communities of practice that are required to deliver the product.

Engaging with vendors can be as formal as a request for information (RFI) or as informal as grabbing a cup of coffee and having a conversation. Vendors will be happy to provide information no matter the method. Typically, the more informal the engagement, the faster the turnaround is for answering requirement questions. Also, the informal nature can be helpful in putting a face to government employees and building stronger relationships with vendors.

Outreach

In an effort to continue to build relationships with the vendor community, there are a variety of ways the government can engage. The stronger the government’s network is with possible vendors, the more options it’ll have when it comes to procurements. This diversification is another way to help de-risk the government as a whole.

Reaching out to industry prior to any procurement is perfectly acceptable and encouraged. Not only does this give the government a better idea of the skillsets that are available on the market to accomplish any goals that it might have, it gives the industry an idea on what the government values, where it might be going in terms of digital transformation, and how to best position itself to meet the needs of the government.

Industry day

Industry days can be a great way to bring together the vendor community for a day or two to share trends, products, services, and overall ideas. They provide interested vendors the opportunity to meet the government’s program and procurement officials. In addition, these types of events provide the small business community the opportunity to engage with agency prime contractors, talk with government small business representatives, network with other vendors, and increase their awareness of the contracting opportunities available within different agencies. Industry days should be open to all industries and business classifications. Attendees will market their products/services, attend business workshops, and participate in one-on-one matchmaking sessions.

Attracting a diverse set of vendors (non-traditional vendors)

The small business community can often be overlooked due to possible lack of past experience within the government. These vendors are an incredible source of information and can provide a set of fresh eyes on development as well as bringing current industry best practices into the government. Many of these vendors are experts in providing and delivering digital services.

Drafting a Statement of Objectives (versus a Statement of Work)

All procurement strategies require a main description of the work that’s being procured. This description is typically documented in either a Statement of Work (SOW) or a Statement of Objectives (SOO). This is where the procurement strategy should map to the development methodology.

For an agile approach, the feedback and revision loop needs to be quick so that the team can regularly assess value and determine what needs to be built next. This feedback loop or reevaluating is in conflict with a traditional SOW because the team is constantly reevaluating what’s next, versus sticking with what was predicted up front.

Statement of Work (SOW)

Traditionally, SOWs dictate what’s going to be built and how it’s going to be built. SOWs use language that’s explicitly binding, such as: “Vendor SHALL produce a white paper on progress on the last day of every month.”

SOWs are written under the assumption that 100% (or near 100%) of requirements have been gathered and will not change. While this approach is appropriate for goods and services that are somewhat static and have been procured by the government for years, this approach is far too restrictive when building a digital product. The most advanced technical companies such as Google, Facebook, and Apple don’t attempt to build products with a 100% understanding of the requirements up front.

Statement of Objectives (SOO)

A SOO is a much better fit for agile development. Shifting from an SOW to an SOO is a shift from a document explaining, “this is what you need to do” to one that says “here are the objectives for this product.” The goal of the procurement is to bring in experts to help determine what needs to be built; this includes understanding the needs of the users through research and discovery, as well as the best way to go about building it in order to meet the objectives.

When thinking about an SOO, the government should focus on defining the mission, vision, and goals of the product or service. Procurement with an SOO provides more flexibility for making changes to the product during the development lifecycle. While changes to the product will be frequent, the goals and objectives of the product will not change. By focusing the procurement towards an SOO, the government avoids having to constantly modify existing contracts as it gains new information and insights.

For an example of an SOO, see Appendix G.

Planning the proposal evaluation process

As we’ve discussed, agile procurement focuses on acquiring the skillsets needed to meet the product’s objectives. Therefore, the proposal process needs to provide the review team with information that allows them to compare and contrast the skills of the vendor pool. This section describes the tools for getting the right types of information from vendors.

Design challenge

This is the classic “show, don’t tell” part of the procurement. The design challenge is arguably the single most important part of the procurement evaluation process. Because each procurement is different, the challenges need to be customized to align with the key components of the SOO.

To get the most accurate insight into each vendor’s ability to successfully execute against the product’s objectives, craft a challenge that closely matches the type of work that will be involved in the product development. If the procurement is for a very technical product or service, the challenge should be technical. For example, it could be building a data pipeline that connects to a variety of government legacy databases. Alternatively, if the procurement will be product management heavy, the challenge could include building out a product roadmap with prioritized milestones.

The government will need to come up with a realistic scenario that vendors can respond to with either working software, design artifacts (for example, wireframes), product management approaches (for example, product milestones and backlog of prioritized work), or any other range of artifacts that will be essential for the specific procurement.

For an example of a design challenge, see Appendix E.

Quality Management Plan (QMP)

A QMP ensures that the government and vendor have a clear, common understanding of expectations for delivering product artifacts (software, wireframes, user research findings, etc). It includes:

• Specifics about how the government will evaluate the vendor’s performance metrics (called a Quality Assurance Surveillance Plan)
• Roles and responsibilities for enforcing those metrics
• The vendor’s planned response if metrics aren’t met (called a Quality Control Plan) — for example, there could be financial impacts tied to some of the performance metrics

Much of the QMP can be captured in a single table. The vendor will have to come up with most — if not all — of the performance metrics they’re to be measured on.

For a Quality Management Plan template, refer to Appendix C.

Quality Assurance Surveillance Plan (QASP)

A QASP consists of a roles and responsibilities section that describes the formal relationship between the vendor and the government’s contracting specialist/officer. It’ll also describe the performance metrics the vendor will adhere to based on their proposal. The performance metrics will describe how the government can verify each metric.

While the roles and responsibilities included in a QASP tend to be fairly static across modular procurements, the performance metrics should both be generic and also tailored to the product that’s being delivered by the vendor. A good QASP will have objective-based performance metrics. These performance metrics can come in many forms, including progress towards a desired result, the dependability of something, or the security of a system.

The measurement of these performance metrics is important, and it should be easy. An effective performance metric avoids generalized goals like, “improve user research.” Instead, an effective metric should be based on a solid, focused goal that can produce qualitative and quantitative measurements. A good example would be: “All web pages shall load in under 3 seconds measured at the 95th and 99th percentile.”

Quality Control Plan (QCP)

A QCP is the other side to the QASP performance metric coin. It describes how the vendor plans to respond to each metric that isn’t being met. Like the QASP, the QCP should include the concrete actions that’ll be taken under specific circumstances.

Additional plays

Page limits

Just as gasses expand to fill the container they’re in, proposals expand to reach the maximum page limit. Boilerplate text doesn’t add value to proposals, but the time it takes to read it is costly when evaluating proposals. Imposing strict page limits forces vendors to prioritize the information they deem important, and it saves the government hours on potentially zero-value content. A proposal limit of 10 pages is often sufficient.

Evaluating proposals

Once the solicitation has been posted, there’ll be a window when vendors are putting together their responses. The product team can use this time to put together the team that will go through all of the proposals submitted.

Depending on the urgency of the procurement and the number of proposals received, the government has a couple of options for setting up the review process.

Group review

If there’s a large number of proposals, evaluators could review the proposals in real-time as a group. This approach helps evaluators keep pace with each other, but it also introduces the possibility for group-think bias. Some people will read faster than others, which can cause others to rush in their personal analysis. And the logistics of aligning people’s schedules for large blocks of time can add to the total time it takes to review proposals.

Individual review

The evaluating group can agree to read through proposals on their own time and schedule regular check-ins. The team will use these check-ins to quickly provide thoughts and feedback on the proposals they’ve read. This model will typically be faster than reviewing as a group, because the feedback sessions are much shorter than the time necessary to do the evaluation.

The check-ins can be remote/call-in meetings if that’s more practical. While it’s always nice to see people face-to-face, it typically isn’t required. This also speeds up the process of evaluation.

Technical Evaluation Panel (TEP)

In the federal procurement process, it’s common to have a technical evaluation panel (TEP) select the winning proposal.

A TEP is a group of subject matter experts (SMEs) from the agency/government that reviews each proposal’s non-price merits and ultimately makes a selection and recommendation to the acquisition office for awarding the contract. Following receipt of quotes, the TEP will perform an analysis on a factor-by-factor basis, noting the positive and negative aspects of each proposal, and assigning each proposal a score based on an agreed upon method.

SMEs are experts in research, design, engineering, or any other subject. These individuals may have years of experience in private industry or be active individual contributors for their respective discipline on government projects. Make sure that the designated SMEs are actually SMEs. They’ll have the ability to provide in-depth analysis on design challenge submissions and be able to back that analysis with industry research and best practices.

The TEP team, like the team for the procurement process mentioned previously, needs to be a cross-functional team comprised of different communities of practices that are going to be necessary in delivering a successful product. This team is arguably one of the most important teams involved in the process of selecting the correct vendor for the contract. This team of experts will evaluate, document, and ultimately make recommendations on who they think is best suited to successfully deliver the product in question.

Down-select

The down-select process is a very powerful tool in the agency’s acquisition toolbelt that saves time for both the government and vendors and dramatically decreases the probability of protest coming from a vendor. The down-select can be used for individual procurements as well as for developing a contracting vehicle such as a blanket purchase agreement (BPA).

The down-select is a phase in the procurement lifecycle that requires the evaluation of some information (for example, past experience, technical skillsets) to determine the likelihood of a vendor’s success in winning the contract. The down-select can be either mandatory or voluntary.

• In a mandatory down-select process, the government notifies a subsection of the applicant pool that they’re not going to be considered and are being removed from the procurement process.
• In a voluntary down-select process, the government notifies a subsection of the pool that they should consider not proceeding through the rest of the procurement process based on the TEP’s findings.

When the government gives transparent feedback to vendors throughout the process — including when some proposals have a low probability of winning — this allows vendors to decide whether they want to put more time and resources into the procurement process. Without this feedback, vendors may feel like they might have wasted time and be understandably upset.

The government can benefit through the terms/language in which it evaluates whether vendors are down-selected. Appendix D provides an example of how the government can be transparent about the down-select criteria.

Selection and award

At the end of reviewing a proposal and design challenge, if included, the TEP needs to reach a rough consensus on the best submission that will have the highest probability of success in delivering the product or service for the government.

Meeting with unsuccessful vendors

This is probably one of the most important, yet most awkward phases of the post-award procurement process. The procurement team should schedule face-to-face meetings (called retrospectives) with each of the vendors that didn’t win. The vendors can choose to take the meeting or ignore it.

The overall goal of offering these meetings is to make the next procurement process even more competitive. During a retrospective, the government provides the vendor with feedback on how their proposals and design challenge submissions can be improved for future submissions. Many times, the vendor will also provide feedback to the government about the procurement process, which is extremely valuable to help improve future procurements.

Finally, procurement retrospectives reduce the likelihood that vendors will protest the award. Vendors make significant investments in bids, and they want to know how they can increase their chances of winning next time. In the absence of feedback, they can only guess at where they fell short. By putting in the time and effort to communicate with unsuccessful vendors, the government builds a vendor pool that better understands its needs and expectations.