Specifications or requirements, call them what you will, should be split into two distinct sections:
1. A clear description of the environment and your desired end state
This is a detailed description of what you need the supplier to do and anything you see as an impediment to them doing it. It can be broad and generalised where the requirements can be somewhat loose where you are relying on the market to deliver a solution, or targeted and specific where the requirements are known and you just need a supplier to deliver it. The intent of this section is to give the vendor as much information as possible to understand what you need done and the environment in which they will do it.
2. A set of very specific questions targeted at getting measurable answers back
This is the section that you use to help you decide which supplier is the right one for the job. In this case the questions need to be specific and targeted with a predictable and consistent outcome. They aren’t aimed at informing the supplier, instead they should guide the supplier on how to respond and aim to allow the evaluator in selecting the best offer.
It is this second part of the specification, often referred to as evaluation criteria, that I am going to discuss here.
Objective versus subjective
Objective evaluation uses criteria that is measurable to determine an outcome, e.g., fencing must be between 2′ 6″ and 3′ 5″ in height. Subjective on the other hand, relies on opinion, e.g., the design must be pleasing to the eye. Clearly in this example, it is easy to have 3 people agree that a measurement is between two required dimensions, but it can be virtually impossible to get everyone to agree on what is pleasing to the eye. With all procurement evaluation we should be targeting an objective assessment of the responses and minimising subjectivity. This is especially important in Government organisations where you are spending public monies and need to be able to defend any decisions potentially in a public inquest, but even in a private sector setting, the use of objective evaluation will improve the quality and consistency of the decisions being made and potentially keep shareholders at bay. Don’t forget, you are always spending someone else’s money, so you should always make sure your decisions are good and you can effectively demonstrate how those decisions were reached.
Further to enabling the evaluation team to select the right supplier, a good specification also allows the vendors to have some control over how they will be evaluated. It can also trigger vendors to recognise if they are incapable of delivering the requirement, and so, not respond, saving time on both sides from creating and reviewing poor quality proposals.
What do I want from the suppliers?
The question we have to ask ourselves is “what do we want from the offerors”? What evidence do I need to be certain their response is a true measure of their capability and aligned with my requirements? How will we differentiate between suppliers? This gets more challenging the greater the competition is in the market as the separation between suppliers gets more difficult to identify. The questions need to be designed to extract exactly the responses you need to identify this information.
You should never leave your vendors in doubt as to what you are expecting in their response.
Beyond crafting the perfectly worded question, you need to make it clear what format you want the response in. To this end, be very specific in your request, use terms such as the following:
- Provide a case study
- Provide a diagram showing
- Provide a demonstrated example
- Provide a sample of
Using these terms achieves two things:
- You are going to have consistent responses between suppliers allowing a more effective evaluation
- Your suppliers aren’t going to waste time and money trying to second guess what you want and experimenting with different responses
This approach also reduces unfair advantage of those suppliers who have a more intimate knowledge of your organisation and the way it works, a vendor won’t have an advantage because they know the organisation loves a good case study or flow diagrams. You can also build your evaluation framework more easily as you know what you are going to be evaluating and how, but more on that later.
The next part of this approach is to use consistent language to identify the importance of a criteria. You may publish your weighting, if you are using a weighted scoring model, but simple words used consistently (and potentially defined in the document) can add an extra layer to ensuring your vendors know what targets they have to hit. Commonly used words and definitions are:
- Mandatory – without this your offer is going nowhere (should not be scored, it is a go/no go decision)
- Required – miss these and your score is going to be lousy
- Desirable – we’re pretty keen on this, they’ll give you a bonus points
- Information – this helps us understand your offer (we may not even score them)
You may want to use slightly different terms (and definitions) in your documentation, but you get the gist of what we are targeting with this model.
Mandatory items should be exactly that, mandatory. If the vendor cannot provide a mandatory item, then they are out of the running unless you have no other options. It is generally not a good idea to score mandatory as how can you have a sliding scale of mandatory. It’s like having a sliding scale of being pregnant, either you are or you aren’t. “Ah,” you say, “but you can be one minute pregnant through to 9 months pregnant”. This is where the other levels come in. Say we have a study to conduct and we require women who are pregnant to participate. The further they are into the pregnancy the better, less than 3 months would give us poor results and as the study will take 4 months, we would get the best results if they are less than 5 months pregnant, but it will still get valid data with those that are greater than 8 months. We would have something like:
- Mandatory – Participants must be pregnant
- Required – Please tell us the term your pregnancy
- Desirable – The pregnancy should be 3 and 4 months advanced at the start of the study
- Information – Do you think you will have a boy or a girl
The mandatory is a go/no go, if you’re not pregnant then you can’t be in the study. This sort of question is very handy in highly competitive markets where you may get a large number of responses, it allows a first cut process of setting aside a large number of offers quickly without detailed evaluation.
Because we shouldn’t score a mandatory criteria, we need to follow it up with a measurable criteria with a range of possible responses. Since this is a 4 month study, those with a term between 3 and 4 months (8 months minus the term of the study to keep them in the sweet spot for the study) will get the best score, those between 4 and 6 months, get the next best, those less than 3 months or greater than 6 months get the worst scores. Easily quantifiable criteria should be a primary objective of setting the model as scoring becomes very defensible.
The reasoning behind the sliding scale approach is that we may not get enough respondents that fit in the sweet spot between 3 and 4 months, so we might be willing to bring through some of the 6 to 8 month candidates. If we had left it at a mandatory question of “You must be between 3 and 4 months pregnant” we would have left ourselves in a situation where we were going against our mandatory requirements repeatedly bringing the validity of the process into question.
But what if we have a huge influx of offers between 3 and 8 months? That is where our desirable comes in, this allows us to differentiate between the good candidates to get the best candidates. The ones in the desirable range will get additional points to float them to the top of the list.
So why are we asking about what gender the mother thinks the child will be? Well that just gives us some insight into the way the mother is thinking about the pregnancy, possibly help understand cultural and other demographic influences, and may help us to adjust our project and the way we interact with the candidates once selected. Information questions may be scored or they may not, but you must decide that before approaching the market and make it clear to the invitees what will be done with the information.
Here is an interesting aside, your conditions of offer should include a clause that allows you to consider non-compliant offers if you choose. Why, well imagine if the only offers we got were from couples who were trying to get pregnant, yet our mandatory requirement states you must be pregnant. In that case we could bring them in and just adjust our project accordingly rather than having to re-run the whole process again.
Clearly too many people in my life are having babies at present.
How much do I want back?
When you issue a procurement requirement to the market, more often than not you are interfacing with sales teams. What do sales team like to do? Talk. So you need to ensure you control the level of verbosity for two reasons:
- It saves time for the evaluation team, they get the concise information they need without having to wade through reams of marketing blurb
- It actually helps to level the playing field.
Large multi-national companies have enormous resources that they can access to generate piles of supporting documents, theoretical modelling frameworks and more diagrams than the local technical college. All of this does not necessarily add to their capability to deliver outcomes effectively, and more often than not, has the undesirable trait of increasing costs.
Limiting suppliers to a consistent volume of information will force them to really focus on the key points and generally improves the quality of offers received. Think of how much you think you would need to write to respond to a requirement and then knock a page off. The kinds of limits you want to apply to each criteria are:
- Answer in the space below
- No more than 3 pages
- Complete attached template
If you find you think the vendors will need more space to respond, then you may have a problem. Either your criteria are not targeted enough or maybe you don’t have enough criteria defined. So review carefully if you really need War and Peace back to understand the vendors capability or solution, it is possible you do, but it is equally possible that you are leaving your evaluation team open to a subjective evaluation style.
Be very clear on what is expected in response to each criteria. To complement this you need to define what a page is (A4 paper size using 10 point Arial text) and you need to make in clear in your Conditions of Offer that a failure to comply with this requirement may result in the offer being deemed non-compliant and set aside from further evaluation. The issuing of standardised templates for responses such as pricing, technical specifications and resources will significantly improve the quality, consistency and comparability of responses received.
If you haven’t enforced this before, you need to start soft by issuing warnings up front during the period the invitation is in the market being very clear that failure to comply is going to reduce their chances of being engaged. I promise you though that so long as you are reasonable with the amount of space allowed, your evaluation teams and ultimately you suppliers, will thank you for these parameters.
How are going to evaluate this?
Here’s the kicker, if you can’t work out how you will evaluate the response, then don’t even bother asking the question. Every criteria must result in a response that can be scored and preferably to the point where the scoring is predictable. It may be key capabilities being described in a case study; quantifiable values such as response times, volumes or sizes; or it may be the inclusion of specific functionalities. The key is to ensure that when you write a question and consider what content and format you want the response in, you also consider what you will do with that response when it is received.
I recall a product owner from a European based solution describing how their product allowed a customer to manage a process that resulted in 9,500 offers coming in for a single requirement (what a nightmare). The way the solution achieved this was by driving the customer to define the bulk of the questions in a way that they could break the possible answers into 3, 4 or 5 possible responses. Each response was given a value and as a result, the first round of evaluation and short listing was actually taken care of by the solution through an automated process.
You can apply this concept to all evaluation criteria, have mandatory or very incisive, highly weighted questions targeted at getting rid of the respondents who are never going to deliver your requirement. This is particularly important when you have a public offer that could result in a large number of responses. Evaluate these criteria for all respondents first and get your list down to the suppliers you are really interested in before putting effort into understanding the details of the offers.
Before progressing with the assessment of offers, make sure you have a clear plan for how you will evaluate and make sure all voting members of the evaluation team are on the same page as to understanding the evaluation model and how to assess and score each criteria. It’s pretty important that all evaluators are actually looking for the solution to the same requirement and this can sometimes be overlooked in the original planning and preparation. Evaluation teams should be multi-skilled and it is often a good idea to have someone from outside the sponsoring department to provide a different perspective and potentially balance bias, but everyone on the team must have the same understanding of the requirements and the key objectives in souring a solution.
A final item to consider is targeting the scoring system to reward the supplier who provides what you are looking for and not to reward the vendors who over supply. Just as it is important that you don’t specify the Queen Mary when all you need is a tug, you should set your scoring system to have a sweet spot for hitting the target and ramp off for over supply. An example that is commonly seen is in engaging skilled resources. You may need a project manager to manage a couple of mid range projects. You receive a response from a highly skilled and experienced project manager who has just completed a complex, multi-regional project. If your scoring system is open ended allowing over specification, the respondent may score straight 10’s and blow all other respondents out of the water, but you will pay for that experience and yet you won’t use that experience. Unless over supplying is a desirable outcome and you are willing to pay for it, be sure to build your specification and your evaluation approach so that they penalise over supply and focus the solution to the requirement to get the best value outcomes.
Implementing these basic concepts and principles will not only improve the quality and efficiencies of your procurement process, but is also likely to help achieve a better outcome from the invitations you issue.
Thank you for taking the time to read to end of this blog, and I hope I have helped trigger ideas on how you will write a current invitation package. Got any thoughts you want to share, then please leave a comment below.