1. Home
  2. Smartpedia
  3. Shortlisting

What is Shortlisting?

Table of Contents: Definition – Applications – Prerequisites – Procedure – Example – Differentiation – Advantages and disadvantages – Questions – Notes

Smartpedia: Shortlisting is a selection method that separates a long list of options into suitable and unsuitable ones on the basis of predefined criteria.

Shortlisting: preselection based on defined criteria

There is a list of thirty ideas on the table, and each of them has an advocate somewhere in the organisation. Realistically, the budget and time available are enough to implement around a third of them. If people simply start debating, the loudest voices could end up winning.

Shortlisting is a way to counteract this. It supports decision-making, or more precisely, preselection: from a large number of options, you use clear criteria to select those that meet your requirements. Every option goes through the same check and ends up on one of two lists. The approach is easy to explain, needs nothing more than pen and paper and can be used both on your own and in a group.

Shortlisting: preselection based on defined criteria

In German-speaking countries, the approach is known as the ‘Aschenputtel-Methode’, loosely translated as the Cinderella method. The name comes from the Brothers Grimm’s version of the fairy tale: the stepmother tips a bowl of lentils into the ashes, and Cinderella has to pick them all out again by evening. She calls the doves to help her, with the instruction ‘Die Guten ins Töpfchen, die Schlechten ins Kröpfchen’, the good ones into the pot, the bad ones into the crop [1]. This scene does not appear in the version best known in the English-speaking world, which goes back to Charles Perrault. In English, the idea is captured by a different image: separating the wheat from the chaff. Shortlisting is not a scientifically defined term, nor does it have a single originator. It refers to the widespread practice of basing selection decisions consistently on criteria agreed in advance.

Key points at a glance

  • Shortlisting sorts a long list of options into suitable and unsuitable ones on the basis of criteria agreed in advance; in German-speaking countries, it is known as the Aschenputtel-Methode, after the Grimm fairy tale.
  • It is suitable for preselection, for example of suppliers, projects or features for a release, and requires only simple tools, whether used alone or in a workshop.
  • The process comprises five steps: clarify the problem and goal, collect options, define criteria, assess the options, and make and document the selection.
  • Options are assessed using a simple scale such as ‘yes, partly, no’ and a selection rule agreed in advance; each option is either pursued or set aside, but no ranking is produced.
  • Its strengths are orientation, transparency and acceptance; where there are very many options, interdependencies or a need for a ranking, methods such as utility analysis complement it.

Applications of shortlisting

Shortlisting is useful wherever a transparent selection needs to be made from many possibilities and a detailed evaluation would still be premature or too costly. In practice, three groups of occasions can be distinguished.

  • Selecting tools, software and partners: Which three of fifteen suppliers do we invite to give a presentation? Which platform makes it into the final round? The method produces a shortlist, so that further assessment can focus on a few candidates.
  • Defining projects and scope: Which projects, measures or investments do we tackle this year? Which features belong in the first release of a piece of software? Here, the method protects against a programme that contains a little of everything and finishes nothing.
  • Choosing approaches and formats: Which project management or workshop methods suit our team? Which training formats do we offer? The selection is based on what suits your own organisation.

In workshops, the method offers an additional benefit: it gives a group a shared yardstick, so that a decision is reached openly and in a way everyone can follow. At the end, there are four results: the list of all options considered, the criteria used, the list of selected options and the list of rejected options. Together, they document the path to the decision, which helps at the latest when someone asks months later why a particular idea was not implemented.

Prerequisites and tools

The method requires no prior knowledge. However, if you use it in a workshop, you should be able to facilitate, as interests quickly clash when it comes to selection. More important than expertise in the method are a few conditions that should be met before you start:

  • A clear goal: The goal determines which criteria make sense and which options are worth considering at all.
  • Openness to different criteria: Measurable criteria such as costs carry the same weight as assessments such as acceptance among employees. Those involved should be willing to use both and to question their own assumptions.
  • Disclosed criteria: All yardsticks are on the table. If the result has secretly been decided already and the method is only meant to justify it after the fact, it loses its purpose.

In terms of information, the main thing needed is the task itself. Also helpful are an existing list of options, known requirements from which criteria can be derived, and the expertise and experience of those involved. The tools required are modest: research, creativity techniques or an AI assistant are sufficient for collecting options. For the assessment, a flipchart, facilitation cards and paper will do, or an online whiteboard or a spreadsheet in distributed teams.

Shortlisting in five steps

The method follows five steps whose order is determined by the task itself: only once the goal and criteria have been set can an option be assessed fairly.

Step 1: Clarify the problem and goal

First, describe the task or problem for which you are looking for a solution. Simple analysis tools such as a problem tree or an Ishikawa diagram help to sort out causes and relationships. Then define the goal of the selection, for example using the SMART formula. This also includes the framework: How many options do you want to consider? How many should remain at the end? What limits apply to the criteria, such as a fixed budget or a deadline? And what requirements must the final selection meet as a whole?

Step 2: Collect options

Depending on the task, options can be very different things: courses of action, service providers, tools, features, methods or strategies. In the simplest case, a list already exists, such as a supplier overview or the result of a previous problem analysis. Otherwise, you have several sources: you can research specialist literature, databases and the internet, or consult experts. In a team, you can collect known options using brainstorming, brainwriting, the 6-3-5 method or a card query. Completely new ideas emerge with creativity techniques such as reverse brainstorming or SCAMPER. AI assistants such as ChatGPT, Gemini or Perplexity can also provide an initial list of suggestions in a short time, although you will need to check it yourself.

Step 3: Define criteria

Criteria are the yardsticks against which you measure each option. They determine which characteristics count for this decision. There are four ways of finding suitable criteria, which can also be combined:

  • Collect them in the group: Participants spontaneously write down criteria and then jointly select the most important ones, for example by dot voting, in which everyone distributes a limited number of sticky dots.
  • Ask stakeholders: Individuals or groups affected by the decision state their requirements in interviews or surveys.
  • Derive them from the goal: You work backwards from the goal set in step 1 and ask what an option must achieve for the goal to be reached.
  • Adopt proven criteria: If there was a similar selection process in the past, its criteria can be reused. The specialist literature also lists typical criteria for many questions.

A handful of criteria is usually enough. Each additional criterion multiplies the number of individual assessments.

Step 4: Assess the options against the criteria

Now assess each option. This can be done quantitatively in an evaluation matrix or qualitatively in a group discussion.

For the evaluation matrix, enter the options in the rows of a table and create a column for each criterion. Each cell contains an assessment of how well the option meets the criterion. Agree on a scale beforehand. The following overview shows the common variants and their effect.

Scale Example levels Effect and use
Two-level yes, no Pure knock-out principle, very fast and the clearest way of separating the wheat from the chaff
Three-level yes, partly, no Allows an intermediate level but remains sharp, good for many criteria
Four-level 1 to 4 or very low to high No midpoint, forces a tendency upwards or downwards
Five-level 1 to 5 or very low to very high Finest common gradation, often more intuitive in words than in numbers

Finer scales are not useful for shortlisting, because they suggest a precision that a preselection does not have. With numerical values, you can add up the total in a final column. If the group wants a finely graded or weighted assessment, this is a signal to switch methods, for example to utility analysis or the Pugh matrix.

The qualitative assessment is suitable for criteria that are difficult to express in numbers, such as whether a solution fits the corporate culture. For each option, the group records on a separate flipchart sheet how it rates the option against each criterion. The sheets are then hung side by side on the wall, and everyone is given time to read them carefully before a decision is made.

Step 5: Make and document the selection

The aim is a clear decision for every single option: it is either pursued or set aside. To achieve this, formulate a selection rule. It can be general (‘at least 15 out of a possible 25 points’) or differentiated (‘no criterion may be rated “low”’). The simplest approach is to treat all criteria as knock-out criteria. Only options that clear every hurdle then remain. This is the purest way of separating the wheat from the chaff.

The list of selected options is the input for the next work step, for example implementation planning, a detailed evaluation or the further development of the selected ideas. If it is still unclear how to proceed, a short reflection format such as ‘What, So What, Now What?’ from Liberating Structures helps to clarify the next steps.

Example: Which features go into the first release?

Lernwerk Nord is a provider of professional training, offering evening and weekend courses in accounting, languages and project management. Around 3,000 participants register each year, so far via an email form or by phone. Course administration transfers every registration by hand, and customer service answers questions about invoices and dates every day. An external software service provider is now to develop a customer portal. The head of customer service takes on project management on behalf of Lernwerk Nord and facilitates the selection.

Clarify the goal: The first release is to go live at the beginning of February, in time for the spring registration period. It should noticeably relieve customer service and course administration. According to the service provider’s estimate, the budget is sufficient for a maximum of ten features.

Collect options: A workshop with sales, course administration, accounting, customer service and two tutors produces 31 feature requests. The project manager adds an analysis of the most frequent customer enquiries over the last six months. The list includes obvious items such as online booking, but also ideas such as learning badges or a rating function for courses.

Define criteria: The group collects nine suggestions for criteria and selects three by dot voting. The first asks whether a feature relieves customer service or course administration. The second checks whether it is part of the core of booking and payment. The third concerns whether, according to the service provider’s effort estimate, it can be implemented by February. The scale used is ‘yes, partly, no’. The selection rule is: a feature goes into the first release if it receives no ‘no’ and at least two ‘yes’ ratings.

Assess the options: The table shows an extract from the evaluation matrix with six of the 31 requests. ‘Shortlisted’ stands for the first release, ‘Set aside’ for the list of later requests.

Feature request Relieves the team Core of booking and payment Feasible by February Result
Online booking with payment yes yes yes Shortlisted
Download invoices as PDF yes partly yes Shortlisted
Self-service rebooking and cancellation yes yes partly Shortlisted
Waiting list with automatic move-up yes partly no Set aside
Rate courses no no yes Set aside
Learning badges for participants no no partly Set aside

The waiting list sparks a lively discussion. When courses are fully booked, course administration regularly phones the people next in line and would like the feature straight away. The group therefore also discusses this case qualitatively. Afterwards, the flipchart reads: the waiting list provides relief mainly during registration peaks and needs to be linked to payment. The service provider expects several weeks of effort. The result remains a no for the first release.

Document the selection: Nine of the 31 requests meet the rule and form the scope of the first release. The remaining 22 are placed on a separate list with a brief justification, with the waiting list at the top as a candidate for the second release. The service provider thus receives a clearly defined brief, and everyone involved can read why their request may have to wait.

Differentiation from other selection methods

Shortlisting sorts options into two groups. It does not provide a ranking within these groups. Related methods have a different focus and can often be combined well with it.

  • Utility analysis: Here, each criterion is given a weight; the ratings are multiplied by it and added up to a utility value. The result is a ranking. Utility analysis requires more effort and is well suited to the options that remain after a round of shortlisting.
  • Pugh matrix: Named after Stuart Pugh, this method compares each alternative with a reference concept and assesses whether it performs better, the same or worse on a criterion. It is a good fit when there is an existing solution against which new variants can be measured.
  • MoSCoW method: The MoSCoW method classifies requirements into four categories: Must, Should, Could and Won’t. This gradation is particularly helpful when several releases are planned one after the other.
  • RICE scoring: RICE scoring calculates a value for ideas and features from reach, impact, confidence and effort, and so puts them in order. It requires estimates that are often not yet available in an initial preselection.
  • Dot voting: In dot voting, all participants distribute a limited number of dots among the options, producing a quick snapshot of opinion. Within shortlisting, it usually serves as a building block in step 3 to identify the most important criteria from many suggestions.

Other methods such as ABC analysis, the Eisenhower matrix or paired comparison also aim to prioritise elements. In short: shortlisting is the coarse filter at the beginning, and the other methods take care of the fine-tuning.

Advantages and disadvantages of shortlisting

The strengths of the method lie in its simplicity and traceability:

  • Orientation through clear steps: The fixed sequence gives everyone involved a framework. Inexperienced teams in particular are less likely to get lost in fundamental debates and reach a decision they can stand behind more quickly.
  • Transparency and documentation: Because criteria and assessments are recorded, every decision can be justified later, including to people who were not involved.
  • Acceptance through participation: In the workshop, different departments contribute their perspectives. Anyone who sees that their request was measured against the same yardsticks as everyone else’s is more likely to accept a no.

However, these advantages only apply as long as the number of options and criteria remains manageable and the criteria are well chosen. This is precisely where the weaknesses lie:

  • High effort with large numbers: Fifty options times eight criteria result in 400 individual assessments. The method then becomes confusing and tedious.
  • No interdependencies: Each option is assessed on its own. The fact that two weak options could together form a strong solution remains invisible.
  • Dependence on the criteria: Criteria that are too broad let too much through, while criteria that are too narrow rule out good options. A poorly worded criterion distorts the entire selection.
  • Transparent, but not objective: The result is only as good as the assessments of those involved. The method makes decisions traceable, but responsibility for them still lies with the people who make them.

Most of these weaknesses can be kept in check with two decisions: the list of options remains manageable, and the criteria are carefully worded and reviewed together before the assessment. If interdependencies between options play a role or a fine-grained ranking is needed, a complementary method such as utility analysis takes over. Shortlisting shows its strength wherever a group has to reduce a long list to a well-founded selection in a short time.

Questions from the field

Here you will find some questions and answers from practice:

What tips help with shortlisting?

Whether shortlisting delivers a useful result is rarely decided at the assessment stage. Most problems arise earlier: with vaguely worded criteria, a group whose make-up is too one-sided or a mistake that goes unnoticed for a long time. The following tips help to counteract this:

  • Define criteria together: For each criterion, clarify what ‘yes’ and ‘no’ mean. Otherwise, two people will rate the same option differently because they understand different things by ‘relieves the team’.
  • Set the criteria before the options: Anyone who formulates the criteria only after everyone has talked about their favourites will unconsciously adapt them to those favourites.
  • Involve experts from different areas: With complex topics, accounting sees different consequences from sales. A mixed group spots the weaknesses of an option earlier.
  • Review interim results: Take a quick look back after each step. If it only becomes apparent during the assessment that a criterion is unclear, correcting it takes considerably more time.

The tips cost little extra time but often save a second round of selection. If you are using the method for the first time, it is best to start with a manageable question and a few criteria. That way, the team gets to know the process before a decision with greater consequences is at stake.

What should you do if almost nothing or almost everything remains after sorting?

In both cases, it is worth looking at the criteria first. If hardly anything remains, the criteria are probably too strict or there are too many knock-out criteria. Check which criterion ruled out the most options and ask whether it is really indispensable. If, on the other hand, almost everything remains, the criteria do not differentiate enough. A second round with an additional, sharper criterion can help, or switching to a method that produces a ranking, such as utility analysis.

What do I do if a new, promising option emerges in the middle of the assessment?

Include it and assess it using exactly the same criteria and scale as all the others. It is tempting to simply wave a late idea through because it is new, but this undermines the fairness of the selection. If several new options appear at once, the collection phase was probably too short. In that case, it is worth pausing the assessment briefly and catching up on step 2 in a targeted way. Also decide in advance up to what point new options will still be considered; otherwise, the selection will never end.

Who decides if the group cannot agree on an option?

This rule should be set before the assessment, because once a dispute has arisen, it is hard to agree on it neutrally. There are three common variants: the majority decides, the person with budget or project responsibility decides, or disputed options go into a separate qualitative discussion. It is important to record dissenting opinions. If a decision later turns out to be wrong, the documentation shows what concerns were raised and whether they were taken into account.

Impulse to discuss:

Does an experienced team even need a method for preselection, or is a jointly agreed goal enough?

Notes (partly in German):

If you like this post or would like to discuss it, please feel free to share it within your network.

[1] Jacob and Wilhelm Grimm: Aschenputtel (KHM 21). In: Kinder- und Hausmärchen, Vol. 1, Große Ausgabe, 7th edition. Dieterich, Göttingen 1857, pp. 119–126.

Here you will find an interesting article on the question: Where did the KO criteria go?

And here you’ll find more information from our Smartpedia section:

Smartpedia: What is the Ishikawa Diagram?

What is the Ishikawa Diagram?

Smartpedia: How does SCAMPER work?

How does SCAMPER work?

Smartpedia: What is RICE Scoring?

What is RICE Scoring?

‘Smartpedia’ is our glossary for people in organisations involved in software development, project management and product management. It is for precisely these people and their organisations that we have been developing and modernising software since 2012. Pragmatic. ✔️ Personal. ✔️ Professional. ✔️

Get to know t2informatik from Berlin as your development partner.