How good is your solution?

Do you work for a company that develops products or provides services? Are you in competition with companies that regularly publish innovations and continuously invest in the further development of existing solutions? Then you are in good company, because continuous changes, extensions, innovations are standard for many companies today. You spend a lot of time to realise improvements and generate sustainable benefits. But is this effort worth it? Do the solutions developed deliver what you expected of them? And how can solutions be evaluated at all? The answer is: with a solution evaluation.

Solution Evaluation – What is it?

Solution Evaluation is a so-called field of knowledge in the Business Analysis Body Of Knowledge Guide – BABOK for short – the standard work on business analysis published by the International Institute of Business Analysis – IIBA for short – in Canada. It describes the evaluation of partially or fully implemented solutions before or after they are implemented in the company and also considers systems, processes, rules and organisational structures. This holistic approach makes a lot of sense, as solutions can have multiple effects within an organisation. Interestingly, this far-reaching approach is often ignored in the requirements engineering and project management literature.

No solution evaluation without good requirements

Every company that invests in the development of a solution hopes and expects something of it. Often it is monetary goals such as cost savings, sales growth or increasing market share. Increasing customer satisfaction is the focus of many companies and more and more organisations pay attention to the needs of their own employees and try to increase their satisfaction. In addition, there are of course – not only since the diesel scandal in the automotive industry – standards and laws that are addressed by solutions.

In general, a solution is an implemented business requirement. It is based on a wish, a goal or a need. To evaluate a solution it is important to understand this basis as clearly as possible. However, this can only succeed if correct, clear, consistent, evaluated, verifiable, modifiable and traceable requirements have been documented. Unclear or incomplete requirements lead to inadequate evaluation criteria, inappropriate prioritisation or weighting. For example, an input mask for entering data in a CRM system can work well for the internal sales force, while it has an unacceptable usability in direct contact with a customer. It quickly becomes clear that a holistic approach is essential for the quality of a solution evaluation.

The definition of evaluation criteria

When does a development become a good solution? Or to put it another way: how can relevant evaluation criteria be developed for a meaningful solution evaluation? Often the simple assessment of whether a feature has been implemented is not sufficient. Of course, the question of whether the monthly sales report of maintenance customers can be printed out as a PDF can easily be answered with “yes” or “no”. But this assessment does not take into account the layout, readability, speed of printing, paper savings through double-sided printing or the possibility of printing in black and white instead of colour. In general terms, this means that evaluation criteria can be derived from the documented, functional requirements, while non-functional requirements in the solution evaluation present a challenge. Frequently, it is the quality of the implementation of non-functional requirements in particular that determines how satisfied a user is with a product, software or service.

In my experience, the Kano Model of customer satisfaction offers very good support in the development of evaluation criteria. It classifies attributes of a solution into so-called basic features, performance features and enthusiasm features. Especially the performance and enthusiasm characteristics provide important information for the definition of evaluation criteria and this for both functional and non-functional requirements.

The weighting of evaluation criteria

Solution Evaluation can be used at different points in a development. For example, it can be applied on the basis of prototypes, mock-ups or already partially developed products and services. It can be used within an organisation to evaluate existing solutions and helps to clarify whether existing products should be further developed, continued to be operated or discontinued. It can also be used in purchasing and specialist departments to evaluate solutions together with suppliers. However, in addition to the definition of evaluation criteria, the weighting of these criteria is also essential for solution evaluation. The use of a weighted evaluation matrix can provide valuable services here. Double weighting has proven to be particularly effective, in which criteria are weighted both individually and in groups. Although this causes some effort, it is worthwhile in practice because the questions that arise during weighting often lead to further discussions with stakeholders. Ideally, therefore, the criteria and weighting should also be determined as far as possible. If the developers know both, there is a good chance that they will come up with a solution that satisfies and possibly even inspires customers.

The continuous review

The solution evaluation is a snapshot. Similar to the Kano Model of customer satisfaction, in which enthusiasm factors can become performance or even basic factors, the benefits can change over time. As a consequence, this means that the results of the solution evaluation should also be questioned again and again. For example, it is simply a matter of clarifying whether the solution still covers the current business requirements, whether these have changed in the meantime or whether they have even been dropped.

In corporate reality, it happens time and again that, for example, the costs of operation and care and maintenance no longer or not significantly exceed the monetarily assessed benefit. As a consequence, companies have to decide whether they want to revise parts of the solution or the entire solution or whether they want to terminate and discontinue the operation of the solution. When making such a decision, however, it is important not only to carry out a pure cost-benefit analysis but also to consider possible opportunity costs or benefits, so that those investments in solutions are identified which make the most sense and have a sustainable, positive effect for the company.

Conclusion

The benefit of a solution is an essential success factor for the solution. Stakeholders determine this benefit through their goals, wishes and needs. However, as stakeholders change their opinions and markets, and thus the products and services in these markets, evolve, it is important to analyse these benefits continuously – and preferably cyclically and iteratively. This can be done in the product development phase with the help of mock-ups, prototypes or proof-of-concept installations, or even during the operation of a solution. However, experience shows that poorly planned and insufficiently structured evaluations of solutions can also entail considerable risks. Once decisions have been made, they often lead in practice to the long-term use of inadequate solutions. And how could the damage that this causes – for example, unnecessary costs, great effort, possible loss of sales and dissatisfied customers or employees – be avoided? The answer is: with a good solution evaluation.

Notes:

Under the direction of Mr. Wendt, masVenta Business GmbH organises an annual conference for business analysis and requirements engineering, the European Business Analysis Day, in short BA-DAY. In contrast to other conference formats in these areas, this event is consistently international in scope and offers many opportunities for networking with the international BA and RE community, in addition to presentations by excellent speakers from all over Europe, America and Canada.

Rainer Wendt has published two more articles in the t2informatik Blog:

What does t2informatik actually do? One click and you'll know.