Empirical requirements engineering
Requirements engineering is about arriving at complete, correct, understandable requirements and keeping them up-to-date. With the help of interviewing, facilitation and research techniques, but also with activities such as analysis and synthesis, modelling and formulation, complexity management, feasibility studies, quantitative forecasts and management decisions, organisations try to solve this task.
In requirements engineering, there is a wide range of recommended methods and techniques, and since every project has different parameters, not every method fits equally well everywhere. A similar situation exists in medicine, where numerous treatment methods and drugs compete with each other. Each individual doctor can only gain random experience himself. That is why evidence-based medicine systematically and scientifically creates the knowledge base for doctors with the help of empirical research, so that they can select the appropriate treatment for each case in a well-founded and objective manner. Analogously, empirical requirements engineering examines the effectiveness of different approaches in different areas of application.
Challenges in empirical requirements engineering
When I started requirements engineering research about 20 years ago, I was shocked by the low level of scientific ambition in the community. I had learned otherwise during my studies. At the turn of the millennium, computer science was not only a playground for explorative trial and error, where generally accepted procedures and quality standards had yet to develop. This has worked out pleasingly well. There are now plenty of standard works of books and much-cited articles that set standards for empirical studies in software engineering, such as Robert K. Yin’s books on case studies1, the article “Preliminary Guidelines for Empirical Research in Software Engineering”2 by Kitchenham et al. or the “Reporting Guidelines for Reporting Experiments in Software Engineering” by Fraunhofer IESE3 as well as guidelines for systematic literature reviews.
The case study in a single project is still the most common empirical research method in software engineering, but has the least scientific validity. More and more controlled experiments are also being carried out and meta-studies are being used to summarise the research results of numerous studies.
Empirical data collection can be supported by quantitative evaluation methods such as hypothesis tests, correlation analyses and factor analyses. The difficulty in making definitive statements, however, is that you basically cannot carry out the same project more than once, supporting it each time with a different method and measuring the difference. You cannot work on the same project twice with the same team in an unbiased way, because they will naturally benefit from the knowledge of the first run during the second run. If you run “the same project” twice with different teams, using different methods, then you don’t know for sure whether differences in the outcome were caused by the method or arose because each person is different. Even if you use the same procedures with the same goal and the same specifications, you end up with a completely different result. This can be observed very well in teaching, where this situation occurs regularly.
Possible strategies for using empiricism
Basically, empirical research is conducted to help practitioners know which technique or practice best fits their project or problem in any given situation. However, it is not so easy to draw the right conclusions from the flood of research findings. There are two strategies for doing so:
1. look for one or more case studies that are similar to your own situation, e.g. in terms of
- team size,
- project size,
- project type (new development versus migration versus further development) and
- customer relationship (individual development for a specific customer versus development of a product for the market, i.e. for numerous customers)
and many more.
However, research is not so far advanced that it could indicate in which of these parameters your project must match a project under study so that the approach successfully used there will also lead to success in your case. This is probably not even possible, because too many factors are needed together for project success without being able to guarantee it.
2. consider meta-studies, which summarise a large number of studies. However, there is currently no systematic help for practitioners to read such studies and draw the appropriate conclusions from them.
The use of empirical methods during requirements engineerings
When I asked ChatGPT what it meant by empirical requirements engineering, it made me think of an additional meaning that the term could have, even if it has not been used that way before: The use of empirical methods during requirements engineering.
ChatGPT put it this way: “This approach emphasises the collection and analysis of empirical information to ensure that requirements meet real needs and challenges.”
This would mean:
- Conducting user surveys, interviews and workshops for requirements elicitation and prioritisation, market analysis and prototype testing with the same care in preparation, execution and evaluation that scientific studies are conducted.
- Prioritise requirements based on collected data rather than pi by thumb.
- Testing requirements using empirical methods such as prototype testing, user testing and simulations to ensure that they satisfy real needs.
Feedback loops to regularly evaluate the results so far.
This in turn reminded me of the book “Lean Startup”4 by Eric Ries, who also recommends empirical methods for product development in a startup, especially getting end-user feedback and measuring or using metrics. Requirements engineering is only one aspect in this book, but an important one.
Development frameworks and process improvement methods such as Lean Production, PDCA, Six Sigma, Kanban and bottleneck theory also fundamentally work with quantitative data to ensure fact-based management. In project management, earned value analysis is used.
The falsification of data
The more the software industry uses tools such as Kanban boards or task boards, burndown charts and ticket systems to log daily work, the easier it becomes to falsify process management metrics like
- degree of completion,
- productivity (velocity),
- work in progress,
- throughput and turnaround time of tasks.
The more detailed the work is managed – at project level, as a three-month work package, as a two-week sprint or one-day task – the more detailed and accurate these evaluations become. With all the side effects such as evaluation effort and data protection.
Of course, it causes employees a certain amount of psychological stress when their work is logged and evaluated to the hour. He avoids this stress by fudging his data or neglecting those tasks that count for few points. Monitoring thus becomes control at the same time. This is also a scientific principle: when you monitor people, they change their behaviour. This falsifies the data.
Conclusion
So, to sum up, I would say: The idea is great. There is eager work on empirical requirements engineering – in many scientific communities such as research institutes, conferences, journals and working groups. However, the last step to overcome the knowledge gap between research and practice is still missing, namely to provide simple selection tools that allow practitioners to draw the right conclusions for themselves from the huge knowledge pool, e.g. to select the appropriate requirements engineering technique for a specific situation.
Notes:
If you like this article or want to discuss it, please share it with your network.
[1] Robert K. Yin: Case Study Research and Applications – Design and Methods
[2] B. A. Kitchenham: Preliminary guidelines for empirical research in software engineering, https://dl.acm.org/doi/10.1109/TSE.2002.1027796
[3] A. Jedlitschka, D. Pfahl: Reporting guidelines for controlled experiments in software engineering
[4] Eric Ries: The Lean Startup
Dr Andrea Herrmann has published other articles in the t2informatik Blog, including
The biggest obstacles in requirements engineering
Agile requirements engineering
Misunderstandings in requirements engineering
Dr Andrea Herrmann
Dr Andrea Herrmann has been a freelance trainer and consultant for software engineering since 2012. She has more than 28 years of professional experience in practice and research.
Dr Herrmann was most recently a deputy professor at Dortmund University of Applied Sciences and Arts, she has published more than 100 specialist publications and regularly gives conference presentations. She is an official supporter of the IREB Board and co-author of the IREB syllabus and handbook for the CPRE Advanced Level Certification in Requirements Management.