The hidden role of expert decisions in code

One pull request, two lines changed. A threshold shifts from 0.72 to 0.65. The comment reads: ‘Too few candidates in the funnel’. The review takes four minutes; it is merged that same afternoon.

No one involved in this process would have answered ‘yes’ if asked whether they had just made a decision about people. And yet, from the next deployment onwards, this figure will determine whose application a human being actually gets to see.

My last post here on the blog dealt with the question of who is allowed to make decisions within an organisation: the AI, a human with its support, or a human without it. Four zones, categorised according to the scope of the decision. This post describes the level below that, because the zone allocation devised in a workshop stands or falls on a point that does not arise in any workshop: in the repository.

The requirement remains silent precisely where it matters

A requirement states: ‘Incoming applications are pre-sorted according to defined criteria.’ A clear sentence, technically approved, scheduled in the sprint.
What the sentence does not say: at what score the sorting stops, and whether anyone looks at applications below that threshold. It also remains unclear how gaps in CVs are handled and on what basis the model learns: from recruitment practices over the past ten years or from a defined requirements profile.
These gaps are not necessarily down to negligence. They arise because the subject matter experts often do not ask the question in this level of detail, yet the development process still requires a concrete answer. Without a value, a weighting or a default, the code will not run.
What remains undecided from a business perspective is decided from a technical perspective.
It is precisely at this interface that a decision-making space emerges, for which organisations have often not designated a specific role. The business department believes it has made a decision because it has approved the requirement. The development team sees itself as implementing the requirement because it is following an approved specification. Both positions are understandable. And yet, in the end, a decision may emerge that has a significant impact without anyone having consciously made it as a business decision.
The problem, therefore, is not that developers secretly take business decisions upon themselves. The problem is that organisations only recognise certain decisions once they are already embedded in the code.

Three areas where this happens regularly

Such unintended technical decisions arise particularly frequently where technical details have a concrete impact on subsequent individual cases.

1. Thresholds and limit values

Every automation process needs a threshold: the amount at which a transaction is sent for approval, the score at which a query is answered automatically, or the confidence level at which the system acts independently.

Such a figure can determine whether a process is handled automatically or whether a human remains involved.

The weight that a seemingly technical threshold can carry was demonstrated by the awarding of English A-level marks in the summer of 2020. Because the exams were cancelled, teachers’ predictions were standardised using a procedure developed by the exam regulator Ofqual. Among other factors, the size of a class played a role in this: for classes with fewer than five candidates, the teacher’s assessment remained unchanged; for classes with five to 15 candidates, it was weighted against the calculated result.[1] A numerical threshold thus influenced when human judgement prevailed and when the calculation took precedence.

The real lesson to be learnt is less the specific figure itself than its nature: A threshold that appears technical can have social or organisational implications.

2. The selection of what is measured in the first place

The decision as to which data and characteristics are fed into a system also affects the outcome.

Amazon developed a recruitment tool that systematically disadvantaged female applicants and was eventually discontinued. The system had learnt from applications over the previous ten years, the vast majority of which were from men.[2] The choice of this data set was technically plausible, but by no means neutral: those who learn from past decisions may also adopt their patterns.

The value judgement then does not lie in an explicitly formulated rule. It lies in the selection of what is presented to the system as relevant past data.

3. Defaults and escalation paths

What happens when the system is uncertain? Does the process continue, does it halt, or is it referred to a human? And if so, which one?

A default provides an answer for all cases in which no one actively intervenes. It is precisely for this reason that such specifications deserve more attention than their technical inconspicuousness might suggest.

‘Merely preparatory’ is not a safe answer

At this point in discussions, the argument regularly arises that the system does not actually make a decision at all; it merely prepares the ground. After all, a human being reviews it afterwards.

In 2023, the European Court of Justice set an important limit on this view in the SCHUFA case. The automated calculation of a credit score may itself constitute an automated decision within the meaning of the GDPR if a third party, such as a bank, relies on this score as a decisive factor in its decision.[3]

What is decisive, therefore, is not merely who formally carries out the final step, but what actual effect an automatically generated assessment has. For organisations, this is more than just a legal footnote.

A human being involved in the process does not automatically become an effective reviewing authority simply by being shown a screen or clicking a button. They must understand the basis of the recommendation, be able to review it, and have a realistic opportunity to deviate from it.

This is why the deviation rate is of interest. Not because a low rate automatically proves that no review is taking place. After all, a good system can frequently make correct suggestions. But near-total agreement over a long period should at least raise one question: is someone actually reviewing this independently, or is the process merely routinely confirming a pre-determined decision made by a machine?

The crucial question is therefore not simply whether a human is involved. It is: can this person really decide otherwise, and do they actually do so from time to time?

Responsibility flows upstream

Once a decision is embedded in the code, it does not disappear. It shifts to those who set thresholds, selected data, configured models, wrote templates or chose providers.

This becomes particularly clear when a model is changed. Your supplier discontinues the version that you tested and approved a year ago. The successor applies different weightings, blocks different content and, in borderline cases, behaves more cautiously or more aggressively than its predecessor.

There is a ticket in the backlog titled ‘Update model version’, scheduled as maintenance and estimated to take two days. No one in your organisation has decided that the boundary between two decision categories should shift. Yet it may have shifted nonetheless. The original approval remains formally valid, even though the system’s behaviour has changed.

This is an inconvenient feature of automation: responsibility shifts further upstream in the process and thus often to points that do not see themselves as decision points. A team that modifies a prompt template may influence thousands of subsequent individual cases. A procurement department choosing between two suppliers also indirectly decides on model properties, training foundations, blocking mechanisms and fallback behaviour.

This shift is almost impossible to avoid. However, that does not mean it has to remain invisible.

What helps in practice

The good news is that development organisations do not necessarily need a new governance system for this. Many of the necessary tools already exist. They simply need to be applied to decisions that have, until now, often been regarded as purely technical implementations.

Monitoring the deviation rate

If a person is designated as the reviewer, it should be clear how often automated recommendations are confirmed, amended or rejected. A very low deviation rate is not in itself proof of an ineffective review. However, it is a good reason to take a closer look: Does the reviewer understand the basis for the decision? Do they have sufficient information? Is there time and an incentive to actually carry out the review? And what happens if they disagree?

This metric therefore does not replace a qualitative review of the process. It shows where such a review is worthwhile.

Making decision-making responsibility visible in the ticket

Every story that involves automated preliminary decisions is categorised into the relevant decision zone. For relevant zones, it is also recorded which functional role bears responsibility and which specific person is currently exercising that responsibility.

Both are important. A role alone remains somewhat abstract. A name alone disappears with the next job change. If, whilst filling in the form, it becomes apparent that neither a responsible role nor a responsible person can be named, something essential has already come to light.

Document decisions where they are made

Architecture Decision Records are well-established in many teams: brief notes on context, alternatives, the decision and its consequences. The trigger for creating them can be broadened. Anyone who sets a threshold that has a significant impact on people, selects a database or defines an escalation path is documenting not only the technical implementation but also the business rationale.

This does not require a twenty-page governance document. Often, just a few lines are enough. The value becomes apparent later, when someone can understand why a particular figure was chosen, which alternatives were discussed and who was responsible for the decision.

A review rule with teeth

Changes to relevant thresholds, attribute lists, prompt templates or escalation rules should not be reviewed solely from a technical perspective. They require a second pair of eyes from the business department responsible.

This adds a little to the turnaround time. In return, it prevents precisely those changes that appear technically trivial but have far-reaching business implications.

None of these measures is particularly spectacular. Nor do they need to be. What is crucial is that organisations treat technical specifications as decisions in those areas where they actually have a decisive impact.

Clarifying decisions right down to the code

The zone assignment from the workshop and the figure in the config file must yield the same result. Otherwise, you have two systems: one that describes how decisions should be made, and one that actually makes them.
The test for this is straightforward. Take one of your automated processes and ask who chose the threshold value that controls it. Then ask why exactly this value applies and who is professionally responsible for it.
If you receive a comprehensible answer, the clarification has extended right down to the technical implementation. If you get a shrug, you have found a decision that your organisation has not yet treated as such.
Not every technical decision is a business decision, but some technical specifications become business decisions because nobody decided on them beforehand.
 
Notes:
Guido Bosbach supports organisations on their journey towards clarity in their working relationships. He would be happy to discuss with you how to clarify responsibilities within your organisation. Simply contact him via his website or message him on LinkedIn.
We also highly recommend his workLIFE Stories blog, where he regularly shares insights from the business world, ideas and inspiration.
[1] Ofqual: Grading gaps in summer 2020: who was affected by differences between centre assessment grades and calculated grades?
[2] Reuters: Amazon scraps secret AI recruiting tool that showed bias against women
[3] Court of Justice of the European Union: Judgment in the SCHUFA case C-634/21 of 7 December 2023
Would you like to support Guido Bosbach and discuss decision-making and responsibility? Then share this post on social media or within your organisation.