Agile documentation in software development

“We’re agile now – we don’t document anymore!” Or: “The source code is enough for documentation.” I have often heard such or similar statements from software development teams.

This can work well for the development of manageable systems with a shorter lifecycle. The same applies if the team that built the system is also available after the system has been put into operation to answer any questions users or system administrators may have.

The development of complex systems with a long lifespan, for example in space travel, is becoming more difficult. Software systems in this field are in operation for more than ten years and have to be maintained and serviced over the entire period. Many of the people involved in the original implementation may have left the company by then or may be working on other projects, so that their support cannot be obtained at short notice. If the documentation is only available in source code, people not involved in the project need to get familiar with the code first. If errors occur in the live system, this can cost valuable time.

But even if the original developers are still available after several years, that doesn’t mean they still understand the source code as well as they did the day they wrote it. Even if all the information is somewhere in the code, developers need to know exactly where in the source code they can find it. According to Carola Lilienthal, a developer is able to oversee about 30,000 lines of source code – but most systems today are much larger.¹

Of course, a well planned package structure and speaking namespaces can make orientation easier. But what about support staff, for example, who may not have access to the source code or are not proficient in reading source code? Without supporting documentation, support often has to turn to the developers who are torn out of their current work and have less time for the development of new features. The use of a SWAT team may help, but even SWAT team members do not have unlimited mental storage space to remember all the details about how a software works for years.

Documentation in one’s own interest

Not documenting at all usually costs more time in the long run than is saved at the beginning by dispensing with documentation. So it would theoretically be in the development team’s own interest to write documentation outside the source code, wouldn’t it? Unfortunately only theoretically. The creation of documentation in software development (with a few exceptions) is rather an unpleasant activity. A “necessary evil”, so to speak, that one only cares about when there is no way around it. If nobody asks for documentation, it can go well for a while. At least until the stated bugs appear in the live system and everyone is trying hard to decide which service was again responsible for what.

If the customer or even the legislator demands the corresponding documentation in the course of a software delivery, “documentation lazy” teams sooner or later start to sweat. Some of these teams run “documentation sprints”, in which all team members only document at the end of the project – often over several weeks. From my own experience, I know that this is usually an extremely tedious undertaking. For very complex system processes, a few weeks or even days are enough to forget details, for example about process sequences. Reconstructing these from the source code is often tedious and error-prone.

So it’s most probably a bit like with the good intention to do more sports with the documentation: We all know that sport is healthy. Nevertheless, we are often hindered from moving more by our inner weakness. When the bad conscience is big enough, we try to compensate it with long and intensive training sessions. A severe soreness of the muscles occurs promptly, which then has to be cured on the sofa. Wouldn’t it be better to exercise regularly and moderately instead?

Documentation as part of the development process

But how does a team manage to outsmart the “documentation slob” and make documentation a good habit? The best way to do this is to integrate documentation creation firmly into the development process.

Some teams include the documentation as a separate item in the Definition of Done. Before a ticket is marked as “Done”, each team member has to check if the documentation of a feature or a user story is sufficient. If necessary, he or she must create or supplement the documentation. It goes without saying that the Definition of Done must be known to all team members for it to work. Unfortunately, as a coach I have often experienced developers saying: “We have a Definition of Done – but I don’t know exactly what it says.” To make sure that doesn’t happen, it’s best to store the Definition of Done in the project wiki or hang it out in the team room.

If the team wants to make sure that they don’t forget to create documentation during development, they create a sub-task called “Documentation” in the sprint backlog for each ticket to be documented. This makes the topic of documentation even more explicit.

As much as necessary, as little as possible

The point in time when the documentation should be created has been clarified. But what about the scope of documentation? Just as excessive sporting activity is no more beneficial to health after a certain amount of training, so is documentation. Too many documents can be quite disadvantageous, because the maintenance expenditure rises. If documents are not kept up-to-date, this is even worse than if there were no documents at all. Outdated information in documents leads to misunderstandings between the project participants and with each discovered error the confidence in the documentation dwindles. And it’s a real pity if for this reason nobody reads the documentation anymore, in the creation of which once a lot of work has been invested!

When documenting in software projects, the motto should be “As much as necessary, as little as possible”. The meaning of this varies depending on the project context and must be agreed in the team with the product owner and/or the stakeholders. Therefore only suggestions can be given in the following.

Typically, most projects require at least the following documents:

• user manual
• operating manual
• architectural document
• Interface specification(s)
• release notes
• test case specification(s)

In addition, it is recommended to create a document that provides a rough system overview. This document should contain the product vision, the most important requirements and/or user stories, the system context (what is the scope of the system to be created and to which systems is it connected), the stakeholder analysis, any personas defined and the glossary. Sometimes it also makes sense to document the business processes underlying the system separately (e.g. through use cases) if these cannot be derived from the user manual.

As far as possible, you should use standard templates for the documentation so that you don’t have to reinvent the wheel every time. For the documentation of software architectures the free template Arc42 is widely used, for the system overview one can be inspired by the specification templates Volere or IEEE 830. Unfortunately, the complete versions of these two templates are no longer for free, but the linked Wikipedia articles give a good overview of the template structures.

Documentation with the lean philosophy

Even if standard templates from more conventional project contexts are used, agile teams should always take care to keep the documentation lean. This is where the lean philosophy can be applied, in which activities that do not add value should be avoided as far as possible. If you transfer this to the documentation, you only document what you still need during later system operation. Documents that are only needed during the development period are avoided as much as possible.

In the case of lean documentation, for example, no separate concept document is created for the graphical user interface. This type of conception can be carried out within the framework of the user manual. The current status of the system is then not only recorded, but also the target status of features that have not yet been implemented is described in the same document. Then, for example, already completed input masks with screenshots from the current release appear, while planned input masks are defined by corresponding wireframes or scribbles. The corresponding backlog item for the implementation of the new input mask refers to the corresponding chapter in the user manual. After the planned input mask has been implemented, the development team only has to exchange the wireframe for the screenshot of the finished mask and, if necessary, update some details that have changed during development.

This approach can also be used to describe test cases – this approach is known as Behaviour Driven Development (BDD). Test cases are written before implementation and used as requirements at the same time. If a structured format such as “Given-When-Then” is chosen and this is done with a tool such as Cucumber, automated tests can be generated directly. This way you can kill several birds with one stone and save further development time.

Handling documents like source code

There is a challenge when a document simultaneously contains descriptions of the actual and target status: versioning. Finally, the documentation status for a corresponding software release should be available at any time (without target status descriptions for a future release).

The “Docs as Code” approach, in which documentation is created with the same tools as the source code and versioned together with it, is promising in this context. This allows the documentation to take advantage of all the features of modern version control software, such as Git feature branches. The documentation is usually created in a simple wiki markup language that even non-developers can quickly learn. Graphics can be integrated without problems and with the appropriate tools such as Asciidoctor the documentation can be exported from the development environment into various formats such as Word, PDF, html and Confluence.

Bottom line

Agile teams should also document the systems they have developed outside the source code – in such a way that the documentation effort is kept to a minimum and the documentation is also read. A team can make this type of documentation a good habit by integrating documentation tasks into the development process via the Definition of Done or dedicated subtasks of features or stories. If the motto “As much as necessary, as little as possible” is followed, nothing stands in the way of agile and successful documentation of software development.

Notes:

[1] Check out Lilienthal, Langlebige Softwarearchitekturen, 2. Auflage, Heidelberg, 2017, page 11

Kathrin Herrmann has published additional posts in the t2informatik Blog, including

t2informatik develops and modernises software as a service provider. Click here to learn more.