The agile migration of applications
In our dynamic and complex world, software development projects can only be planned to a limited extent, because requirements often change, new ones are added and existing ones lose their significance. So it makes a lot of sense to develop applications iteratively and put them into operation. This works very well in many companies when developing new software, but how does it work when replacing existing software? What is the so-called Minimal Viable Product for the replacement of an application and how does the agile migration of such an application work?
The MVP in the replacement of an application
Agile software development generally proceeds in steps. First the minimum viable product (MVP) is realized. It represents a small, already functional set of the software to be developed. Based on this, the software is developed iteratively, always with the idea that it should be fully executable and productively usable. Through this step-by-step procedure, the software can be developed very purposefully on the basis of the current customer requirements, thus creating real benefits for the customer at an early stage.
When replacing an existing application, however, the situation is different: The application has already been developed and customer needs and requirements do not have to be determined first. Since most applications are an active part of an application landscape, it is very difficult to start with a minimal viable product and extend it iteratively. In addition, when replacing an application, you must ensure that the existing functionalities are also available again with the new application. The interfaces to surrounding systems are also often indispensable for the functioning of the overall processes.
So is an iterative approach inappropriate when replacing an existing application? Could even a classical replacement with the waterfall method be more suitable? In fact, a classic project management approach is suitable for replacing an existing solution, because many requirements are already fixed before the project begins. But how can the benefits of the new application be realised at an early stage and not just at the end of the project? In order to proceed iteratively in a migration project, you must identify function blocks that are executable in their own right and can be developed on the basis of one another. This is the intellectual challenge of agile migration.
In an agile migration project, you pursue the idea of running the first versions of the new application parallel to the existing application. Use cases that can be realised with the first blocks of functionalities run on the new application instead of on the existing one. More complicated use cases remain on the existing application until the necessary functionalities are available in the new application. You develop these iteratively according to the function blocks and make them continuously productive. In this way, you can iteratively migrate use cases and benefit from the benefits of the new application at an early stage.
Of course, this procedure requires that you think about the parallel operation of the old and new application. For example, you need to consider what happens if a previously simple application case has been migrated, but then unexpectedly becomes a more complicated one that has not yet been implemented. And what happens with data dependencies in the new and old application? How you solve such challenges can only be clarified on a case-by-case basis.
An example of agile migration
I would like to tell you a concrete example from my work at idealo, Germany’s largest price comparison and shopping portal. idealo imports around one billion offer data from online retailers every day. Since the amount of data increases continuously every year, it became clear that the database used for an import application would soon no longer be scalable. The database had reached the limits of its capabilities and the development of a new application with optimised import logic became inevitable. As a special challenge, we had to realise countless, complicated functionalities for individual dealers, which were additionally embedded in various processes. It was not clear when the overloaded database would give up, but this disaster scenario had to be avoided in any case.
How did we proceed with the agile migration? We started to cut application functionality into blocks. This was the biggest challenge for us. We decided to separate the functionalities according to their use by different dealer groups. For example, there are German merchants who also ship their goods to Austria, so that their offers including the corresponding delivery conditions and the applicable value added tax had to be available on both the German and the Austrian idealo side. The implementation of this functionality was planned for a later function block. In return, we were already able to migrate the German retailers to the new application, who ship their goods within Germany but not to Austria.
A further functional block revolved around variants of offers, such as mobile phones, which can optionally be sold in different colors and with different memory configurations. The import of variants was also planned for a later block of functionalities. Previously, we were able to import dealers who did not use variants in their offers.
The test automation
With approx. 560,000 dealers, it is not that easy in practice to identify the required function blocks of individual dealers. What would happen if a retailer, who had previously worked without any offer variants, wanted to use some after all? Perhaps we had already migrated his data to the new application, which, however, could not yet avoid dealing with variants. To overcome such challenges, we invested about a third of our total efforts in testing and automating them.
The actual migration process was fully automated. Our setup included two test tracks: the old and the new import track. Each dealer was processed in parallel via the old and the new import and the results of both imports were compared. If there were no inexplicable deviations, the dealer was automatically switched to the new import application. In this way, each retailer was automatically migrated as soon as the necessary functionalities were implemented.
Was this iterative approach worthwhile with the extensive testing effort? Yes, in any case. Through the earliest possible migration of dealers, we were able to relieve the existing, business-critical database and thus significantly reduce our operational risk. In addition, we were also able to put new features into operation at an early stage, such as the individual control of the import frequency per dealer. And we were even able to identify technical errors in the old application, so that the quality of the quotation data was also increased at an early stage.
In my opinion and experience, it is worth considering an agile, iterative approach when migrating complicated applications. Finding the suitable cuts for the iterations is a challenge, because many functionalities and interfaces must be available right from the start. The art lies in subdividing the use cases according to the complexity of their functional requirements. In this way, function blocks can be found that can be implemented one after the other in iterations and that allow the migration of a subset of the data and use cases.
Of course, the parallel operation of the two applications is an additional challenge, but with a good test automation you can cope with this. I can’t say exactly whether the total effort would have been lower with a waterfall migration approach. However, the benefits realised much earlier and the significantly reduced risk compared to a big bang migration speak clearly for an agile migration.
Jan Hegewald has published another post in the t2informatik Blog: