Lessons learned adding a second line of business to a 20-year-old application
At some point in your career you’ll be tasked with a project that will strike fear into the very core of your soul.
It will require you to do things you’ve never done before, to make choices for which you have no former wisdom to rely on.
You’ll pour endlessly over ancient programming lore, online tutorials, articles, and blog posts, hoping you’ll find someone, anyone who has gone through the same scenario and can tell you exactly what to do.
And you won’t find it.
You’ll be all alone.
And if you don’t get it right, if you fuck it up – you’re toast.
I just came out the other end of this very scenario, alive and unscathed, and I’d like to shine some light on the topic in which it was born.
With the proper strategies and mindsets, you can be confident that you can handle this particular kind of problem. We’ll take a look at the problem itself, some of the strategies available to you, the path my team chose (and it’s present day result), and finally review the main lessons learned.
This article won’t cover the technical, code-level details of how it was done. If you’re interested in that part, jump on over to part 2.
The Problem: Adding a second line of business to a 20-year old codebase
If you work for a company that is doing well, it’s common that it will expand into new markets.
For example, if you work for an insurance company that sells car insurance, it is not inconceivable that your company could eventually sell RV, boat, life, rental, or home insurance.
The timeline of when this branching out might occur could be within the first year of business, all the way until 100+ years down the line.
To understand the problem my team encountered, you can imagine that the software we work on has served only one line of business for 20 years. For the sake of example, let’s continue with the insurance example and say that this is strictly a car insurance business.
Architecturally-speaking, you have 20-years worth of components, classes, methods, and variables, all modeled around car insurance.
This sytem models things like automotive insurance policies covering car accidents, based on things like vehicle age, make, and model, insured driver history, where the car is stored, and how it’s used. Things like a policy’s status and lifetime. Recurring or one-time payments. All the accounting data for these transactions. This is your typical line-of-business application that has grown to support all the different departments in the company.
And since this software has been supported for 20 years, the business language and concepts can get heavily embedded, and on top of that it’s usually got a lot of legacy baggage being dragged along with each new release.
Now, after 20 years time, your car insurance company decides it’s going to start selling home insurance.
A new line of business.
Home insurance is of course still within the overarching business of insurance, but it is also quite different. Despite both businesses dealing with the act of of insuring, they have grown and developed separately, so the terminology is not always the same, the properties, statuses, and aging of policies is vastly different, and the way the insurance can be paid for follows substantially different rules.
About the only thing that is similar is the fact that they involve something called an insurance policy, and that there are entities like the people being insured (insureds), insurance companies, and agents.
For now, the company can get by doing things the old fashioned way with spreadsheets and file shares, but they’re looking to you to come up with something better in a relatively short amount of time.
How you accomplish this is entirely up to you, but there are a few requirements, and some obvious conditions that you’ll need to weigh carefully.
Constraints and other mentionables
- The company needs something better than spreadsheets, relatively quickly (less than a year, preferably sooner).
- This system will most likely need many of the same features as the existing system, when it comes to things like reports, mailing notices, and accounting data. It won’t need these things on day 1, but eventually.
- The company has found a way to partially perform some of the home insurance business work using the existing system by pretending car insurance policies are home insurance policies, and various other mental hacks.
- Your team is small, 1-2 developers, no QA, and you need to keep supporting the existing line of business.
- Obviously don’t break the existing system. It runs the entire business.
Weighing Our Options
After thinking about the problem and it’s constraints for awhile, we determined that we had three practical options.
Option 1 (a.k.a New Toys) - Create a new application from scratch.
Option 2 (a.k.a Safe Duplication) - Branch the existing app source code into a new app and heavily modify it to work with the new line of business.
Option 3 (a.k.a Obvious But Hard) - Modify the existing app to support the new line of business side-by-side.
Each option had a plethora of positives and negatives, and even to this day I wonder if one of these options would have been better than the one we chose.
Next we’ll take a look at the pros and cons of each option and play plinko with the constraints until we end up with the decision.
Option 1 (New Toys) - Create a new application from scratch.
After years of working on ASP.NET Web Forms applications, and only getting to play with ASP.NET Core on the side, this option was obviously very appealing.
A new app would not disturb the existing app, it would allow us to take advantage of faster development (and deployment) practices, and it could become the training ground for all future development at the company, freeing us from the dying branch that is .NET Framework.
I even recall taking a few nights of my own time to whip together a starter app to work from, transferring many of the existing tables and POCOs into equivalent Entity Framework code-first versions.
It wasn’t all puppies and candy though. Building this as a new app would be a lot like rewriting an app from scratch (something that is rarely a good idea.)
This new system was going to share many of the existing features from the other system, so we’d need to duplicate a ton of it or find some way to share it via APIs. With that comes lots of testing. Testing that in theory was already done for 20 years to battle-harden the existing system. All those iterations of testing and bug fixing have to be redone for any features you duplicate into the new app.
Then comes maintaining this app with separate releases, and a host of other problems that come with having a small team working on multiple systems.
I think to do this option we would have either needed a dedicated team, or much more time. I would have liked this route, but in the end, there was way too much meat and potatoes in the existing app that we would have missed out on, and so we passed on it.
Option 2 (Safe Duplication) - Branch the existing app source code into a new app and heavily modify it to work with the new line of business.
This option was more like converting a school bus into an apartment rather than a refactor.
We would branch the code into a new repository, and then play hack ‘n slash to convert all the car insurance behavior and appearance into it’s home insurance counterpart.
There were many benefits to this strategy. It would be safer because we’d leave the proverbial nest of our original application. Any bugs would be isolated from the main application, and that was quite enticing. All throughout my research I clung to this idea as a safety net because I was afraid my lack of experience with large refactorings would inevitably lead to a ball-of-mud situation.
Further, this option may have been the fastest to reach a version 1, as we’d only need to modify the quirks that the business already knew about by trying to force home insurance into the existing application’s domain. A lot of the code changes would be adaptations of existing behaviour and UI, rather than entirely new code.
The downside with a second branch is that it would heavily complicate deployment and the sharing of features and bug fixes.
Any bug you fix must be fixed and tested in both branches. If there are large differences between two areas where the bug exists, it may not be as easy as a git merge. It could require some finesse. And often you simply might not have enough time or resources (see above: small team).
The same is true for new features. Both systems may not be able to get them at the same time (or ever), so you end up playing a game of cat and mouse to keep them in sync.
Every fix and feature has to have the added conversation “Should we put this on the other system too?”.
If possible, you could pull certain features out into separate shared libraries, but again, you need time and resources for that, and the stickiness of a 20-year old system isn’t as welcoming to that path.
We did not choose to go this route, although to this day I wonder if we would have had a version 1.0 out the door months earlier. And as we implemented the option we did choose, this option was still appealing as a way to jump ship in case things started to get bad.
There were times where I felt very unconfident that it would turn out well, and the fear of creating a ball-of-mud kept this option in my periphery.
Option 3 (Obvious But Hard) - Modifying the existing app to support the new line of business side-by-side.
This option was the first to arise because the business folks had already explained that they were able to sort of use the existing application on top of a new database by playing pretend with the existing features. For example, because the system allows for custom car insurance policy coverages, they could simulate home insurance coverages and make due with the random vehicle and driver references in the user interface here and there.
In the end though, it was merely a familiar place to store some data, arguably better than an Excel spreadsheet, and it didn’t help automate any of the real home insurance business processes.
But because we could see it in action, it was clear that the existing application had a lot of similarities and usable meat and potatoes, so figuring out how to refactor it to support two lines of business seemed plausible, although daunting.
There was sooooo much car insurance jargon and concerns laced throughout this 20 year old juggernaut.
Using some DDD terminology, from a higher level we could see that the existing line of business and the new line of business shared the same primary aggregate roots in concept. I.E. both lines of business revolve around the lifetime of an insurance policy.
The fundamental policy quote and policy account aggregate root classes on the existing application acted as the glue, tying together all other entities and transactions, they simply differed in properties and behavior.
It turned out that there were a few different ways we could do this modification.
We could A) simply add all new code and create new interfaces for things we wanted to reuse from the existing system.
B) Refactor it so that the differences between car insurance and home insurance were abstracted away via techniques like the strategy pattern or plain old inheritance.
Or C) The rapid ball-of-mud-inducing option of modifying existing code via basic if-else statements scattered everywhere to accommodate differences in behavior and appearance.
Because I was absolutely terrified of turning this 20-year-old beast into a ball-of-mud, I was staunchly against option C, and rightfully so. But it is unfortunately an option many will take. In my desperate attempt to find guidance with this project, several developers even recommended it via something like “Hey man, the code is probably not that great anyway, just add a bunch of if-elses. NBD.”
Option A sounded nice in theory. Adding new code is always safer than modifying existing code. All living in one app would allow us to have simple deployments, and we could still share code and features, albeit needing to write new interfaces for them.
The downside of option A had to do with the fact that the fundamental car insurance policy quote and account classes were deeply engrained throughout the app via their primary keys.
QuoteID and AccountID lived in almost every table, every class, as a reference back to the primary policy it originated from.
If we were to create brand new classes for home insurance in order to keep isolated from car insurance, we would need new interfaces for almost every existing feature, meaning we wouldn’t really save the time we were intending by sharing.
We’d also have to sprinkle new IDs all throughout the existing tables and classes.
Yuck.
And what happens if our business expands again to a third line of business?
Option A was not scalable with the business in that sense.
Of the three options for modifying the existing application, option B had the most chance for success. Abstracting away the differences between car and home insurance, whilst maintaining the base concepts of policy quotes, accounts, payments, insurance companies, agent, and insureds, simply felt right.
If done correctly, it would make sharing code and features almost automatic, and would also support an infinite number of additional lines of insurance business. Those precious QuoteID and AccountID primary keys could be maintained and shared between any new line of business.
It seemed like a no brainer. It even fit all the classic examples of inheritance structures we learn about in computer science classes or software design books. Base class of insurance policy, child classes for specializations (car insurance, home insurance.)
Duh, right?
Although this option seemed fitting, I had zero experience taking on a refactor of this size. And when I encounter the unknown, it puts me into a kind of robotic dissociative state where my body becomes an afterthought to my mind’s rabid information thirst.
I consume everything I can on the topic to ease my discomfort of not knowing, hoping I’ll find some wisdom that points the “one true way”.
Massive fear and lack of experience was a solid barrier to this option.
The Chosen One
After matrix brain downloading endless books and blogs to overcome the dread of unknowning, it was time to set forth on a plan.
Ultimately, based on our small team and the 20-years of features and bug fixes we wanted to make use of, we decided to try option 3. We would set forth refactoring the existing application so that it could handle multiple lines of business, share code between those lines, and keep deployment simple.
We also laid down some guidelines for this project to ensure we wouldn’t break the existing system in the process.
These were things like:
-
Other than splitting the fundamental aggregate root classes into two (to share QuoteID and AccountID, remember?), resist modifying existing code and pages. Just about all business logic is specific to a line of business, so trying to reuse it would be high risk and an if-else nightmare. If it has the smell of car insurance, avoid it if at all possible.
-
Similarly, always lean towards creating brand new code, pages, stored procedures, dependencies, etc. New code is low risk because it doesn’t interact with existing code.
-
There will be times where existing code can be shared and may need an adjustment, if this is the case, do so with great caution, write tests beforehand, and be confident it is truly a shared business concept. Never try to shoehorn an existing feature to work in the other line.
Further, while I tried my darnedest to use delegation over inheritance, I ultimately couldn’t make it work the way I wanted to, so I went for a more traditional use of inheritance to break our aggregate roots apart.
This meant converting two massive 10,000+ line god objects into abstract base classes, extracting two line of business subclasses, and very slowly pushing down all variables, properties, and methods that were exclusive to each subclass, while leaving any metadata-like shared property and behavior in the base class.
This was daunting and resulting in a lot of git resets where I’d start over from scratch to get a cleaner run. Keep an eye of for an upcoming article that details the strategy for this large and risky undertaking.
At first we did this refactoring gradually, meaning that we made use of the new line of business subclass even before all the existing line’s concerns were pushed down from the base class. This was an unsettling experience because it meant any developer could make use of (or be confused by) the large assortment of inappropriate members being offered by our code editor’s autocomplete.
Fortunately though, with a small team, we could communicate this risk and tip-toe around it while we iterated quickly on the new system.
After enough bumps and bruises, we finished up the last of the refactoring, resulting in two lines of business sharing the same codebase for things like payments, accounting, email notices, and entities common to both worlds of insurance.
Key Takeaways
You may find yourself in a situation like mine, but fear not, it is surmountable! Below are some high level key takeaways from this project.
1) Get your whole team on board so everyone knows what’s up and can share the decision.
Don’t put the burden entirely on your shoulders, even if you are most qualified to handle it. Keep everyone on the same page and unified by analyzing and sharing the various options. This project seemed a whole lot more intimidating when I was trying to tackle it all on my own.
When I brought my other team members into the picture, they were able to see things more realistically, rather than exclusively through my constricting developer perfectionism.
2) Benefit from the collective wisdom (books, articles, people).
In times like these, when you simply don’t know what to do and don’t feel confident, you can assume someone has encountered a similar problem and created a book or article about it.
In my case, I couldn’t find anything regarding whether or not it’s even a good idea to split up a large system in this way, so I had to go about asking real people their opinions. Combining that with the excellent refactoring books from folks like Fowler and Kierevsky, I was able to make gradual progress forward.
3) Understand and accept that you can’t always know if it will be the best decision until much later.
With large systems, it’s very hard to know if the path forward will end up fruitful without actually using it for some length of time.
Do your research, come up with a plan B and C in case it goes to hell quickly, and then march forward. Software is malleable and even with large decisions like these, you can work your way out if you have to.
Put your perfectionism aside and accept some amount of messy, unfinished business.
Is software ever really done anyway?
4) Remember that software is, by it’s nature, malleable. You always have the option of taking things in another direction.
Yes, it may be more difficult after some changes, but it’s not impossible. I found it was good to remind myself of this to help quell my fears and to keep moving forward. At any time if things started to look fishy, I could jump ship and branch the code into a separate app (I still could!).
Take your time and ideally do small releases so issues are caught fast.
Break the work up into small tasks and don’t worry about following patterns to the T, software can be a bit messy, especially very old systems.
5) Jump in and begin despite your fear.
I know it sounds a bit cheesy in the context of writing code, but you will ultimately have to just start. I had to accept that I wouldn’t be able to find a resource that laid out exactly what to do or told me if I was making the right move. I simply surrounded myself with the collective wisdom and then made tiny adjustments forward in a direction while communicating heavily with my team.
Refactoring can be done in small bits, it doesn’t have to be all at once. With our project, we went several months with half-divided quote classes that interfered with each other but were able to handle it because our team is small.
6) If you have a small team and a short_er_ deadline, consider sharing the existing system’s foundation.
Without a lot of team bandwidth or time on your side, you’ll want to lean on all your strengths and eliminate excess. By adding a line of business to your existing app, you’ll benefit from years of foundational code, bug fixes, and testing, and you’ll only have one application to deploy and manage.
7) If you have a large team and/or plenty of time, consider a new application or branched application.
With plenty of hands to spare or no impending deadline, you can benefit from the conveniences of modern technology. New platforms are great for shrinking dev and deployment time by reducing boilerplate code and being designed for easy cloud integration. If the new business line is unique enough, it may be worth building on a new foundation.
If you truly want to share existing features, you have the option of making them available via APIs or shared libraries. Just remember that the network is still the slowest part of a system.
God speed and good luck!
In part two I’ll share the tactical, code-level steps and order of execution that I used to make this line of business halving a manageable task.
My hope is that if you ever encounter this kind of problem you’ll be able to reference it to make it much easier rather than binging on architecture books like I did.