IEEE Software - The Pragmatic Designer: Fix Technical Debt with Virtuous Cycles

Dec 1, 2022 | George Fairbanks

This column was published in IEEE Software, The Pragmatic Designer column, March-April 2023, Vol 40, number 2.

ABSTRACT: In recent years, teams have found it easy to quickly deliver a working system, but increasingly hard to deliver new features because of tech debt. Tech debt arises from what teams do – and more importantly, what they don’t do – each day.

Teams can use virtuous cycles to produce great code and keep improving it. Virtuous cycles can be found at the scale of methods, modules, and systems. With minimal effort, developers can slow the buildup of tech debt by shifting their perspective on development activities from a checklist to virtuous cycles.

In recent years, teams have found it easy to quickly deliver a working system, but increasingly hard to deliver new features because of tech debt. Tech debt arises from what teams do each day – and more importantly, what they don’t do [1]. If current team practices create tech debt, then different practices can avoid or repair it. My teams have done exactly that by using virtuous cycles.

You can avoid and repair tech debt by shifting your perspective on writing code from a checklist of obligations to a virtuous cycle. Virtuous cycles exist at different scales: when building a method, a module, and a system. They are the core of every continuous improvement process. Whenever you reflect on what you’ve done and decide to change your ways, you are in a virtuous cycle. I also see it in the writing about agile software development, especially when the author talks about the spirit of agility.

Virtuous cycles are a feedback loop that improves code over time. As a result, a lot of tech debt can be avoided with frequent minimal effort, in the vein of gardening, not rewrites. Here, I offer a sketch of some virtuous cycles. It is not a full development process, so I expect that it will work on many software projects exactly because it is incomplete.

The problem

Today, many teams endlessly repeat a checklist like this: pick up the highest priority feature, write a test for it, write the implementation, and maybe refactor – all while conforming to a coding style guide. Systems grow organically, feature by feature. You might think that what distinguishes this from older processes is the addition of testing, which became mainstream in the early 2000s, but it is instead the radical idea that such a short checklist might possibly work [1].

A few decades of experience shows that this short checklist works, but there’s a catch. It works in the same sense that doctors can perform operations without washing their hands. Such success is short-lived, as patients suffer from infection and software projects suffer from technical debt. So, the short checklist works, but the price of radical process simplicity is, ironically, the buildup of project-crushing complexity, which I call sedimentary development [2].

Organic growth rapidly creates complexity. Since there was no master plan, when you read the code you will not find consistency. There will be no cross-cutting principles or decisions to aid your reasoning. To understand it, you must memorize all its quirks.

As Tony Hoare said nearly half a century ago in his Turing award lecture:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

Organic feature growth quickly exceeds the intellectual control of its authors, so they settle for statistical control through testing. A simple checklist is a kind of statistical control, where the tests ensure that specific cases work. What is lacking is intellectual control – an understanding that developers hold in their minds that convinces them that the system works in the general case [3]. Intellectual control is possible only when the design is simple.

If refactoring alone were strong enough then teams would not still be suffering from tech debt. Refactoring can slow and even repair tech debt, but not when refactoring is a step in a checklist. People use checklists grudgingly. They admit that they make mistakes and that checklists can catch those mistakes, but emotionally they want to zip through the checklist as quickly as possible and return to fun activities like building more features.

Vicious and virtuous cycles

A vicious cycle is a chain of events that reinforces a bad outcome. Here’s how a checklist becomes a vicious cycle: You pick up a feature request, write some code, and write a test. Sometimes you refactor. You feel pressure to be productive, so each day you try to zip through this checklist a bit faster. Your manager cannot measure tech debt or code complexity, but can see all that code you are cranking out. As weeks pass, it becomes harder to add the next feature because it’s hard to understand the code it sits on. But you want to keep improving productivity, so you refactor less often. You test fewer cases. You use the first thing you think of. The more you seek productivity, the more you compromise quality, which leads you further away from productivity.

Increasing productivity means getting more done in the same amount of time, so you cannot improve productivity by slowing down. But speeding through a checklist ends up reducing productivity and slowing down also reduces productivity. Is this not a paradox?

You would be right in thinking that the answer is a virtuous cycle, which is a chain of events that reinforces a good outcome. As many have observed, writing a test or refactoring code creates an opportunity for reflection. The tests and the refactoring are themselves good, but the real driver of a virtuous cycle is the reflection and the opportunity to course-correct that follows. The virtuous cycle starts when you ask “can I do better?” and then improve.

Reflection leads you to fix your first attempt. It leads you to recognize trouble piled on top of trouble. At those moments, the checklist transforms into a virtuous cycle. You return to writing code with new insight, you revisit your bureaucratic tests with a way to lighten them, and you refactor beyond superficial code de-duplication. When you follow that virtuous cycle, productivity improves as you avoid and repair tech debt a little bit each day.

You may be skeptical. Consider any team that has good testing practices. Ask them if they would be more productive without their tests. They will tell you: of course not. But doesn’t writing tests take time away from coding? Yes, but because the team is in a virtuous cycle, the time investment pays off immediately. If you are still skeptical, please delete the tests on your current project and let me know if your productivity improves.

Here’s an overly tidy way of summarizing this idea. When you try to improve productivity by working harder, you encourage a vicious cycle; when you work smarter, you encourage a virtuous cycle. The virtuous cycle guides you to keep the code as simple as possible, which preserves your intellectual control, so feature requests are easy to reason about and incorporate.

In my experience, the virtuous cycles I’ve used aren’t strong enough to keep a system healthy forever. If you imagine two teams, one using a checklist and the other using virtuous cycles, you’ll see the virtuous cycle team consistently being more productive and its system has a few more years of useful life. Let’s see how that works at three scales, starting with methods.

Method virtuous cycle

Let’s work through an example of building a method using a virtuous cycle (See Figure 1). Revealing the virtuous cycle would be easier if we could work on a project together for a few hours, staring at the same code. Let’s acknowledge the awkwardness of text and do our best. Assume we are working on a first draft implementation of a method, where c is the ID of a customer and a is the ID of an account:

bool isOwner(int c, int a)

We have already written another method:

void setOwner(int c, int a)

The first step in the virtuous cycle is to write a test for the success case. The test calls setOwner(), then asserts that a call to our new method – isOwner() – succeeds. So far, there’s no opportunity for insight. Nothing we’ve done encourages us to revisit the body of the method. Some developers would stop now.

The second step is to write a contract for the method that explains what a caller cannot know from the method signature, such as: “Returns true iff customer owns the account.” (“iff” is shorthand for “if and only if”). We’d write this in a comment to document the method. Most callers would have guessed this is the contract. Again, there’s not much opportunity for insight so let’s move to the next step, which is interesting.

The third step is to consider error handling. Our test case works because we call setOwner() before isOwner(). What happens if we omit that? Let’s write a failure test and assert that isOwner() returns false, as the contract says it should.

We are surprised when this test fails: Instead of returning false, the isOwner() method crashes with an error about an index out of bounds. For brevity, the body of isOwner() isn’t shown here, but let’s say that we notice the customerId parameter is used as an index into a table of customers and accounts. Aha! In the success case, setOwner() configures the table, but our new test omits the call to setOwner(), which triggers the crash.

Instead of rushing to refactor the body and re-run the test, let’s reflect on what we’ve designed. The isOwner() method can tell callers about ownership (the boolean return value) but not about errors caused by unexpected parameters or conditions. We’ve designed a loaded question where the isOwner() method is forced to assume its parameters and conditions are valid, but in practice they might not be.

Let’s revise the contract: “Returns true iff customer owns the account and parameters are valid IDs.” This helps because it warns callers that IDs can be invalid. It doesn’t say what happens though. The method is now underspecified: it’s a partial function. Methods like this are a burden on callers, who must ensure that the IDs are valid before calling isOwner(). Remember this burden, as we will revisit it.

Writing a contract takes seconds, so that has no effect on your productivity. Pondering and deciding a contract does take time, but you must spend that time when writing the test anyway. Before you could write a test case, you had to decide the expected behavior and avoid testing the method’s implementation details. That’s the contract. When you spend time deciding a contract, you feed a virtuous cycle, leading to tests that cover all cases, methods that are simpler, and callers who find the method easy to use.

The fourth step is to consider typeful programming, which is the pervasive use of types that can be checked by the compiler. So far, the isOwner() method takes two integer parameters. Callers could accidentally transpose them yet the compiler could not catch that mistake. You can help the compiler by creating two new types, CustomerId and AccountId. The signature becomes:

bool isOwner(CustomerId c, AccountId a)

Not only does this make transposing the parameters impossible, it also feeds our virtuous cycle. Recall that the contract burdened callers with ensuring that the IDs are valid. Now that the IDs are their own types, you can enforce ID validity when the types are created, which removes the caller’s burden. That means we can simplify the contract, so it is again simply “Returns true iff customer owns the account.”

You could add more steps to this virtuous cycle, such as looking for chances to make data immutable and functions total. The goal of this detailed example was to reveal the difference between a checklist and a virtuous cycle. Next, let’s look at virtuous cycles for modules.

Module virtuous cycle

Modules consist of hundreds or thousands of lines of code, so they are too big to show here, let alone show them evolving as you work through a virtuous cycle (See Figure 2). To make that cycle apparent, I’ll resort to metaphors and general descriptions.

What is a module? A module groups together a bunch of methods and creates a distinction between inside and outside. (Here comes the metaphor). Consider your kitchen. Your refrigerator is a kind of module that you can use to keep food cold. It has an inside and an outside. Inside, there are a bunch of parts. Outside, it interacts with the world in limited ways: the electrical plug and its doors. You can understand how to use your refrigerator without understanding all the parts inside it.

Some modules are better than others. Your kitchen likely has a drawer where you keep miscellaneous tools. Both have an inside and an outside. Both contain parts on the inside. Your refrigerator, however, is a better module. The drawer cannot be understood except by understanding each thing it contains. When you build source code modules, you prefer ones that can be understood simply, like a refrigerator, not an arbitrary bag of parts that must be mastered individually, like the drawer.

In source code, organic growth leads to complexity, bit by bit, like tools accumulating in your miscellaneous drawer. Virtuous cycles tame complexity, guiding you to group methods to hide inner complexity while presenting a simple interface.

The virtuous cycle for modules has familiar activities including writing methods, tests, and contracts. At the scale of a module, however, you use them differently than when writing a single method. When writing tests for a module, you have an opportunity to evaluate the module’s interface. You can scrutinize what you’ve kept inside versus allowed out across its interface. With each test, you are looking for secrets leaking out of the module.

One example I’ve seen many times is that implementation details leak out via exceptions. When you are testing error cases and the test must catch a specific exception (such as PostgresException or OracleException), that means users of your module now depend on your technology choice. This is a failure of what David Parnas called information hiding. Better information hiding means changes to one module don’t ripple and force changes to its neighbors.

The virtuous cycle for modules also includes contracts, but, compared to methods, the stakes are higher. The contracts on module interfaces are often used by people not on your team, perhaps not even at your company. They might not be able to read your source code to figure things out, so the contract is their only insight into how the module is intended to work. Writing the contracts helps you build a better module interface. Modules with clear contracts can, if necessary, be rewritten. Rewriting a module with fuzzy contracts means chasing quirk-for-quirk compatibility, which is so hard that you rarely ever attempt it.

The virtuous cycle for modules also includes new activities, like evaluating dependencies. Some of your modules will be leaf modules that depend on no other modules. Fewer dependencies leads to easier testing and reuse, so you should be looking for opportunities to prune dependencies.

To build good modules and fight creeping complexity, you must keep tinkering over the life of your system. It’s easy to create bad modules, ones like that drawer in your kitchen instead of a refrigerator. Activities like testing, writing contracts, create opportunities to evaluate the module as a division between inside and outside, a division that keeps some secrets inside while presenting a useful interface to users.

System virtuous cycle

At the scale of an entire system, the virtuous cycle shifts your focus to quality attributes like latency, security, modifiability, or reliability. All systems have architectures, whether chosen consciously or not, that promote or even ensure certain qualities. The virtuous cycle guides you to an architecture that is best suited to the qualities you prioritize.

As before, this is just a sketch of the virtuous cycle (See Figure 3). And again you start the cycle with testing. Writing an end-to-end test that shows the whole system working for a simple task forces you to use all of the modules in the system. If you pause at this moment, you will notice that setting up this first system test is painful, difficult, and fragile. Take the opportunity to remove some friction, to make the next test less painful to write, to make the test setup less likely to break.

Next, look for inconsistency across the modules in your system. That could be in the vocabulary of types used in module APIs: perhaps you can remove unnecessarily different types used in those APIs, which makes testing easier and removes corner cases. Look at how errors are signaled and handled across the modules because unnecessary diversity leads to mistakes. Look for places where developers must be vigilant in their coding practices, as architecture hoisting can reduce that burden.

Then evaluate how you have partitioned your system into modules. Perhaps there is a better way to divide up the code. Perhaps a single responsibility has been smeared across many modules, leading to unwanted coupling. As you change code, does your edit touch a single module, or ripple out into its neighbors? Ripples are a sign that the partitioning into modules needs attention. Also look for standard problems, such as parsing, validation, and loading/storing. Align your modules with these problems instead of entangling them.

As your system grows, it’s increasingly important that it has a hierarchy of modules and a pattern of organizing them. You may start out with a neat tree but don’t expect that to last. There’s no single right way to organize modules, but neglect quickly leads to tangled dependencies.

The final step in the virtuous cycle is to re-evaluate your desired qualities, accepted trade-offs, and chosen architectural styles. If you haven’t prioritized which qualities you need most then you are giving up the opportunity to choose a style that matches your needs. It’s expensive to change architectural styles, so, in the early days of your system, this step is critical because if you need to change, you’d prefer to do that before the system is huge.

A systematic approach

The Boy Scout rule says to leave the campsite cleaner than you found it. How exactly do you do that? In software development, you can use virtuous cycles at three different scales: a method, a module, and a system. The steps here are ones that have worked for me in a career spent in application development. If you build device drivers or self-driving cars, you might have different steps, but you can find your own virtuous cycles.

Some teams perform better than others even when doing similar activities under similar conditions. I think that the higher-performing teams are linking those activities into virtuous cycles, not speeding through tasks on a checklist. Each step in a virtuous cycle gives you a different perspective on your work and an opportunity to notice a better solution. That opportunity is critical. Use it to recognize and repair tech debt, often before your code reaches production.

Some people think that seeking productivity this way is too expensive. Can teams instead seek productivity by being scrappy and simple? History says no. The most productive people aren’t amateurs with a checklist, they are reflective experts who are continually improving. It takes practice and reflection to make something look effortless.

References

1. G. Fairbanks, The Rituals of Iterations and Tests, IEEE Software, Vol 37 number 6. November-December 2020.
2. G. Fairbanks, Ur-Technical Debt, IEEE Software, Vol 37 number 4. July/August 2020.
3. G. Fairbanks, Intellectual Control, IEEE Software, Vol 36 number 1, January/February 2019.