The Real Value of Writing Automated Tests

The product manager asks in the standup, "Why did this start to fail? Don't we have tests that cover this?"

"Yes, we have tests. They all passed," we respond.

"Then your tests aren't very good."

Well, maybe. It's true, we could write a lot more tests. In general, if your software team is committed to automated testing, you'll write at least as much test code as application code. Project estimators should plan on writing two applications - the one with all the features, and a second one to test the first one.

In our case, the unit tests were all passing, but some of the integration tests started failing. These are the tests that verify the interactions among the various parts of the application. Of course, each time we find a new situation where a test fails, we try to figure out why, and cover that scenario with yet another test. In this way, what evolves is a system that informs us when our software parts interact in ways we didn't expect.

Or maybe it devolves? It takes a lot of discipline in a team (even a team of one) to keep your testing system clean and orderly. Often it becomes a hodgepodge of bolted on test cases that can start disagreeing even with themselves! We've all been victims of someone else's failure to tearDown() completely.

Even in mature projects, the tests don't seem to catch everything, no matter how thorough we think we've been.

So why go to all the extra effort? If the tests require as much maintenance as the application itself, what value do they really bring to the project?

First, tests make sure existing interfaces between parts of the application remain consistent.

Second, tests make you as a developer consider the more subtle interfaces than you might not have been aware of. If you consider that your module's interface might be more than just the method signature and return value, you begin to understand some of these more subtle interfaces "Whoa, why did that fail? I didn't even touch that?!" It could be because that code you just refactored had a side effect that other code has come to depend upon. Oops. Good thing the test caught that. Of course, sometimes the tests don't catch every subtle interface impact. But they work often enough that their value to the project cannot be disclaimed.

Finally, and in my opinion, most importantly, when tests are automated (and hopefully executed by a continuous integration system), they keep you from cutting corners when it becomes crunch time. I've developed software under various methodologies. Regardless of the individual tenets of each methodology, they always seem to slip into a "just get it done" mode as milestones approach. Often, when a trouble ticket arrives, the developer jumps right in, makes a quick fix, commits the code, gets a buddy to review it, merges it into the head (tip, trunk, whatever), then receives the email from the CI server that they broke the build. The automated tests catch the fact that the developer didn't think about everything they should have thought about when they made their change.

I once worked in a team that rewarded the build-breaker with a rubber chicken that had to hang over their cubicle until the build turned green again. Yes, I flew the rubber chicken a lot. My role in that team was mostly in the infrastructure of the application, so when I goofed up, it broke a lot of things. Fortunately, we had a decent test system in place, and the CI server let me know about my shortcomings very quickly. So in crunch time, when everyone's focus is naturally narrowed to a dangerous tunnel vision, our automated testing system forced us to widen our vision and take a breath, and think more broadly about everything our simple little change might affect.

Summary

• Testing is hard work. Plan on writing two applications.
• Testing helps maintain a consistent interaction between software modules.
• Automated testing helps you consider more of the overall effect of your change when under pressure.

And finally,

• Fly a rubber chicken in the team. It's a lot of fun. The look on other teams' managers' faces when they walk down the hall is priceless.