Automated Testing Save Lives
August 20, 2020•2,384 words
Automated testing saves lives.
It's a fact. And not only for software engineers building rocket software for carrying folks to Mars or controlling nuclear reactors. By automating your testing, you will literally save yourself and your teammates precious hours in our short lives where we don't have to chase down regression bugs.
And if you're not testing, you shouldn't call yourself a software engineer.
One huge difference between programming and software engineering is the testing.
A "programmer" is someone that just needs the thing to work. Once it's working, they move on to shiny new problems (at least until something breaks). I am often in a programming mindset when I write software for fun, since nobody depends on what I write and I'll probably get bored of it in the next week or two. That's a programming mentality. And there's nothing wrong with that! There are tons of situations where programming is all that is needed, but when you're getting paid to develop software you need to be a software engineer.
And software engineers write tests.
Why Are Tests So Important?
Let's say you were a construction worker building a high-rise apartment, where you often dangled over ledges to get bolts and beams into place. If your company forgot to provide you with a harness, tether, and safety net (or even actively told you that you didn't need them), you would likely be scared to do your job. You'd play it safe, maybe leaving some of the far-reaching bolts a little loose since a proper tightening means climbing out to the edge with nothing to protect against a fall.
Tests are your harness, tether, and safety net. They give you an automated way to encode business rules, protocols, and other standards into your codebase and allow you to check that you haven't broken anything accidentally.
Having a suite of tests allows you to code with confidence. You can make improvements to the codebase when you see opportunities without the need to do extensive manual regression testing. You can try new programming patterns or technologies and refactor to your heart's content. The freedom to experiment and improve your existing codebase leads to growth in your technological prowess as you discover new ways of working.
In addition, testing saves money. Sure, they take "extra" time to implement and run, but the benefits far outweigh the costs. Computers are really good at doing the same thing over and over. That's a perfect fit for checking that the existing state of your application won't break with your new changes! Plus, once you have a test suite in place, it's incredibly cheap to run them many times a day. Compare that to the cost of manual testing, where a human takes longer and is less accurate in reporting the results or covering all cases.
That's not to say that humans shouldn't be testing your software. You absolutely need a great squad of folks that ensure the software quality remains high, but they are the last line of defense and should be focused on the higher-level concerns of usability, accessibility, perceived performance, and other "softer" metrics that can make a huge difference for the end user.
Testing allows for a more frequent release cadence, which gives your team a chance to quickly test your new code and make changes in response to new information and requirements. If you have a well-designed test suite and everything passes, the likelihood of catastrophic failure to release the new code drops substantially.
Finally, tests lead to better written code with clean interfaces. When your tests are a consumer of your code, you have to put a little extra thought into what those boundaries look like. You're able to decouple your codebase into stand-alone pieces that can be shared with others, which can easily be swapped out for better technology in the future. In addition,
Alright, now that I've ranted about why testing is important, let's get into some specific tools. This article covers the Jest and .NET testing tool sets, but the overall principles can be applied to pretty much any technology.
Unit Test Framework Examples
In general, tests follow this pattern:
- Arrange your test data
- Act on the function being tested
- Assert that the result matches the expected output
Jest
Jest is a widely used JavaScript/TypeScript testing tool and is built into the create-react-app tool.
By convention, test files in JS/TS projects are named like: [file_being_tested].test.ts
Here's a simple test from a router.test.ts
that tests the addSearchParams
in the router.ts
file. The addSearchParams
function builds a URL from the provided parameters and values.
describe('router tests', () => {
test('addSearchParams adds param', () => {
// arrange
const testUrl = 'https://testingurl.com/api/stuff';
const testParams = 'Param1';
const testValues = ['1'];
// act
const result = addSearchParams(testUrl, testParams, testValues);
// assert
expect(result).toBe(`${testUrl}?${testParams}=1`);
});
});
Let's break it down by the Jest-provided pieces:
describe
- An optional function for labeling your test sections, displayed on the CLI when running teststest
- The actual test that will be runexpect
andtoBe
- Jest helper functions that run the assertions for the test (there are tons more, liketoBeTruthy
,toMatchObject
, etc.
The addSearchParams
function is a good one to test because it has a single job and a clean interface. By writing a test like the one above, we can fearlessly refactor and improve the internals should the need arise!
In fact, the need to refactor the method did arise when I added this test:
test('addSearchParams adds param with list', () => {
const testUrl = 'https://testingurl.com/api/stuff';
const testParam = 'Param1';
const testValues = ['1', '2', '3', '4'];
const result = addSearchParams(testUrl, testParam, testValues);
expect(result).toBe(`${testUrl}?${testParam}=1,2,3,4`);
});
When I ran that test, it failed! For some reason, the result looked like ${testUrl}?{testParam}=1&2&3&4
!
After taking a peek at the addSearchParams
function, I noticed that the delimiter between parameter values was using an &
instead of a ,
. The &
would signify an entirely new URL parameter, which is not how the function should behave.
This particular function had been around for a long time, but nobody had caught this bug. After a bit of searching the codebase, I did see that a few developers had been passing in an optional third parameter called delimiter
using the ,
, so they had seen the problem and solved it by fixing the inputs instead of addressing the root cause.
Before I went crazy with new code to fix up the function, I decided to add a few more tests that covered all the edge cases involved with building a URL that I could think of. Once those tests were in place, I confidently started refactoring the function until they all passed!
Once everything was passing, I could say with confidence that my changes had materially improved the codebase and the developer experience. I didn't need to boot up the application, spend five minutes getting to the specific web page that used the function, and then testing things manually. All of that ceremony was removed, and with the test suite in place I could quickly iterate on a solution instead!
I ended up adding 50+ tests over the next week and found even more bugs around edge cases. But now with the tests in place, it's easy to check that they will keep behaving far into the future!
Jest is a great tool. It just works and has fantastic documentation. If you're working with the create-react-app
or really any other JavaScript/TypeScript project, I'd recommend you consider it for your testing framework. Give the official documentation an afternoon, and you'll be up and running writing all kinds of tests!
Microsoft Testing Tools
If you work with .NET, there are a ton of testing frameworks to choose from. The default tools from Microsoft are solid, so we'll take a quick peek at what that looks like:
[TestClass]
public class BlobStorageServiceTests
{
[TestCategory("PDF Tests")]
[TestMethod]
public void GetSignedApplicationPdf_Test()
{
IBlobStorageService svc = new BlobStorageService();
byte[] bytes = null;
string _samplePackageId = "3b2c8766-2706-40c7-b46c-911047996c2c";
bytes = svc.GetSignedApplicationPdf("MyApplication", _samplePackageId);
Assert.IsTrue(bytes.Length > 10000);
string _sampleEnrollmentId = "01086456694P07302019";
bytes = svc.GetSignedApplicationPdf("Medicare", _sampleEnrollmentId);
Assert.IsTrue(bytes.Length > 10000);
}
}
The unit test pieces are:
[TestClass]
- Indicates that the class is used for testing, and the testing tools will automatically find and include it[TestCategory]
- A human-readable tag to group multiple tests together[TestMethod]
- Marks a single testAssert.IsTrue
- One of the manyAssert
functions to check your test results. Others includeAssert.Fail
,Assert.Equals
,Assert.IsNotNull
, etc.
This particular function being tested returns a PDF from cloud storage. We're not touching on the technical details of mocking inputs in this article, so just assume that the BlobStorageService
doesn't need to actually talk to the internet to fetch the PDF.
Notice how similar the process is compared to the Jest tests? It's the same process, just with a little more pomp and circumstance that a typed language like C# demands.
I don't have a story about successfully adding more tests and fixing bugs in this particular codebase, but that's because the 1000+ unit tests are currently broken. Those are thousands of tests represent dozens (if not hundreds) of hours put into building that impressive safety net. They could be helping check for errors when making changes to the code base today, but instead those tests have been left to decay over the years and are at a point that a major refactoring would be needed to fix them all and reintroduce them back into the build pipeline.
That's why you should never remove your tests from the automated pipeline. If you refuse to let broken tests get merged into your codebase in the first place, you'll never lose the hours of work that were put in to build the test suite in the first place!
How to Approach Your Work with a Testing Mentality
A test-first mentality leads to cleaner, reusable interfaces with your code. While you don't need to go full-on Test Driven Development to be effective, simply considering how your code will be tested leads in better outcomes.
When I need to write new code, I tend to do the following:
- Break the problem down into small, single-purpose functions (ideally ones that don't mutate state)
- Get those functions working, with unit tests proving it
- Compose those smaller functions together to solve the problem
- Write more tests demonstrating that the larger function works as expected
- Compose the larger function together with other pieces of the application
- Repeat until everything works
I've found it easiest to work from the backend out to the UI layer, following the pattern of smaller, easily verifiable functions that together solve a larger problem. Breaking things down into manageable chunks allows you to lessen your cognitive load as you move further up the stack (meaning, towards the user-facing interface). As you build each piece and test it, you can be more confident that the higher level solutions will work properly. Now, that's not to say you shouldn't be doing integration or end-to-end tests, but if you unit test and encapsulate your logic smartly, the odds of any one thing breaking too badly decreases.
I'm sure you'll find your own cadence regarding how much testing you need to work through problems and how you write your code. Just like writing in spoken languages, each person has their own style and approach when they talk to their machines through code! Experiment and find what works best for you!
How to Create a Culture of Testing on Your Team
Now the hard part. When I was on my very first project out of college in 2014, we spent about a month fixing hundreds of broken unit tests. They had previously been taken out of the build process (or potentially had never been part of it at all), so the tests got ignored and atrophied over time. It hadn't been a requirement to have a working test suite before moving on to the next feature, but our team was determined to change that. Five years later, I've found myself back on the same project again, only to discover there are over a thousand tests that no longer function.
As a consultant (i.e. an outsider) I can't just add tests and hope that they keep running in perpetuity. It takes buy-in from the developers and leaders who actually own the codebase, and without out it, those tests will be left to rot. You can't just say "unit testing is important" and expect a culture of testing to pop up overnight.
The best way I've found to get teams to care about testing:
- Start small. Don't try to fix a 1000+ unit test repository in a Sprint
- Integrate testing to any new initiatives from the very beginning
- Add those working tests to the build process
- Don't allow the test step in the build process to be bypassed for any reason
- Encourage developers to add tests when reviewing their Pull Requests
- Start a grassroots approach on your team and continually talk about the importance of tests
- Ask your leadership to push the importance of testing from the top down
- Track metrics on the types of bugs that come up (regressions vs new) to show that regressions go down as testing increases
- Push for more sophisticated testing once you have a good unit test suite in place
What's Next?
There are many different dimensions to testing. This article has focused on the technical details of unit tests, but the general principles apply whether you're doing end-to-end, integration, UI, performance, accessibility, or any other kind of testing.
If you use JavaScript and want a fantastic overview of the different types of testing and when they're appropriate, check out https://testingjavascript.com/ (Disclosure: Kent is a friend of mine.)
As always, please reach out to me if I've completely bungled something or you have even better ideas to add! I'm always looking to learn and grow, so drop me a comment in my guestbook or find me online!
This is the sixth of nine articles delving into the processes that every effective development team should use. Stay tuned for more!