What kind of test am I writing?

In pursuit of a robust test suite

One common problem that I see in test suites is confusion about what each test should cover. This confusion often leads to tests that are either too broad or too focused to accomplish the goal of their creator. When writing a test, it is important to think about what kind of test it will be and the constraints that make that type of test effective. I have 4 broad categories of tests that I keep in mind to help focus my testing.

Unit Tests

Also called micro tests, unit tests form the foundation of any good testing strategy. These are tests that test a single class or method in isolation from its dependencies. Test doubles help us isolate the code under test and make tests that are more focused and expressive. A test that touches any external resource such as the database, the file system, a web service, etc is never a unit test.

The primary purpose of unit tests is to validate that the subject under test is functionally correct for the expected inputs and outputs. Additionally, unit tests provide documentation on how the class is expected to function and be used by consumers.

Good unit tests are fast, atomic, isolated, conclusive, order independent, and executed automatically by the continuous integration server on each commit to the source control system.

Smells for unit tests include too much setup, long test methods, race conditions, reliance on strict mocks, lack of CI integration.

Integration Tests

There are two primary sub-categories of test that are called integration tests.

External Dependency Tests

The first category includes tests that interact with a specific external dependency. This includes testing the code that written within the system that interacts with the external component. Examples include tests of your data access layer, web service proxies, file system proxies, etc. In order to provide real value, these tests must exercise the real dependency. For that reason, they often require more set up and are slower to write and run than unit tests, but in order to have confidence in a test suite, these tests are absolutely necessary.

The primary purpose of these integration tests is to validate the code that that manipulates the external system. Additionally, these tests may validate some portion of the remote system. As an example, consider a data access object that is implemented in terms of an object relational mapper with stored procedures in the database. If you call Save, then Load and validate that the retrieved object is equivalent to the stored object, you have tested your DAO, your OR mapping, the stored procedure(s), the network connection, the database connection string, the database engine, the script evaluation sub-system in the database, etc. You can see that these tests exercise a larger block of code and infrastructure than unit tests. For this reason, they tend to be more brittle and prone to breaking.

Good integration tests validate the features of the external system that you use in your application. They do not attempt to cover the full set of functionality, but only to validate that what you need works. Like your unit tests, these tests should be atomic, isolated, conclusive, order independent, and executed automatically by the continuous integration server as often as is reasonable. You also want them to be as fast as possible, but keep in mind that they will never be as fast as your unit tests.

Smells for these tests include too much setup, long test methods, race conditions, reliance on any test doubles, lack of continuous integration.

Unfocused Partial System Tests

The second category of integration tests are those that depend on more than one of your own components. These are tests that are generally confused about their role. JB Rainsberger gave an excellent talk about these tests that you can watch here on InfoQ: Integration Tests are a Scam.

Acceptance Tests

Acceptance tests are often overlooked, but critical to a solid test suite. These are tests that execute your entire stack; with the possible exception of the user interface. To be clear, this means using no test doubles of any kind. You may choose to bypass the UI and attach at the Application Layer or API just under the UI. These tests complement the unit and integration tests. While unit and integration tests validate correctness in the small, these tests validate correct composition of your building blocks. Acceptance tests are often, but not always, written using different tools than those used in unit and integration testing.

The primary purpose of these tests is validate things like component wire up, application stack integration, basic use cases/user stories, system performance, and overall application stability. These tests will likely be run the least often as they will be fairly time consuming to execute and may require extensive (though automated) setup.

Good acceptance tests can be understood by a user and are written in terms common to the business.

Smells for acceptance tests include attempting to validate every path through the system.

User Interface Tests

These are tests that manipulate your application the way the user would. In theory, you can test just the UI this way, but in practice, this most often means that these tests exercise the whole application stack. Writing UI tests often requires special tools to manipulate the application. These are the most fragile, brittle, expensive tests to write and maintain. They are also the slowest to run. It is often best to use these tests only to validate that the application is navigable, doesn’t fall over and has all of its dependencies met.

Good UI tests are simple and limited in scope.

Smells for UI tests include inconsistent failures, testing application correctness.

Constantly Improving

There are many ways to categorize tests. I find these 4 categories help me to focus my testing and build a faster and more reliable test suite. What techniques do you use to improve your testing suite?