Unit testing vs Scenario testing

Miško Hevery gives some excellent advice on designing testable software here, and elaborated here:

• Unit Testing as the name implies asks you to test a Class (Unit) in isolation.
• If your code mixes Object Construction with Logic you will never be able to achieve isolation.
• In order to unit-test you need to separate object graph construction from the application logic into two different classes
• The end goal is to have either: classes with logic OR classes with “new” operators.

Here's the explanation / summary:

Asking for your dependencies instead of constructing them withing the application logic is called “Dependency Injection” and is nothing new in the unit-testing world. But the reason why Dependency Injection is so important is that within unit-tests you want to test a small subset of your application.

The requirement is that you can construct that small subset of the application independently of the whole system. If you mix application logic with graph construction (the new operator) unit-testing becomes impossible for anything but the leaf nodes in your application.

Without Dependency Injection the only kind of testing you can do is scenario-testing, where you instantiate the whole application and than pretend to be the user in some automated way.

My emphasis.

Now, this is not to say that you don't need scenario testing, but I have found by bitter experience that trying to mock bits and pieces in scenario tends to be more trouble than it is worth: You end up with large and unwieldy pieces of testing infrastructure.

My current recommendation is that you need to test in both isolation and integration:

1. For isolation (unit testing) use TDD and dependency injection, or DBC
2. For integration, use scenario-based testing on the actual application using some kind of scripting interface.

Other specialist tools may be useful. For example, for highly graphic applications it is invaluable to be able to step-through the scenario tests to visually confirm what is happening on-screen. Visual diagnostics are also a great help for this "semi-manual" testing.