We applied automated testing during the project to allow us to deliver tested software at the end of each Sprint, without regression bugs. Even as the system grew we managed to do this with only one tester per eight-person Scrum team while maintaining high quality: the external testing team found less than 1 defect per KLOC.
We automated tests on two levels: unit tests and acceptance tests. For the former, we used JUnit and measured code coverage using Clover, using a target for server-side code of 80% coverage. Acceptance tests were automated using FitNesse. For every implemented user story, a set of acceptance tests was written in FitNesse. Having an extensive test suite, regression bugs were usually found and fixed during the Sprint. Another advantage of this approach is that the testers can be effective from the start of the Sprint, creating test cases before the user story is implemented.
There was one area where we struggled with automated testing. Part of the system was an application with a complicated rich user interface. Automating tests for this proved to be harder than automating server-side tests. Therefore we largely depend on manual testing for user interface-specific functionality. As the system grew, regression testing took more and more time. Even worse, external testers found regression bugs only in this part of the system. Having automated tests could have prevented this. The lesson learned is that, even though automating tests may sometimes be hard, it will pay off when it counts, late in the project.
