Testing

Testing
All software systems need testing to determine if they perform correctly according to their specifications. There are a wide range of testing systems and several are covered here. Automating tests and their reports is a critical part of software engineering. In some systems, there is much more test code than the system under test. Perl comes with several useful modules and CPAN has more all under the Test:: namespace.
Unit Tests
This form of Testing focuses on the smallest Granularity of code, typically individual subroutines and simple modules and classes. It is also the best place to do coverage Testing as in most cases you can create tests that exercise 100% of the target code. Perl comes with several useful modules to make creating test scaffolds and reports. CPAN also has more test related modules.
Integration Tests
This form of Testing focuses on mid-level granularity of code, typically subsystems and complex modules and classes. Regression Testing is commonly done at this level. Coverage Testing can be done but it is usually harder to get 100% coverage due to the increasing complexity of the code under test.
System Tests
This form of Testing focuses on the highest level of Granularity of code, i.e. complete running systems. These tests are usually conducted in an environment (commonly called staging) as close as possible to the real world. Because of the size of the code under test, Regression Tests are most common here and code coverage testing is rarely done. This is also a common granularity level to run load testing and profiling.
Regression Tests
One common concern when developing large systems is making sure that previously fixed bugs don't reappear (which is called regressing). Checking for this is called regression testing. These tests are usually run at the integration or system test level but even Unit Tests can be called Regression Tests. Typically they are run with a set of input data and a set of expected output data which are compared to the actual output. The comparison can be very complex for large systems and may have filters to ignore known or expected discrepencies such as timestamps. As bugs are discovered and fixed, Regression Tests should be added to make sure they don't resurface in later versions of the code.
TLoad Testing
This form of Testing is designed to determine how a system behaves under heavy use. It is usually part of itegration or system testing. Input data can be randomly generated, played back from logs, or manually created. Sometimes very subtle and difficult to find bugs can only be found by long load test runs.
Coverage Tests
This form of Testing is usually done at the unit test level but sometimes it is done at the integration level. It involves writing tests that exercise each line of code under test. Creating Coverage Tests is trickier than basic unit Testing as you need to create test cases for all the possible code paths and sometimes for complex cases, it is impossible. You also need special modules to instrument the code under test that reports which code lines have and have not been executed. CPAN has the Devel::Cover module which does this for Perl.
Profiling
If a subsystem has been determined to be too slow or inefficient for the Requirements, then profiling can help determine the bottleneck. Similarly to Coverage Tests, a special module is used to instrument the code being profiled and it will report how much time is spent in what sections of the code. Perl comes with the Devel::Dprog module and there are domain specific profile modules on CPAN.
Benchmarks
Sometimes there are multiple ways to solve a software problem, be they different code, algorithms or data structures. Benchmarking is a special form of Testing which runs different code with the same input data and compares their speed and/or resource usage (such as memory). As with other tests, it is a skill unto itself to create benchmarks that accurately isolate the problem and properly compare apples to apples.


Copyright © 2005 Sysarch, Inc -- Last Modified Sat Jun 25 03:23:36 2005