Software Testing
New versions of publicly distributed scientific software codes released by SCEC software engineering group undergo rigorous software testing during development and prior to release.
Overview
CME Project software developers integrate complex scientific codes together into larger computational systems that we call Computational Platforms. Each SCEC computational platform is designed to reliably perform some useful or valuable research calculation. In some cases, SCEC provides source code distributions of computational platforms for use by the SCEC community. To ensure these platforms meet the design specifications and user requirements, rigorous software testing is employed. The two key forms of testing, Unit and Acceptance testing are employed to ensure the quality of the released software.
Types of Testing
Text on types of testing from C. Titus Brown’s presentation, Choosing Infrastructure and Testing Tools for Scientific Software Projects.
- Automated testing
- The basic idea behind automated testing is to write test code that runs your main code and verifies that the behaviour is expected
- Example – regression test
+ Run program with a given set of parameters and record the output
+ At some later time, run the same program with the same parameters and record the output
+ Did the output change in the second run, and if so, do you know why?
+ This is different thing from "is my program correct"
+ If results change unintentionally, you should ask why
- Example – functional test
+ Read in known data
+ Check that the known data matches your expectations
+ Does you data loading routine work?
+ It works best if you also test with "tricky" data
- Example – assertions
+ Put "assert parameter >=0" in your code
+ Run it
+ Do I ever pass garbage into this function?
+ You’ll be surprised that things that "should never happen", do happen
+ Follow the classic Cold War motto: “Trust, but verify”
- Other kinds of automated testing (acceptance testing, GUI testing), but they don’t usually apply to scientists
- In most cases, you don’t need to use specialized testing tools
- One exception is a code coverage tool
+ Answers the question “What lines of code are executed?”
+ Helps you discover dead code branches
+ Guide test writing to untested portions of code
- Continuous integration
+ Have several "build clients" building your software, running tests and reporting back
+ Does my code build and run on Windows?
+ Does my code run under Python 2.4? Debian 3.0? MySQL 4?
+ Answers the question: “Is there a chance in hell that anyone else can use my code?”
- Automated testing locks down "boring" code (that is, code you understand)
+ Lets you focus on "interesting" code – tricky code or code you don’t understand
+ Freedom to refactor, tinker, modify, for you and others
Unit Testing
Unit testing, also referred to as White-box testing is employed to test the internal structure of individual code modules or sub-systems in the platform. This ensures these modules were 'built right'. Unit testing allows us to test parts of the program work correctly and reduces the time required to integrate these modules.
Unit testing also reduces uncertainty in units as these modules are re-factored into different platforms.
Acceptance Testing
Acceptance testing is also referred to as Black-box testing or validation testing. It consists of a suite of tests that are run to validate the end-to-end functionality of the integrated system. The test environment is designed to emulate the anticipated user environment. Unlike unit testing which tests the internal code structure of individual modules, Acceptance testing is more high level and is designed to test the integration of the underlying modules. Acceptance testing validates the overall functionality of the built system and ensures 'we building the right thing'.
The tests can be run with supplied input data for various use cases or using scripts that generate work-flows of exhaustive test cases to include every possible combination of sub-modules. The results of these tests give confidence that the integrated system is functioning as expected and is fit to be released.
