Increase Software Quality with Static Analysis and Dynamic Tests

Author: Klaus Lambertz, Founder and Managing Director of the German software quality specialist Verifysoft Technology GmbH

Good software fulfills the expected functionality, is secure, reliable, and easily maintainable. It is important that software has as few errors as possible, so that both functional safety and security are ensured. For this purpose, two different and complementary methods are ideally used during software development: static code analysis and runtime testing, also known as dynamic analysis. Due to their nature, both methods only uncover part of the existing problems. Only when they are used together the quality of the software can be decisively increased.

Which software project manager or developer does not know this: delayed deliveries, exceeding budgets, high time and cost expenditure for error corrections, as well as difficulties in correctly implementing the promised features, … Software quality is not a matter of course, but can only be achieved through consistent action, adherence to standards and the use of sophisticated testing and analysis tools.

Good software quality contributes to a company’s reputation and helps decisively to save costs in the medium and long term. In the development of safety-critical software, procedures and tests are prescribed by standards to rule out any risk to health and human life from defective software.

To ensure quality, there are two complementary approaches that can only ensure highest code quality when they interact: Static code analysis and the testing of executable software in connection with the proof of sufficient code coverage.

Static code analysis finds errors very early in the development cycle

Particularly in the development of embedded systems, languages such as C and C++ are often used because of their hardware-oriented programming. These languages enable high-performance and compact code. C and C++ give developers a lot of freedom in programming – and unfortunately also in writing erroneous source code.

For example, a C/C++ compiler does not insert code that tests whether requested memory has also been allocated or whether a string to be copied fits into the target array. Errors such as null pointer exceptions or buffer overflows can be the result. Uninitialized variables can also cause indeterminate and dangerous program behavior. Variable types that are relatively easy to change in C and C++ can lead to implicit value changes. Checking the correct procedure becomes even more difficult when many developers are working on different but interdependent modules at the same time, and no one has the “big picture” of the program developments in mind.

Static code analysis helps decisively here. In an analysis run, images of the complex processes and data accesses are created, which are then extensively evaluated. Analysis tools such as CodeSonar from GrammaTech (1) create models of the data flows and accesses based on the source code, analyze them, and list all potential weak points. It quickly becomes clear where functions and global variables are used. Graphical outputs show the relationships in the software. Problems are reported with a detailed error description and can thus be quickly corrected.

Programming guidelines ensure clean programming

Because of the vulnerabilities of hardware-related languages such as C and C++, early efforts were made to develop guidelines that limit the flexibility of these languages in favor of greater security and provide clear specifications as to which language elements may and may not be used. These include coding guides such as MISRA C and MISRA C++, but also Build Security In (BSI) of the US Department of Homeland Security, the Power-of-Ten rules of NASA and the JPL rules of the US Jet Propulsion Laboratories, which are built on the Power-of-Ten and MISRA-C rules to avoid risks of using multi-threaded software.

Good static code analysis tools help to verify compliance with these rules. The tools can be integrated into all common development environments and can be used in the context of Continuous Integration.

Tools for static code analysis thus contribute on the one hand to the development of rule-compliant software and on the other hand also uncover programming errors that have “crept into” the code despite the observance of coding standards.

Dynamic tests check the functional correctness of the software – code coverage tools ensure the completeness of the tests

In addition to static code analysis, dynamic tests – i.e. tests at runtime – must be performed to prove the functional correctness of the system. As soon as the first code components are ready to run, dynamic testing should begin. In principle, the following applies here: the more serious the consequences of errors can be, the more comprehensively testing must be carried out. To determine which parts of the source code have been tested, code coverage analyzers such as Testwell CTC++ (2) from Verifysoft Technology (3) are used. These tools for measuring test coverage place counters at all relevant points in the source code, which log whether or how often the corresponding code part has been tested. When testing embedded software, it is important that these counters take up as little memory as possible and have a minimal impact on performance, to avoid malfunctions in time-critical software. After the test runs, the Code Coverage Tool generates an overview that shows in detail which functions were executed and which were not.

In the case of very critical software, it is necessary to test every single condition of multiple conditions. Standards such as ISO 26262 in the automotive sector and DO178-C in aviation prescribe Modified Condition/Decision Coverage (MC/DC) for this purpose. Coverage tools such as Testwell CTC++ support this level of coverage and provide specific indications as to why certain functions have not been called. This makes it clear to the tester which combinations of the respective conditions still need to be tested.

Software quality needs static analysis and dynamic testing

Both approaches, static source code analysis with tools such as GrammaTech CodeSonar and dynamic testing including the measurement of test coverage with a code coverage tool such as Testwell CTC++ complement each other ideally. Only both methods together can detect almost all errors. In addition, the tools also provide important insights into how the various parts of the code interact with each other, thus helping to reduce complexity with a view to improving maintainability. Integrated into common software development environments, they are an integral part of the software development life cycle and support programmers in every phase of software development.