Automated Analysis Tools

Secure coding standards alone are inadequate to ensure secure software development because they may not be applied consistently and correctly. Manual security code audits can be supplemented through the use of automated analysis tools, including static and dynamic analysis tools and tools within a compiler suite. However, source code analysis has many problems and limitations. Static analysis techniques, while effective, are prone to both false positives and false negatives. For example, a recent study found that not one of five C and C++ source analysis tools was able to diagnose 41.5% of 210 test cases, while only 7.2% of test cases were successfully diagnosed by all five tools. A similar study showed that not one of six Java code analysis tools was able to diagnose 39.7% of 177 test cases, while 0% of the test cases were discovered by all six tools. Dynamic analysis tools, while producing lower false positives rates, are prone to false negatives along untested code paths. The NIST Static Analysis Tool Exposition (SATE) also demonstrated that developing comprehensive analysis criteria for static analysis tools is difficult because there are many different perspectives on what constitutes a true or false positive.

To address these problems, the CERT Program is working with analyzer vendors and with the WG14 CSCG SG to precisely define a set of analyzable secure coding guidelines for C99, as well as for the emerging C1X major revision. Having such a set of guidelines and standardizing them through the ISO/IEC process should eliminate many of the problems encountered at NIST SATE and also improve the percentage of defects found by more than one tool. In addition to developing a set of analyzable secure coding guidelines, the CERT Program is coordinating a test suite under a BSD-type license that will be freely available for any use. This test suite can then be used to determine which tools are capable of enforcing which guidelines and to establish false positive and false negative rates. Depending on the application, consumers of these tools may have different preferences for tools that can, for example, trade off a high false positive rate for a low false negative rate or vice versa.

The CERT Program has extended the Compass/ROSE tool (developed at Lawrence Livermore National Laboratory) to diagnose violations of the CERT C and C++ Secure Coding Standards. The CERT Program has also developed a GCC prototype of the as-if infinitely ranged integer model that, when combined with fuzz testing, can be used to discover integer overflow and truncation vulnerabilities.