CERT-SEI
CERT/CC Blog

CERT Linux Triage Tools 1.0 Released

By Jonathan Foote on 04/25/2012 | Permalink

As part of the vulnerability discovery work at CERT, we have developed a GNU Debugger (GDB) extension called "exploitable" that classifies Linux application bugs by severity. Version 1.0 of the extension is available for public download here. This blog post contains an overview of the extension and how it works.

Background
 
CERT recently released version 1.0 of the Failure Observation Engine (FOE), a fuzz testing framework for Microsoft Windows platforms, for public download. You can learn more about FOE in David Warren's recent blog post. As FOE accumulates crashing test cases for the application-under-test, it uses MSEC's !exploitable debugger extension to classify the associated application errors by severity. This allows the auditor or developer who is using the tool to determine which bugs to investigate first.
 
The CERT Basic Fuzzing Framework (BFF), which was initially released for public download in May 2010, is a fuzzing framework designed for use in Linux and Apple OS X. Support for Apple OS X was added in version 2.5 and will be described in an upcoming blog post. When running in OS X, the BFF uses Apple's CrashWrangler tool to assign crash severity in a manner similar to how FOE uses !exploitable on Windows. We were unaware of a similar, lightweight application error classification tool on the Linux platform, so we developed the CERT exploitable GDB extension for this purpose. While the GDB extension is designed to be integrated with future versions of BFF, it can be used as a standalone tool as well.
 
Inspired by !exploitable and CrashWrangler
 
MSEC's !exploitable and Apple's CrashWrangler are simple, but powerful, application error classification tools that have enjoyed widespread adoption on their respective platforms. Our team has found their lightweight execution and simple heuristics amenable to our black box fuzzing approach. In order to foster adoption, promote ease of use, and to allow for orthogonal design in our fuzzing frameworks, the CERT exploitable extension is modeled closely after the MSEC and Apple products. Just take a look at some respective output from the tools:
 
The CERT exploitable GDB extension:
 
(gdb) exploitable
Description: Access violation on destination operand
Short description: DestAv (7/21)
Hash: 056f8e491910886253b42506ac8d7fa0.056f8e491910886253b42506ac8d7fa0
Exploitability Classification: EXPLOITABLE
Explanation: The target crashed on an access violation at an address matching the destination operand of the instruction. This likely indicates a write access violation, which means the attacker may control the write address and/or value.
Other tags: AccessViolation (20/21)
MSEC's !exploitable WinDbg extension:
 
Description: User Mode Write AV near NULL
Short Description: WriteAV
Exploitability Classification: PROBABLY_EXPLOITABLE
Recommended Bug Title: Probably Exploitable - User Mode Write AV near NULL starting at module!function+0x00000000000010a5 (Hash=0x6a652d72.0x71652d0c)
 
User mode write access violations that are near NULL are probably exploitable.
Apple's CrashWrangler tool:
 
exception=EXC_BAD_ACCESS:signal=11:is_exploitable= no:instruction_disassembly=movzbl (%eax,%esi),%ebx:instruction_address=0x00000000969def61:access_type=
read:access_address=0x0000000079757675:
Crash accessing invalid address. Consider running it again with libgmalloc(3) to see if the log changes.
Test case was (null)
While the output from these tools may appear to be similar, the CERT exploitable extension includes some key differences. For example, where MSEC's !exploitable command performs taint analysis of the basic block containing the instruction that caused the application error, the CERT exploitable extension does not look beyond the faulting instruction. The CERT extension is similar to Apple's CrashWrangler in this respect. Additionally, the CERT exploitable extension is written entirely in Python, whereas MSEC's !exploitable is written in C++, and CrashWrangler is written in a mix of Ruby and C. We chose the Python language for the CERT tool to promote modification and enhancement of the code by third parties. It is also important to note that the CERT exploitable extension is designed to work specifically with user space applications on the x86 and x86_64 platforms. For more information on how the code works and some tips on modifying it, see the readme files.
 
Using the CERT exploitable GDB extension and triage script
 
The CERT exploitable GDB extension is designed to be relatively simple to invoke. If you have experience using MSEC's !exploitable extension, then using the CERT exploitable GDB extension may seem familiar. First, download the CERT triage tools package and extract it to a directory on a Linux host. Be sure to check the readme files for system requirements.
 
Once you have the tools extracted, you should be able to run GDB and load the exploitable extension:
 
$ gdb 
(gdb) source exploitable/exploitable.py
(gdb) help exploitable
 
A GDB Command that determines how exploitable the current state of the 
Inferior (the program being debugged) is. Either prints the result to 
GDB's STDOUT or pickles the result to a file.
 
This command is designed to be run just after the Inferior stops on 
a signal, before any commands that might change the underlying state 
of GDB have been issued. WARNING: This command may change the underlying 
state of GDB (ex: changing the disassembler flavor).
 
Type <cmd> -h for options. Note specifying incorrect command options may
cause GDB to exit.
 
Now you are ready to run an application test and use the exploitable extension to classify the output:
 
(gdb) file exploitable/tests/bin/crashwrite.test 
Reading symbols from ./src/exploitable/tests/bin/crashwrite.test...(no debugging symbols found)...done.
(gdb) run
Starting program: ./src/exploitable/tests/bin/crashwrite.test exploitable/tests/bin/crashwrite.test
 
Program received signal SIGSEGV, Segmentation fault.
0x080483a4 in main ()
(gdb) exploitable
Description: Access violation on destination operand
Short description: DestAv (7/21)
Hash: 056f8e491910886253b42506ac8d7fa0.056f8e491910886253b42506ac8d7fa0
Exploitability Classification: EXPLOITABLE
Explanation: The target crashed on an access violation at an address matching the destination operand of the instruction. This likely indicates a write access violation, which means the attacker may control the write address and/or value.
Other tags: AccessViolation (20/21)
(gdb)
For more details on installation and usage of the exploitable extension, check out the readme file. In addition to the GDB extension, the CERT triage tools package also includes an example wrapper script called "triage." While the GDB extension classifies a single application error, the triage script runs an application with a set of test cases and classifies each with the GDB extension. This script was developed as proof-of-concept for integration with later versions of the CERT BFF and as an example for integration into other testing frameworks by other developers; however, it can used as-is to classify a set of test cases.
 
To use the script, you will need to download the CERT triage tools package and extract it to a directory on a Linux host if you haven't already. Again, be sure to check the readme files for system requirements. Once the tools are extracted, run a properly formatted triage command. Here is an example of running the tool against the set of Apple CrashWrangler test cases that are included in the package.
 
$ python triage.py \$sub `find ./exploitable/tests/bin -type f`
The input syntax for the triage script is somewhat tricky, and the tool isn't particularly robust. This is a side effect of the tool's primary purpose as a proof-of-concept and example wrapper script. Regardless, once you have executed the script correctly you should see this:
 
... (libc output) ...
 
EXPLOITABLE: StackBufferOverflow
./exploitable/tests/bin/stack_buffer_overflow.test
 
EXPLOITABLE: PossibleStackCorruption
./exploitable/tests/bin/variable_length_stack_buffer.test
 
EXPLOITABLE: DestAv
./exploitable/tests/bin/cpp_crash.test
./exploitable/tests/bin/crashwrite.test
./exploitable/tests/bin/fastMalloc.test
./exploitable/tests/bin/invalid_address_64.test
./exploitable/tests/bin/recursive_write.test
 
EXPLOITABLE: BadInstruction
./exploitable/tests/bin/illegalinstruction.test
 
EXPLOITABLE: HeapError
./exploitable/tests/bin/malloc_abort.test
 
PROBABLY_EXPLOITABLE: BranchAvNearNull
./exploitable/tests/bin/bad_func_call.test
./exploitable/tests/bin/crashexec.test
 
PROBABLY_EXPLOITABLE: BlockMoveAv
./exploitable/tests/bin/read_and_write_instruction.test
 
PROBABLY_EXPLOITABLE: DestAvNearNull
./exploitable/tests/bin/nullderef.test
 
PROBABLY_NOT_EXPLOITABLE: SourceAvNearNull
./exploitable/tests/bin/uninit_heap.test
 
PROBABLY_NOT_EXPLOITABLE: FloatingPointException
./exploitable/tests/bin/divzero.test
 
UNKNOWN: SourceAv
./exploitable/tests/bin/crashread.test
 
UNKNOWN: AbortSignal
./exploitable/tests/bin/abort.test
 
Failed to triage:
./exploitable/tests/bin/nocrash.test
Customization
 
While we kept third-party adoption and development in mind, we developed these tools primarily for some specific testing purposes in the CERT Program. The tools have not been exhaustively tested, and have not been tested at all on many Linux distributions! Further, the nuances of various Linux distributions and GDB versions may change how the exploitable extension classifies application errors. If you choose to use the GDB extension, I encourage you to become familiar with the source code and edit it to suit your needs. It is designed to be simple and easy to modify--check out the readme files to get started.
 
That's all for now--if you have any comments, questions, patches, or other feedback please drop me a line.

Topics: Research , Vulnerability Analysis , Vulnerability Discovery