This chapter begins with a discussion of the evolution of CERT^®/CC incident response. This is followed by a discussion of the characteristics of the CERT^®/CC records, and the methods used to construct the individual incident records. The categories of data extracted from these constructed incident records is then presented.

4.1. CERT^®/CC Incident Response

The organization and operation of the CERT^®/CC appears to have gone roughly through three periods: 1) an early, informal period from November, 1988 to around January, 1992, 2) a transitional period for the next year and a half, and 3) a more formal period beginning in the summer of 1993. CERT^®/CC records reflect these changes in organization and operation.

4.1.1. Early, Informal Period -- November, 1988 to January, 1992 - After the Internet Worm incident in November, 1988, DARPA quickly moved to establish the CERT^®/CC in order to institutionalize the incident response capability that was spontaneously formed during the incident. Within weeks, the CERT^®/CC was functioning at the Software Engineering Institute (SEI) of Carnegie Mellon University, Pittsburgh. Beginning in these early weeks, and continuing for this early period, the CERT^®/CC responded to incidents in an ad hoc, informal manner. Communications were primarily through electronic-mail (e-mail), supplemented by the telephone.

Records during this early period reflect resistance from CERT^®/CC personnel to efforts to formalize incident responses, although there were continuous efforts to formalize the process of formulating responses. The rationale was to maintain the greatest flexibility for CERT^®/CC personnel, who could then use their own judgment in determining the correct course of action during any incident. This system remained in place throughout the period studied in this research.

CERT^®/CC personnel have never formalized the rules of incident response beyond that necessary for a very basic training in incident response. Instead, CERT^®/CC personnel relied on an extensive and lengthy apprenticeship training program, as well as prior experience, for new personnel to learn incident response.

Consensus was achieved in the early period in some areas of the incident response process. The ground rules for confidentiality discussed in Chapter 3 and 14 were established fairly quickly. Patterns were also developed for personnel scheduling, as the number of people responding to incidents increased in the first year from the initial two to around a half dozen. Incidents were responded to by personnel who were assigned to the hot-line position for one to two weeks at a time. When "on point" at the hot-line, CERT^®/CC personnel handled all aspects of all the open incidents. At the end of their period on point at the hot-line, they would brief the incoming personnel on the open incidents and then hand over the incident response function. As a result, during this period, all incidents were handled by different people every one to two weeks.

Incident response was the initial motivation to establish the CERT^®/CC. This is a reactive role with CERT^®/CC personnel waiting for an incident to be reported before taking action. From the beginning, however, the CERT^®/CC charter also included the more proactive role of providing security information to the Internet community. As a result, CERT^®/CC quickly became a repository for information on vulnerabilities in Internet systems.

Information on possible vulnerabilities came into the CERT^®/CC from both the Internet user community, and from hardware and software suppliers. CERT^®/CC personnel would then test the reported vulnerabilities to see if they were real. CERT^®/CC personnel maintained records of these vulnerabilities. These records evolved into a vulnerability database that was maintained throughout the period studied in this research. CERT^®/CC personnel included both the vulnerabilities, and the "fixes" or "work-arounds" that were developed either by CERT^®/CC personnel, by the software and hardware suppliers, or by others in the Internet community.

This established the position of the CERT^®/CC as a single point of contact for system and network security as described in Chapter 3. CERT^®/CC personnel on point at the hot-line were, therefore, responsible for three types of contacts from Internet constituents: 1) requests for assistance during an incident (incident response), 2) information from Internet users and vendors on vulnerabilities, and 3) requests from Internet users for information on how to reduce vulnerabilities and to increase security.

4.1.2. Transition Period -- January, 1992 to September, 1993 - By the beginning of 1992, the number of incidents grew to where the ad hoc process of incident response was not satisfactory. CERT^®/CC personnel were overwhelmed in two ways. First, the method of keeping track of incidents was informal, involving handwritten notes and electronic mail (e-mail). Incidents were not tracked by numbers, nor by specific sites. As the number of incidents increased, CERT^®/CC personnel had increasing difficulty keeping track of information and responding effectively. Second, passing the responsibility for all incidents to the incoming team was increasingly difficult, time-consuming and confusing.

The first adjustment for CERT^®/CC personnel was to begin tracking incidents by site. As discussed below, this primarily involved manually summarizing e-mail into one file under a site name. Also, CERT^®/CC personnel began numbering e-mail messages to aid in referring to them. This procedure was continued through 1992. However, the adjustment proved to be inadequate, and in the beginning of 1993, the CERT^®/CC began tracking incidents by assigning a single, random number, such as CERT^®#1234, to each incident, in addition to continuing the assigning of other numbers to individual messages. Inquiries for information were assigned information numbers (example: INFO#45612), and information involving vulnerabilities was assigned a vulnerability number (example: VUL#789). During the first half of 1993, an automatic e-mail sorting and summarizing system was developed based on these numbers. This was an improved system, although the summaries were terse and generally required CERT^®/CC personnel to refer frequently back to the original messages.

The second adjustment made by the CERT^®/CC during this period was to transition the response team away from handing off incidents to different people every week or two. Instead, each incident was assigned to one person in the CERT^®/CC to handle comprehensively from the beginning until the incident was closed. This helped to ensure continuity in incident response. The assignments were made according to the workload of CERT^®/CC personnel.

4.1.3. Formal Period -- September, 1993 to December, 1995 - By the end of the transition period, incident response was formalized. CERT^®/CC personnel responding to the hot-line and e-mail inquiries were now known as technical coordinators. One change that was made during 1994 was to improve the program used to sort and summarize e-mail. This included having the program copy more of the body of each message into the summary file for that incident number. This significantly reduced the need to refer to the original messages.

4.2. CERT^®/CC Record Characteristics and Methods of Analysis

The CERT^®/CC records reflect the purpose of the CERT^®/CC to respond to Internet incidents, investigate vulnerabilities, and disseminate information. As discussed, this required the development of a vulnerability database that could be accessed by CERT^®/CC personnel during an incident, and when information was requested. CERT^®/CC also disseminated information through the CERT^® Advisories mailing list (e-mail listserver), through an anonymous FTP (file transfer protocol) site, and later through a World Wide Web site.

For incident response, all CERT^®/CC records were maintained "on-line" in the CERT^®/CC local area network. These records could be searched for key words in order to find similar events. In addition, as discussed above, beginning in 1992, each message that arrived at the CERT^®/CC was assigned a unique number in the incident summary file. This number could be used to view the original message.

While the CERT^®/CC records were useful for the "real-time" CERT^®/CC operation, the records did not represent a source of information valuable for analysis. For example, the actual number of incidents reported to the CERT^®/CC could not easily be determined. This was because, even in the period after the transition, multiple records could be opened for the same incident, if it was reported by more than one site. The records themselves usually indicated the relationship between these records, but this required reading the individual incident summaries.

4.2.1. Early Period Records -- November, 1988 to May, 1992 - Records from the early period and the beginning months of the transition period consist primarily of the e-mail and other files sent to the CERT^®/CC. These messages and files were archived together in chronological order, without any other organization. For the first two years, the records also include a limited number of DARPA-requested periodic summaries. These summaries proved to be of limited use.

In order to gather data about incidents during this period, I had to create the incident records from the more than 10,000 messages in the CERT^®/CC archive. Since there was no organization to the file, I read each message in chronological order, and then processed it as follows:

1) If the message did not contain information about an incident, it was eliminated from consideration. Examples of eliminated messages include information from a user or vendor about a vulnerability, or a request for information from a user.

2) Unix search tools, such as the grep utility, were used to relate key words and phrases to the incidents already created. The primary key word used was the site name, but searches were also conducted using other distinctive words or phrases, such as the method of attack, or the name or location of the attacker.

3) If a match of key words or phrases was found, the message was compared to the incident it matched with in order to judge whether it was part of the same incident. If the message was determined to be part of that incident:

a) The message was appended to the end of the incident's file.

b) Keywords in the message were then used to search the remaining CERT^®/CC records near this time frame for further matches. If other messages were found to be related, they were also appended to the incident's file.

4) If a match of key words or phrases was not found for a message, a new incident file was created and the message was copied into it. The file was assigned a unique number that indicated the reporting date of the incident. For example, the incident file 90-054-06 would indicate the incident was first reported to the CERT^®/CC on the 54^th day of 1990 (February 23^rd). The last number, 06, indicates that it was the 6^th incident reported to the CERT^®/CC that day.

4.2.2. Later Period Records -- May, 1992 to December, 1995 - Starting in May, 1992, summaries were available for the incidents reported to the CERT^®/CC. These summaries were originated or "opened" when CERT^®/CC personnel determined that an incident had probably begun or taken place. These summaries were kept on-line in a large file of open incidents that could be accessed by all CERT^®/CC personnel. When it was determined that an incident was completed, it was marked "closed." Once a week, the CERT^®/CC archived records as follows:

1) All closed incidents were removed from the open file and placed in a separate file of closed incidents for that week.

2) The file with the remaining open incidents was then copied into a separate archived file.

In 1992, the summaries were created and maintained through manual entries by CERT^®/CC personnel. These entries included notes and excerpts from e-mail and other files sent to the CERT^®/CC. The completeness of these summaries depended upon who created and maintained them, with some being relatively detailed in their entries, which meant the summary could be used without reference to the original e-mail. Other CERT^®/CC personnel were less detailed in their entries, which made the summary shorter, but also required more frequent references to the original messages. These summaries were sometimes an incomplete record of an incident.

In 1993, the CERT^®/CC incident summaries were changed to include the CERT^®, INFO, and VUL numbers. This allowed the summaries to be initiated and maintained through an automated e-mail sorting program. Unfortunately, until the middle of 1994, this program appeared to excerpt very little from the incoming e-mail - often only the subject line. This probably required CERT^®/CC personnel to reference the original e-mail frequently. This also made the summaries a relatively incomplete record during this year.

In the summer of 1994, the summaries became more extensive. Throughout the remaining records, the summaries generally contained the bodies of the e-mails sent to the CERT^®/CC. Response personnel could probably use these summaries without reference to the original messages. In this last period, the summaries represent a relatively complete incident record.

As stated earlier, the correspondence between incidents and summaries was not one-to-one. An incident summary was opened when an incident report was received by the CERT^®/CC. Many of these summaries later proved to be related to each other. Once CERT^®/CC personnel determined that two or more summaries were related, the usual course of action was to indicate this relationship in the summaries, but to keep all the summaries open. As such, the number of summaries in the CERT^®/CC records is greater than the number of actual incidents. Occasionally, a summary was closed and the information from that summary was copied to a related summary.

In order to gather data about incidents during this period, I had to create the incident records from the CERT^®/CC summaries. Because there was more organization to the summaries than to the e-mail, it was easier to reconstruct the incidents using the summaries. I processed each CERT^®, INFO and VUL summary as follows:

1) If the summary did not contain information about an incident, it was eliminated from consideration.

2) Unix search tools, such as the grep utility, were used to relate key words and phrases to the incidents already created. For the early summaries, the primary key word used was the site name, with searches also conducted using other distinctive words or phrases, such as the method of attack, or the name or location of the attacker. After CERT^®, INFO, and VUL numbers were assigned to the summaries beginning in 1993, these numbers became the primary key words for searching.

3) If a match of key words or phrases was found, the summary was compared to the incident it matched in order to judge whether it was part of the same incident. In this process, any notes in the summary relating to other summaries were used to aid in determining the relationship. The judgment of CERT^®/CC personnel was given strong weight. For example, a common phrase was "related to CERT^®#XXX." This usually resulted in the summaries being combined into one incident. The phrases "may be related to CERT^®#XXX," or "possibly related to CERT^®#XXX" were given less weight. If the summary was determined to be part of the same incident:

a) The summary was appended to the end of the incident's file.

b) Keywords in the summary were then used to search the remaining CERT^®/CC records near this time frame for further matches. If other summaries were found to be related, they were also appended to the incident's file.

4) If a match of key words or phrases was not found for a summary, a new incident file was created and the summary was copied into it. The file was assigned a unique number that indicated the reporting date of the incident. For example, the incident file 93-035-05 would indicate the incident was reported to the CERT^®/CC on the 35^th day of 1993 (February 4^th). The last number, 05, indicates that it was the 5^th incident reported to the CERT^®/CC that day.

As noted earlier, the incident summaries from the Spring of 1993 to the Summer of 1994 were incomplete. Because of the number of incidents in this period (over 1,400), time did not allow extracting information from the original messages for these incidents. As such, for this period, the incident records created as part of this research did not give complete details.

4.3. Data Extraction

After the incidents were reconstructed from the CERT^®/CC records, I examined each of the incidents 1) to ensure that the incident was reconstructed correctly, and 2) to extract data from each incident. The following fields of data were then placed in a summary file:

1) Reporting Date - The first field in the summary file was the incident file identifier, which indicated the date the incident was reported to the CERT^®/CC. The field contained three numbers separated by the letter "i" and two dashes. Using the example file name cited earlier, an example of an entry in this data field is "i93-035-05," which indicates the incident was reported to the CERT^®/CC on the 35^th day of 1993 (February 4^th). The last number, 05, indicates that it was the 5^th incident reported to the CERT^®/CC that day.

2) Starting Date (SD) - The starting date was assumed to be the same as the reporting date, unless there was some other information in the file to indicate the incident actually began at an earlier date. If there was such information, this was used to determine the starting date. This field in the file contained two numbers separated by a dash. An example is "92-015," which indicates the incident began on the 15^th day of 1992 (January 15^th).

3) Ending Date (ED) - The ending date of an incident was more difficult to determine, particularly in the later files. Some preference was given to the date the incident was closed, but closing the incident in the CERT^®/CC summaries was an administrative function that was not necessarily related to the actual ending date of the incident. Other possibilities were to use the date of the last activity recorded in the file, or to use a date discussed in the narrative of the file. These possibilities were examined in each of the incident files to make a judgment as to the ending date. This field was also entered as two numbers separated by a dash.

4) Number of Sites (NS) - This field listed the total number of sites involved in the incident. This included both the sites that reported the incident, and the other sites involved. The majority of incidents involved two sites (60.2%): the attacking site and the attacked site. Some incidents (91 incidents, 2.1%) involved only one site, which meant the attacker was located at the site being attacked. The remaining 1,699 incidents (37.7%) involved more than two sites. More than 100 sites were involved in 31 of the incidents, and the largest incident involved more than 1,500 sites. This field was recorded as a positive integer.

5) Number of Messages (NM) - The number of messages received by the CERT^®/CC may give some indication of the CERT^®/CC workload. In some instances, the CERT^®/CC was involved in an incident only to a limited degree, even if the incident was large. For example, an incident that involved 100 sites, but only two messages to the CERT^®/CC may indicate limited CERT^®/CC involvement or workload. This field was recorded as a positive integer.

6) Reporting Sites (RS) - The site name was recorded for each site that reported the incident. In the records after 1992, this generally corresponded to the sites that were assigned a CERT^® number for the incident. The site names listed were as discussed in Chapter 2, such as cmu.edu or widgets.co.uk. For some of the sites, the site name was not available, but the IP address was. In these cases, the first two octets of the IP address were recorded instead of the site name. For example, for an IP address of "111.222.333.444," the octets "111.222" would be recorded. All of the reporting sites were listed in this field of the summary file. After the data for all incidents were extracted from the records, the site names were replaced with numbers and top-level domain names. For example, widgets.co.uk might have been replaced with 123.uk. IP addresses were replaced with a "z" domain, such as 123.z.

7) Other Sites (OS) - The incident file was examined to determine if there were other sites involved that had not reported the incident. If there were other sites that could be determined, they were listed in this field. If site names were not available, and the IP address was, then the first two octets of the IP address were entered in the field. As with reporting sites, after the data for all incidents were extracted from the records, the other site names were replaced with numbers and top-level domain names.

8) Level (LV) - Each incident was classified as discussed in Chapter 7. This was recorded as a single integer as follows:

1 root break-in 5 access attempt

2 account break-in 6 disclosure of information incident

3 denial-of-service incident 7 false alarm

4 corruption of information incident

9) Methods of Operation (MO) - CERT^®/CC personnel began recording a field of information in the CERT^®/CC incidents in 1992 called "MO." CERT^®/CC personnel used this field for two types of information. First, they recorded their judgment as to the severity of the attack. This was the level of attack which, for this research, was separated out into the Level (LV) field (discussed above). Second, CERT^®/CC personnel recorded in this field the tools and vulnerabilities used for access as depicted in Figure 6.9 in Chapter 6. If information was available in this field of the record, or in the text of the record, regarding the methods of operation used in the incident, they were recorded in the MO field of the summary file in the form of key words. In addition, the level of attack was written in key words in this MO field. Finally, a limited amount of information about attackers, results and objectives (defined in Chapter 6, and shown in Figure 6.9) were also recorded in this field. The keywords used and their frequency of occurrence are discussed in Chapter 8.

10) Corrective Actions (CA) - CERT^®/CC records gave little information as to the corrective action taken in each incident. If information was available on corrective actions taken, it was recorded in the form of key words in this field of the summary. The keywords used and their frequency of occurrence are discussed in Chapter 8.

11) CERT^® Number (CN) - The last field of data extracted from the incidents was the number or numbers assigned to the incident by CERT^®/CC personnel. As discussed earlier, assignment of these numbers began in 1993. If an incident was reported by multiple sites, typically there were multiple CERT^® numbers assigned to the incident. In addition, incidents sometimes also had VUL (vulnerability) or INFO (information) numbers assigned to them. All numbers that were assigned by CERT^®/CC personnel to the incident were listed in this field of the summary. For incidents prior to 1993, this field in the summary record is blank.

An example of a record in the summary file is shown below. This is an incident that was reported to the CERT^®/CC toward the end of 1995. The incident began four days before it was reported and it ended 40 days into 1996. Three sites reported the incident, which caused CERT^®/CC personnel to assign three CERT^® numbers to the incident. An additional 18 sites were involved. This incident was a level 3, denial-of-service incident, with methods of operation and corrective actions as shown. The example incident record is as follows:

i95-362-01 SD: 95-358 ED: 96-040 NS: 0021 NM: 0042 RS: 006.edu, 468.net, 192.net OS: 775.com, 595.com, 316.com, 348.com, 945.com, 600.com, 405.com, 1763.com, 347.com, 150.com, 011.fi, 1764.com, 815.com, 1309.com, 055.net, 097.com, 1765.com, 772.com LV: 3 MO: dos attack, mail spoofing, mail subscribing, majordomo CA: notify site, filter, police, close account CN: CERT#6995, CERT#16821, CERT#16470

4.4. Summary of CERT^®/CC Records

The organization and operation of the CERT^®/CC appears to have gone roughly through three periods: 1) an early, informal period from November 1988 to around January 1992, 2) a transitional period for the next year and a half, and 3) a more formal period beginning in the summer of 1993. CERT^®/CC records reflect these changes in organization and operation.

For incident response, all CERT^®/CC records were maintained "on-line" in the CERT^®/CC local area network. While the CERT^®/CC records were useful for the "real-time" CERT^®/CC operation, the records did not represent a source of information valuable for analysis. For this research, I had to construct the incident records from these records. In the early period, the records consisted primarily of the e-mail and other files sent to the CERT^®/CC archived together in chronological order, without any other organization. Starting in May 1992, summaries were manually created for each site reporting an incident to the CERT^®/CC. Since multiple sites could report the same incident, multiple summaries could be open for a single incident. In 1993, the CERT^®/CC incident summaries were changed to include the CERT^®, INFO, and VUL numbers. This allowed the summaries to be initiated and maintained through an automated e-mail sorting program.

Data were extracted from each incident after the incidents were reconstructed from the CERT^®/CC records. These data included reporting date, starting date, ending date, number of sites, number of messages, reporting sites, other sites, level of attack, methods of operation, corrective actions, and CERT^® number.

The next chapter develops a definition of computer security. This is followed by the development of a taxonomy of attacks in Chapter 6. In the remaining chapters, the incident records described in this chapter (Chapter 4) will be analyzed.

Back to the Table of Contents
   [4]