RAVE

Network Flow Visualization

Problem Addressed

Network situational awareness depends on informed and effective analysis of network activity. Network monitoring and measurement tools provide copious information on the behavior of actors in instrumented networks. Sifting through this data for useful, actionable information presents a constant challenge. Automated analysis approaches sometimes yield excellent results that would be difficult to produce with visual inspection. One example is CERT’s work on detecting scanning activity [1].

Nonetheless, the human eye can quickly and reliably discern detail in extremely dense information sets [2]. It is one of the most sensitive instruments available to network and security operations. Automation can assist in analysis by selecting and manipulating data and by generating visualizations.

The Retrospective Analysis and Visualization Environment (RAVE) is an operational environment for generating visualizations and making them available to presentation applications. To facilitate rapid workflow, RAVE caches visualizations and intermediate analysis results for reuse in different contexts, for repeated viewings, or for browsing of related data. We have used RAVE to improve client operations and as a springboard to investigate a different problem—how to improve the visualization of network monitoring data.

Research Approach

The core of RAVE is a file-based cache containing intermediate data (for use in RAVE analyses) and visualization data that external applications consume and present to the end user. This cache is presented to the Python-based programming environment through an API based on Python decorators [3]. The cache then stores the results of these analyses. If more complex analyses are composed of simpler ones, the return values of the simpler analyses will be implicitly cached and reused by any other code which invokes the analysis with the same arguments.

To return the results of analyses to end users, we provide raved, an HTTP-based visualization service provider. Presentation applications (usually web based) retrieve analyses from raved in a manner similar to the retrieval of static images. If a cached visualization exists, it is returned; otherwise, the analysis that generates that visualization executes, and that result is cached and returned instead. The raved application is both a working service provider and a proof of concept; other service providers could quickly be written for other network service architectures, such as XML Remote Procedure Calling protocol (XMLRPC).

Time-Series Network Visualization

The analysis and visualization framework RAVE provides gives us a starting point for addressing problems in network flow visualization. One area in which we have focused our efforts is the visualization of network flow volume from disparate sources. The results of this work are available to RAVE users in the RAVE visualization layer.

Especially in large networks, a particular challenge in visualizing network flow volume over time is its high variability. We’ve found through our observations in client networks that flow volume can briefly deviate significantly from the normal flow volume for a sample: either positively (e.g., if a large file transfer during a period of inactivity increases the observed number of bytes) or negatively (e.g., if a router or sensor goes down). If we scale the visualization to include these important but obvious events, less obvious changes in flow volume become less noticeable.

We can remedy this by denoting such obvious events without directly plotting them on the scale. Carets at the top of the scale denote observations that fall outside the 95th percentile of observations for this time period. With the outliers noted but not plotted, detail that would have been lost is now clearly visible.

We see another common challenge in time-series network information when comparing the traffic volume of different sensors for the same time period. Frequently, the median volume between sensors is quite disparate. When the two series are plotted on the same axis, we encounter a problem similar to the one above; any scale that can accommodate both series “flattens” detail within the series, making it difficult, for example, to identify proportional changes that are highly correlated between the two series.

However, we wish to retain information on the magnitude of traffic observed by the sensors relative to one another. One approach is to adjust the scale to show both relative traffic magnitude and detail within each series. Alternately, we can plot both series over one another on different scales.

These approaches work for some data sets; for others with more variance within the data and/or similar magnitudes between the two data sets, observations within the two data sets may overlap in ways that make the visualization confusing. This is undesirable for automatic visualization; the approach should generate an informative data visualization in all but the rarest cases. Our response to this is to present independent series of observations in independent axes, all scaled equally by time and differently depending on the observations in the series. Because the scales are independent, it is impossible for one series to affect the scaling of another.

The maximum plotted value for each scale is available at the upper left of each plot for comparison. The technique described above for handling outlier observations prevents them from washing out detail within individual series.

Figure 1 illustrates our approach. Changes over time are clear and comparable. By plotting events outside the 95th percentile of observed activity outside the scale, we preserve a more detailed characterization of activity.

Figure 1: Incoming Network Traffic by Region

Non-Time-Series Network Visualization

An exciting area for future research is visualizing non-time-series network monitoring information. One example of such research is the host characterization plot.

Figure 2 shows a radar plot [4] characterizing host activity over a given time period on seven axes. The axes on the left denote network activity in which the host behaved as a server, while the axes on the right denote activity as a client. Each side is broken down into web-related, mail-related, or other activity. The axis pointing straight down is activity that cannot be classified as either client or server behavior; this is implicitly “other” activity as well, as we cannot determine whether it is mail or web related. The labels at the end of the “other” axes display the most prevalent type of traffic in that sample, which can be TCP or user datagram protocol (UDP) traffic to/from a given port; Internet control message protocol (ICMP) traffic of a given message type; or traffic corresponding to a different protocol, in which case we display the Internet Assigned Numbers Authority (IANA) protocol number as listed in the IP header.

Figure 2: Characterization of a Specific Network Host

Expected Benefits

RAVE is most useful in operational environments, where we intend for it to facilitate good decision making by network and security operations personnel. In these environments, personnel frequently look at the same or very similar data repeatedly, either to review another’s work or to refine an interesting observation from another analysis. In these environments, working from cached data can reduce time the operations personnel spend waiting for analyses to run, improving response time to network events. Caching also reduces load on the primary data store, improving performance for those queries which it must perform.

RAVE can provide visualization capability to analysis tools that currently have none, without introducing a requirement for a specific user interface. This includes the tools in CERT’s NetSA Security Suite [5]; in addition, work on RAVE includes development of Python-based methods for manipulating data with these tools.

2006 Accomplishments

RAVE has been deployed with a sponsor as the analysis backend to a web portal. Feedback from this deployment has been used to improve RAVE’s robustness and scalability. The portal is currently in a pre-production phase, with rollout expected to occur in 1Q 2007.

To facilitate further integration of RAVE into a web application, we have developed an Apache module to proxy RAVE connections. This allows us to use Apache for authentication and access control.

2007 Plans

We expect to further deploy RAVE with additional clients during 2007. We will integrate new analyses with RAVE and explore the feasibility of other user interface techniques such as AJAX or thick-client user interfaces.

References

[1] Gates, Carrie; McNutt, Josh; Kadane, Joseph B.; & Kellner, Marc. Detecting Scans at the ISP Level (CMU/SEI-2006-TR- 005). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2006. http://www.sei.cmu.edu/pub/documents/06.reports/pdf/06tr005.pdf.

[2] Tufte, Edward. The Visual Display of Quantitative Information, 161-162. Cheshire, CT: Graphics Press, 2001.

[3] Smith, Kevin D.; Jewitt, Jim; Montanaro, Skip; & Baxter, Anthony. Decorators for Functions and Methods. http://www.python.org/dev/peps/pep-0318.

[4] Wilkinson, Leland. The Grammar of Graphics, 2nd ed., 214- 215. New York, NY: Springer Science+Business Media, Inc., 2005.

[5] CERT Network Situational Awareness Group, Software Engineering Institute. Monitoring for Large-Scale Networks. http://tools.netsa.cert.org.

Disclaimers and copyright information

Last updated May 4, 2007