 |
|  |
V-RATE ProjectVendor Risk Assessment & Threat Evaluation for Survivable COTS-based SystemsPrinicpal Investigators: Howard F. Lipson, Nancy R. Mead, and Andrew P. Moore Problem AddressedUsing COTS components to build large, complex systems has become the standard way that systems are designed and implemented by government and industry. Much of the literature on COTS-based systems concedes that such systems are not suitable for mission-critical applications. However, they are being used in domains where significant economic damage and even loss of life are possible in the event of a major system failure or compromise. How can we assess and mitigate the risks of COTS usage and thereby improve the survivability of mission-critical COTS-based systems?Research ApproachBuilding survivable systems using COTS components is a daunting task, because the acquiring organization (its integration team, system architect, etc.) has little or no access to the artifacts of the software-engineering process used to create the components. These artifacts are the primary sources from which assurance evidence for a composite system is derived. One way to partially compensate is to use vendor-risk assessments as a tool to help build, maintain, and evolve survivable systems. Such an assessment can be used as a new source of assurance evidence of a system’s survivability [1, 2, 3].Our proposed vendor-risk assessments are based on the vendor-risk assessment and threat evaluation (V-RATE) taxonomy described below. Two broad categories are at the highest level of our taxonomy: (1) vendor-inherent risk elements and (2) vendor-risk elements that are associated with the acquiring organization’s risk-management skills. The output of an assessment based on the V-RATE taxonomy is a vendor-risk profile for the system being evaluated. We envision a large and growing collection of vendor-risk profiles tied to real-world performance histories, providing empirical data against which a newly generated risk profile can be compared. A vendor-risk profile can be used to assess the risk associated with the use of a product in a particular threat environment and to identify areas for additional risk-mitigation activities. Because a single numerical rating would not provide sufficient guidance for these risk-mitigation activities, the vendor-risk profile helps the acquiring organization to identify its risks in each of the V-RATE taxonomy areas and to consider its risk tolerance with respect to each element of the taxonomy.
The V-RATE Taxonomy
1.1.2 Independent testing organizations
1.2.2 Existence of ratings/certifications 1.2.3 Evidence of adherence to applicable industry standards and government regulations 1.2.4 Demonstrated diversity and redundancy in products and services 1.2.5 Existence of a team that deals effectively with security/survivability issues 1.4 Compliance
1.4.2 Responsiveness to requests for new features and improvements 1.4.3 Willingness to cooperate with third-party testers and certifiers
1.5.2 Evidence of skill at evaluating trustworthiness of personnel
1.6.2 Risk-management skills in dealing with subcontractors
1.7.2 Product integration stability 1.7.3 Product evolution that supports continual survivability improvement
2.1.2 Skill at evaluating vendor technical competence 2.1.3 Awareness of existing vendor ratings and certifications 2.1.4 Demonstrated diversity and redundancy in the integration of vendor products and services 2.1.5 Use of architectural tools and techniques (e.g., wrappers) to limit risks associated with a vendor product 2.1.6 Association with expert security/survivability organizations and the existence of a dedicated security/survivability group within the organization
2.2.2 Economic 2.2.3 Political and social
2.4 Exposure 2.5 Mission Alignment / Vendor Compatibility 2.6 Negotiating Skill / Bargaining Power Expected BenefitsToo many organizations take an all-or-nothing view with regard to the use of COTS components in mission-critical systems (e.g., either COTS components are never safe to use, or COTS use should be maximized). The V-RATE method provides criteria to help decide when and how COTS products can be used to build survivable systems, and to assess and mitigate the risks of COTS usage (see Figure 2). Factors that influence this decision include not only attributes of the COTS products themselves, but also attributes of the system’s mission, the vendor, the vendor’s development life-cycle process, and the acquiring organization’s risk-management skills.A highly significant external contribution to the V-RATE project was made by Professor Heidi Ellis and her master’s student, Jason Dickerson, at Rensselaer University. They performed and documented an excellent case study on the application of V-RATE to the evaluation of COTS software for a real-world, large-scale, military command and control system [4]. In summary, V-RATE provides a systematic process for COTS risk analysis and management. It supports the survivability assessment of critical COTS-based systems, and embodies a risk reduction process for COTS acquisition and integration.
2004 AccomplishmentsWith graduate student assistance, a draft requirements specification was created for a software tool to automate key aspects of the V-RATE method, so as to make the process easier to apply. An initial prototype of a web-based tool was then developed to help support the process of gathering and organizing evidence of assurance, and is a first step toward automating the process of generating, displaying, and comparing vendor-risk profiles. A pilot of the V-RATE method for an automated control systems vendor was conducted as a CMU graduate student course project. A presentation on the V-RATE method (including a description of the V-RATE tool) was given at the Information Assurance Technical Framework Forum (IATFF) [5].2005 PlansThe project team plans to continue to refine the V-RATE method to improve its effectiveness and make it easier to use. In FY2005, CERT researchers will be participating in a new, broad-based software assurance project at the SEI. Although V-RATE is an assurance methodology specifically designed for COTS-based systems, it is expected that aspects of V-RATE method can be applied in a broader context and will make a significant contribution to this new project.References[1] Lipson, H. F.; Mead, N.; & Moore, A. P. Can We Ever Build Survivable Systems from COTS Components? (CMU/SEI-2001-TN-030, ADA3399238). Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2001. http://www.sei.cmu.edu/library/abstracts/reports/01tn030.cfm.[2] Lipson, H. F.; Mead, N.; & Moore, A. P. "Can We Ever Build Survivable Systems from COTS Components?" Proceedings of the 14th International Conference on Advanced Information Systems Engineering (CAiSE’ 02). Toronto, Ontario, Canada, May 27-31, 2002. Heidelberg, Germany: Springer-Verlag (LNCS 2348), 2002. [3] Lipson, H. F.; Mead, N.; & Moore, A. P. "Assessing the Risk of COTS Usage in Survivable Systems." Cutter IT Journal 15, 5 (May 2002): 15-23. [4] Dickerson, J. & Ellis, H. A Case Study Examining the Usefulness of the Vendor Risk Assessment and Threat Evaluation (V-RATE) Taxonomy (Technical Report RH-DOES-TR 03-01). Hartford, CT: Dept. of Engineering and Science, Rensselaer (RPI) at Hartford, June 2003. http://www.rh.edu/~heidic/pubs/dickerson_vrate_0603.pdf. (pdf) [5] Lipson, H. F. "Building Survivable Systems from COTS Components." Presentation at Information Assurance Technical Framework Forum (IATFF) on Engineering Methods for Building Secure Systems. Laurel, MD, John Hopkins University Applied Physics Laboratory, June 24, 2004. Disclaimers and copyright information
Last updated May 15, 2005. |
