Congratulations to Ben Smith of Realsearch, who passed his oral preliminary examination!

Title: “A Pattern Catalog to Guide Black Box Testers with Security Testing”

Date: September 12, 2011
Time  8-10am
Place:  EBII, Room 3211

Examination Committee:

Dr. Laurie Williams (Chair& Advisor)
Dr. Annie Antón
Dr. Ting Yu
Dr. Mladen Vouk
Dr. Jacqueline Halladay

Abstract:

The United States is suffering from a shortage of software security experts.  The software development community needs a vehicle for knowledge transfer with which security experts can proliferate their insights. We have adapted the notion of a software design pattern to the domain of black box security testing. Expressing proven security testing techniques as patterns makes them more accessible to people who are not experts in security.  Patterns also make it easier to reuse successful testing strategies in different systems, and in different areas of the same system. The goal of this research is to help security experts proliferate their knowledge by introducing and evaluating a software security test pattern catalog. In this paper, we present the first six test patterns based on tests that target a list of common vulnerabilities. These patterns contain keywords, that when found in software system’s natural language artifacts, guide testers toward the appropriate pattern. We created test cases from these patterns using 284 functional requirements from a public specification to generate 137 black box tests. We then executed these tests on five electronic health record systems (685 test results in total), which are currently used to manage the clinical records for approximately 59 million patients, collectively. Thirty-seven percent (37%) of these tests revealed 253 vulnerabilities in the five systems. Our evaluation shows that our patterns target different vulnerabilities than automated techniques like automated penetration testing and static analysis.  Our preliminary pattern catalog provides a foundation for disseminating security expertise that can be expanded with contributions from the community.

Y. Shin, A. Meneely, L. Williams, J. Osborne, “Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities”, IEEE Transactions on Software Engineering (to appear), 2010.

Security inspection and testing requires experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations.  If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics.  We performed two empirical case studies on large, widely-used open source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all the three types of metrics together predicted over 80% of the known vulnerable files with less than 25% false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71% and 28%, respectively, for both projects.

Andrew Meneely and Laurie Williams, “Strengthening the Empirical Analysis of the Relationship between Linus’ Law and Software Security”. Empirical Software Engineering & Measurement (ESEM) 2010.

Open source software is often considered to be secure because large developer communities can be leveraged to find and fix security vulnerabilities. Eric Raymond states Linus’ Law as “many eyes make all bugs shallow”, reasoning that a diverse set of perspectives improves the quality of a software product. However, at what point does the multitude of developers become “too many cooks in the kitchen”, causing the system’s security to suffer as a result? In a previous study, we quantified Linus’ Law and “too many cooks in the kitchen” with developer activity metrics and found a statistical association between these metrics and security vulnerabilities in the Linux kernel. In the replication study reported in this paper, we performed our analysis on two additional projects: the PHP programming language and the Wireshark network protocol analyzer. We also updated our Linux kernel case study with 18 additional months of newly-discovered vulnerabilities. In all three case studies, files changed by six developers or more were at least four times more likely to have a vulnerability than files changed by fewer than six developers. Furthermore, we found that our predictive models improved on average when combining data from multiple projects, indicating that models can be transferred from one project to another.

Our very own Ben Smith was recently awarded a prestigious 2010 IBM Ph.D. Fellowship Award, totaling around $30k.  Congratulations, Ben! More info can be found here.

Andy Meneely took first place at the ACM Grad Student Research Competition @ SIGCSE 2010 in Milwaukee, WI.

A. Meneely, M. Corcoran, L. Williams, “Improving Developer Activity Metrics using Issue Tracking Annotations” Workshop on Emerging Trends in Software Metrics (WETSoM ’10), to appear.

Understanding and measuring how groups of developers collaborate on software projects can provide valuable insight into software quality and the software development process. Current practices of measuring developer collaboration (e.g. with social network analysis) usually employ metrics based on version control change log data to determine who is working on which part of the system. Version control change logs, however, do not tell the whole story. Information about the collaborative problem-solving process is also documented in the issue tracking systems that record solutions to failures, feature requests, or other development tasks. To enrich the data gained from version control change logs, we propose two annotations to be used in issue tracking systems: solution originator and solution approver. We examined the online discussions of 602 issues from the OpenMRS healthcare web application, annotating which developers were the originators of the solution to the issue, or were the approvers of the solution. We used these annotations to augment the version control change logs and found 47 more contributors to the OpenMRS project than the original 40 found in the version control change logs. Applying social network analysis to the data, we found that central developers in a developer network have a high likelihood of being approvers. These results indicate that using our two issue tracking annotations identify project collaborators that version control change logs miss. However, in the absence of our annotations, developer network centrality can be used as an estimate of the project’s solution approvers. This improvement in developer activity metrics provides a valuable connection between what we can measure in the project development artifacts and the team’s problem-solving process.

Williams L., Meneely A., Shipley G., Protection Poker: The New Software Security “Game” in IEEE Privacy & Security 2010, to appear

Tracking organizations such as the US CERT show a continuing rise in security vulnerabilities in software, increasing awareness of insecure coding practices. Not all discovered vulnerabilities are equal – some have the potential to cause much more damage to organizations and individuals than others. In the inevitable absence of infinite resources, software development teams need to prioritize security fortification efforts to prevent the most damaging attacks. We propose the Protection Poker “game” as a collaborative means of guiding this prioritization. A case study of a Red Hat IT software maintenance team demonstrated the potential of Protection Poker for improving software security practices and team software security knowledge.

R.A. Syed, B. Robinson, L. Williams, “Does Hardware Configuration and Processor Load Impact Software Fault Observability?,” Proceedings of Third International Conference on Software Testing, Verification and Validation (ICST 2010), To Appear.

Abstract

Intermittent failures and nondeterministic behavior complicate and compromise the effectiveness of software testing and debugging. To increase the observability of software faults, we explore the effect hardware configurations and processor load have on intermittent failures and the nondeterministic behavior of software systems. We conducted a case study on Mozilla Firefox with a selected set of reported field failures. We replicated the conditions that caused the reported failures ten times on each of nine hardware configurations by varying processor speed, memory, hard drive capacity, and processor load. Using several observability tools, we found that hardware configurations that had less processor speed and memory observed more failures than others. Our results also show that by manipulating processor load, we can influence the observability of some faults.

Andrew Meneely and Laurie Williams.  Computer and Communications Security (CCS) 2009.

Open source software is often considered to be secure. One factor in this confidence in the security of open source software lies in leveraging large developer communities to find vulnerabilities in the code. Eric Raymond declares Linus’ Law “Given enough eyeballs, all bugs are shallow.” Does Linus’ Law hold up ad infinitum? Or, can the multitude of developers become “too many cooks in the kitchen”, causing the system’s security to suffer as a result? In this study, we examine the security of an open source project in the context of developer collaboration. By analyzing version control logs, we quantified notions of Linus” Law as well as the “too many cooks in the kitchen” viewpoint into developer activity metrics. We performed an empirical case study by examining correlations between the known security vulnerabilities in the open source Red Hat Enterprise Linux 4 kernel and developer activity metrics. Files developed by otherwise-independent developer groups were more likely to have a vulnerability, supporting Linus’ Law. However, files with changes from nine or more developers were 16 times more likely to have a vulnerability than files changed by fewer than nine developers, indicating that many developers changing code may have a detrimental effect on the system’s security.

Congratulations to Laurie on receiving the first ACM SIGSOFT Influential Educator Award at ICSE 2009!  Check out the CSC news story.