Modifying Without a Trace: General Audit Guidelines are Inadequate for Electronic Health Record Audit Mechanisms
J. King, B. Smith, L. Williams, “Modifying Without a Trace: General Audit Guidelines are Inadequate for Electronic Health Record Audit Mechanisms”, Proceedings of the International Health Informatics Symposium (IHI 2012), to appear, 2012.
Without adequate audit mechanisms, electronic health record (EHR) systems remain vulnerable to undetected misuse. Users could modify or delete protected health information without these actions being traceable. The objective of this paper is to assess electronic health record audit mechanisms to determine the current degree of auditing for non-repudiation and to assess whether general audit guidelines adequately address non-repudiation. We derived 16 general auditable event types that affect non-repudiation based upon four publications. We qualitatively assess three open-source EHR systems to determine if the systems log these 16 event types. We find that the systems log an average of 12.5% of these event types. We also generated 58 black-box test cases based on specific auditable events derived from Certification Commission for Health Information Technology criteria. We find that only 4.02% of these tests pass. Additionally, 20% of tests fail in all three EHR systems. As a result, actions including the modification of patient demographics and assignment of user privileges can be executed without a trace of the user performing the action. The ambiguous nature of general auditable events may explain the inadequacy of auditing for non-repudiation. EHR system developers should focus on specific auditable events for managing protected health information instead of general events derived from guidelines.
B. Smith, “Systematizing Security Test Case Planning Using Functional Requirements Phrases“, Proceedings of the International Conference on Software Engineering Doctoral Symposium (ICSE Doctoral Symposium), Honolulu, Hawaii, pp. 1136-1137, 2011.
Security experts use their knowledge to attempt attacks on an application in an exploratory and opportunistic way in a process known as penetration testing. However, building security into a product is the responsibility of the whole team, not just the security experts who are often only involved in the final phases of testing. Through the development of a black box security test plan, software testers who are not necessarily security experts can work proactively with the developers early in the software development lifecycle. The team can then establish how security will be evaluated such that the product can be designed and implemented with security in mind. The goal of this research is to improve the security of applications by introducing a methodology that uses the software system’s requirements specification statements to systematically generate a set of black box security tests. We used our methodology on a public requirements specification to create 137 tests and executed these tests on five electronic health record systems. The tests revealed 253 successful attacks on these five systems, which are used to manage the clinical records for approximately 59 million patients, collectively. If non-expert testers can surface the more common vulnerabilities present in an application, security experts can attempt more devious, novel attacks.
Using SQL Hotspots in a Prioritization Heuristic for Detecting All Types of Web Application Vulnerabilities
B. Smith, L. Williams, “Using SQL Hotspots in a Prioritization Heuristic for Detecting All Types of Web Application Vulnerabilities“, Proceedings of the International Conference on Software Testing, Verification and Validation (ICST 2011), Berlin, Germany, pp. 220-229, 2011.
Challenges for Protecting the Privacy of Health Information: Required Certification Can Leave Common Vulnerabilities Undetected
B. Smith, A. Austin, M. Brown, J. King, J. Lankford, A. Meneely, L. Williams, “Challenges for Protecting the Privacy of Health Information: Required Certification Can Leave Common Vulnerabilities Undetected“, Proceedings of the Security and Privacy in Medical and Home-care Systems (SPIMACS 2010) Workshop, co-located with CCS, Chicago, IL, pp. 1-12, 2010.
Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities
Y. Shin, A. Meneely, L. Williams, J. Osborne, “Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities”, IEEE Transactions on Software Engineering (to appear), 2010.
Security inspection and testing requires experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely-used open source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all the three types of metrics together predicted over 80% of the known vulnerable files with less than 25% false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71% and 28%, respectively, for both projects.
A. Austin, B. Smith, and L. Williams, “Towards Improved Security Criteria for Certification of Electronic Health Record Systems“. Proceedings of the Second Workshop on Software Engineering in Healthcare (SEHC), Cape Town, South Africa, 2010.
Andrew Meneely and Laurie Williams, “Strengthening the Empirical Analysis of the Relationship between Linus’ Law and Software Security”. Empirical Software Engineering & Measurement (ESEM) 2010.
Open source software is often considered to be secure because large developer communities can be leveraged to find and fix security vulnerabilities. Eric Raymond states Linus’ Law as “many eyes make all bugs shallow”, reasoning that a diverse set of perspectives improves the quality of a software product. However, at what point does the multitude of developers become “too many cooks in the kitchen”, causing the system’s security to suffer as a result? In a previous study, we quantified Linus’ Law and “too many cooks in the kitchen” with developer activity metrics and found a statistical association between these metrics and security vulnerabilities in the Linux kernel. In the replication study reported in this paper, we performed our analysis on two additional projects: the PHP programming language and the Wireshark network protocol analyzer. We also updated our Linux kernel case study with 18 additional months of newly-discovered vulnerabilities. In all three case studies, files changed by six developers or more were at least four times more likely to have a vulnerability than files changed by fewer than six developers. Furthermore, we found that our predictive models improved on average when combining data from multiple projects, indicating that models can be transferred from one project to another.
A. Meneely, M. Corcoran, L. Williams, “Improving Developer Activity Metrics using Issue Tracking Annotations” Workshop on Emerging Trends in Software Metrics (WETSoM ’10), to appear.
Understanding and measuring how groups of developers collaborate on software projects can provide valuable insight into software quality and the software development process. Current practices of measuring developer collaboration (e.g. with social network analysis) usually employ metrics based on version control change log data to determine who is working on which part of the system. Version control change logs, however, do not tell the whole story. Information about the collaborative problem-solving process is also documented in the issue tracking systems that record solutions to failures, feature requests, or other development tasks. To enrich the data gained from version control change logs, we propose two annotations to be used in issue tracking systems: solution originator and solution approver. We examined the online discussions of 602 issues from the OpenMRS healthcare web application, annotating which developers were the originators of the solution to the issue, or were the approvers of the solution. We used these annotations to augment the version control change logs and found 47 more contributors to the OpenMRS project than the original 40 found in the version control change logs. Applying social network analysis to the data, we found that central developers in a developer network have a high likelihood of being approvers. These results indicate that using our two issue tracking annotations identify project collaborators that version control change logs miss. However, in the absence of our annotations, developer network centrality can be used as an estimate of the project’s solution approvers. This improvement in developer activity metrics provides a valuable connection between what we can measure in the project development artifacts and the team’s problem-solving process.
Williams L., Meneely A., Shipley G., Protection Poker: The New Software Security “Game” in IEEE Privacy & Security 2010, to appear
Tracking organizations such as the US CERT show a continuing rise in security vulnerabilities in software, increasing awareness of insecure coding practices. Not all discovered vulnerabilities are equal – some have the potential to cause much more damage to organizations and individuals than others. In the inevitable absence of infinite resources, software development teams need to prioritize security fortification efforts to prevent the most damaging attacks. We propose the Protection Poker “game” as a collaborative means of guiding this prioritization. A case study of a Red Hat IT software maintenance team demonstrated the potential of Protection Poker for improving software security practices and team software security knowledge.
Idea: Using System Level Testing for Revealing SQL Injection-Related Error Message Information Leaks
B. Smith, L. Williams, A. Austin, “Idea: Using System Level Testing for Revealing SQL Injection-Related Error Message Information Leaks“, Lecture Notes in Computer Science, vol. 5965, Engineering Secure Software and Systems (ESSoS 2010), pp. 192-200, 2010.
Completely handling SQL injection consists of two activities: properly protecting the system from malicious input, and preventing any resultant error messages caused by SQL injection from revealing sensitive information. The goal of this research is to assess the relative effectiveness of unit and system level testing of web applications to reveal both error message information leak and SQL injection vulnerabilities. To produce 100% test coverage of 176 SQL statements in four open source web applications, we augmented the original automated unit test cases with our own system level tests that use both normal input and 132 forms of malicious input. Although we discovered no SQL injection vulnerabilities, we exposed 17 error message information leak vulnerabilities associated with SQL statements using system level testing. Our results suggest that security testers who use an iterative, testdriven development process should compose system level rather than unit level tests.