Continuous Deployment – Testing Practices
We identified automated testing to be an essential part of all the 19 adoptees’ deployment practice. Majority of the adoptees’ have reported to perform unit testing. Some of them also reported to perform functional testing, integration testing, A/B testing, acceptance testing, regression testing, and perceptual testing.
We briefly describe each of these terms as following:
- Unit Testing: Whether or not the adoptee is using automated tests to test individual components of the software
- Functional Testing: If the adopting company is testing the functionality of the software automatically
- Integration Testing: Whether or not the adoptee is automatically testing the integrated version of the software changes
- A/B Testing: If the adopting company is doing A/B testing, where users are getting different versions of the software and reporting their feedback.
- Acceptance Testing: Acceptance testing is used to test whether or not the software fulfills the contractual requirements of the end-user.
- Regression Testing: Regression testing is performed to check if a software change has adversely affected the functionality of the whole software.
- Perceptual Testing: Perceptual testing is a testing mechanism that automatically compares two different front-end designs of user interface.
We present the number of companies who have reported to perform the above mentioned testing practices as part of their deployment process.
Separate Testing Team
In traditional software engineering, separate testing teams or testers, are responsible for performing different kinds of tests on the developed product. In such setting, usually developers who write code, do not participate in testing. Our investigation shows continuous deployment adoptees have taken a different stance in this regard. Out of the 19 adoptees, only one company (Netflix) have reported to have a separate testing team, who test other team’s code. Rest of the adoptees have described their testing practice to be performed in an automated setting where the developer who writes the code is actively involved. The following pie chart summarizes our findings.
Links to other practices and references: