After Acceptance: Reasoning About System Outputs

Modern software development allows us to prove that new work is functionally complete. We write a set of executable specifications. We automatically execute them in the form of acceptance tests as part of our continuous delivery pipeline. When all the tests pass, we are done!

This approach is superior to what came before it but is by no means perfect. Testing frequently ends at the point of release, meaning that bugs in production can be caught late and by the end users. Pragmatism dictates that exhaustive acceptance testing is infeasible. Indeed, tests are likely to represent only a simplified version of user interactions. In production, data will almost certainly be generated by code paths that have not been fully exercised in acceptance tests. That data is usually decoupled from our acceptance testing environment. If the current version of the system generates durable data, how do we know that future versions will consider it valid and act upon it appropriately? How can we find out about bugs after acceptance, but before our customers do?

This session will walk through some techniques for bringing your testing to production. It will show you how to sanity check a live system using end to end testing, limiting interference with real user interactions and outputs. Finally it suggests ways to observe and integrate real production data into a continuous delivery pipeline and assert on the validity of the production system's outputs.