Testing in Production — Building Observable Distributed Systems

Complex systems exhibit unexpected behavior. — John Gall, The Systems Bible One reaction to the myriad failure cases we encounter with distributed systems is to add more testing. Unfortunately, testing is a best-effort verification of system correctness — we simply cannot predict the failure cases that will happen in production. What’s more, any environment that we use to verify system behaviour is — at best — a pale imitation of our production environment. [Read More]

Improving Test Coverage Using Exploratory Outcomes

In general, developers test features by focusing on the positive outcomes — the so-called “happy paths”. Unfortunately, this optimism can blind us to the less obvious or less probable outcomes that can cripple an application. I found one way to counteract this tendency is to try and “go beyond the happy path” by exploring the outcome of unhappy paths through a feature. I came across this idea in the excellent book Fifty Quick Ideas to Improve Your Tests by Gojko Adzic, David Evans, and Tom Roden, and expanded upon the book Writing Great Specifications by Kamil Nicieja. [Read More]

Tests are Never Enough

The request was fairly simple — iterate through some data and update some dates. Something as simple as this. for entry in db.query(): entry.date = entry.date + timedelta(days=10) entry.put() Unfortunately, the NoSQL database we were using did not support schemas and, unbeknownst to me, some data had been changed in unexpected ways on production. Something as simple as this. Notice the Outlier? My code seemed correct. [Read More]

Unit Testing Cloud Endpoints

Writing unit tests for App Engine Cloud Endpoints is a fairly straight forward process. Unfortunately it is not well documented and a few gotchas exist. This article provides a template you can use to unit test Cloud Endpoints including full source code for a working example.

[Read More]