Testing

I was recently testing file upload performance, and needed several large files of different sizes to test with. To make the math easier, it was helpful if I had files with round numbered sizes like 10MB, 20MB, or 100MB. After searching around for files of the right size, it turns out the easiest solution is to generate one yourself using the Linux command line. Depending on your needs, you can use two different methods of generating files using some simple commands. ...

A few years ago, a company I worked for was suffering from quality problems. We were expanding into new areas, pushing the product in new directions, and releasing new features quickly. As we pushed out new and evolved products, quality declined. Everything came to a head when customers began openly complaining about software quality. Our CTO stepped in with a mandate to solve the quality problem. His approach? The Zero Bug Policy — no-one was allowed to work on new features until the number of bugs in the product backlog was zero. ...

Complex systems exhibit unexpected behavior. — John Gall, The Systems Bible One reaction to the myriad failure cases we encounter with distributed systems is to add more testing. Unfortunately, testing is a best-effort verification of system correctness — we simply cannot predict the failure cases that will happen in production. What’s more, any environment that we use to verify system behaviour is — at best — a pale imitation of our production environment. An alternative to adding more tests is to add more monitoring and alerting so we know as soon as possible that a system shows signs of degradation. Unfortunately, monitoring and alerting suffers from the same faults as testing — we monitor for behaviour that is predictable in nature or that we have experienced in the past. Together, monitoring and testing try to enumerate all possible permutations of partial and total system failure. Unfortunately, they only provide a simulation of how you expect a system to actually function in production. ...

In general, developers test features by focusing on the positive outcomes — the so-called “happy paths”. Unfortunately, this optimism can blind us to the less obvious or less probable outcomes that can cripple an application. I found one way to counteract this tendency is to try and “go beyond the happy path” by exploring the outcome of unhappy paths through a feature. I came across this idea in the excellent book Fifty Quick Ideas to Improve Your Tests by Gojko Adzic, David Evans, and Tom Roden, and expanded upon the book Writing Great Specifications by Kamil Nicieja. ...

The request was fairly simple — iterate through some data and update some dates. Something as simple as this. for entry in db.query(): entry.date = entry.date + timedelta(days=10) entry.put() Unfortunately, the NoSQL database we were using did not support schemas and, unbeknownst to me, some data had been changed in unexpected ways on production. Something as simple as this. Notice the Outlier? My code seemed correct. Unit tests passed. Code review passed. It ran flawlessly on the test environment. QA signed off on the change. So we deployed it. And things broke. ...

Testing concurrent objects can be challenging. One particular pattern that is useful for objects used in producer-consumer designs is to ensure that everything put in to a shared concurrent queue by a producer is correctly executed by consumers. ...

Writing unit tests for App Engine Cloud Endpoints is a fairly straight forward process. Unfortunately it is not well documented and a few gotchas exist. This article provides a template you can use to unit test Cloud Endpoints including full source code for a working example. ...