Previous Entry Share Next Entry
Sufficiently Advanced Monitoring is Indistinguishable from Testing
buzzed, B&W
Listening to a Test Talks podcast this morning about Terrible Testing. The guest, Todd Gardner, talks about bad test practices.

There's a slightly interesting bit where he talks about how the test pyramind isn't always a good model, but he bases that statement on the fact that the number of tests is what the test pyramid shows. Gardner goes on to talk about how risk should guide test and how test coverage can actually become more of a burden than a benefit (since writing new code might require updating a lot of tests). I don't think the test pyramid is necessarily reflective of number of tests nor effort, but just a mental model - make sure you've considered the risk and write appropriate unit tests, then move on to service level, etc.

The more interesting bit was when he said:
I can’t remember where I heard this quote, and I’d love to attribute it to somebody, but I’m just going to repeat it anyway and say that it wasn’t me but it’s really clever. “If you have a sufficiently, any sufficiently advanced, automated testing system is indistinguishable from monitoring.”
A little bit of digging, and it looks like the original presentation is from Ed Keye's 2007 Google Test Automation Conference presentation Sufficiently Advanced Monitoring is Indistinguishable from Testing (super short, quick presentation).

Overlaps a bit with some discussion we recently had at work about QuickCheck and property based testing, especially when Keyes starts talking about how generalized monitoring tests have to be.

There's clearly a lot of challenges for most web applications - e.g. how to trace a single user session/transaction from start to end, what to check for besides HTTP status codes, etc

We've definitely got some new monitoring stuff going in at work, so it'd be interesting to use writing checks against that output would be a good way to learn that. Not likely that I'll have the time to go down that route any time soon, but definitely could be fun.
Tags: , ,


Log in