Gourmet Crow, or Wearing a Different Hat

For readers to whom English isn't their first language, we use the phrase "eating crow" to describe a situation when you must admit that you were wrong after taking a rather strong position about something. While this isn't exactly that case, hence the second title, it does illustrate a lesson in perspective.

First Hat - "Sherpa"

While I was working at Shopify, one area on which I focused was product quality. The good news was that there were (and still are) tons and tons of automated tests for the system. What I felt needed improvement, though, was a more widespread application of testing practices to ensure that the system worked as intended beyond a superficial run on a developer's laptop.

This differentiation is likely why people like Michael Bolton prefer to use the term "check" rather than test, because "testing" is a very different activity. Using that terminology, my view at the time was that Shopify had plenty of checks but not enough testing.

Like any large system, there were defects and my position was that the group needed to improve their testing skills in order to prevent more defects from making it to production.

Second Hat - "Merchant"

Then something rather funny happened. After I left Shopify and rejoined the consulting world, I decided to use my own Shopify store as the platform for my services and some fun Agile swag ideas in The Agile Store. I was no longer viewing the system from the perspective of an insider who knew about the defects, but rather as a customer working with the product.

In 2 weeks of setting up and tweaking my store, I believe I encountered one single "glitch" that I barely even noticed and easily worked around. Even my critical eye was surprised at how smoothly it all worked. That isn't to say that the software is perfect - I know that by entering boundary or out of range values in some places I can cause some errors - but in everyday use from the perspective of a consumer of the software, it works not just fine but quite well indeed.

It was a good lesson on the need to balance striving for perfection from a development perspective vs. striving for the best customer experience from a business perspective.

Of course, you have to determine the optimum balance for your particular domain.  For example, it's tax season, and let's think about the tax return preparation software you use. Imagine that it has a defect that only occurs under very specific circumstances of income and family size, etc., and it leads you to believe that you're going to receive a generous refund. You happily submit your return and await that nice fat cheque! Except the software was wrong and you actually owe money, so you could be charged interest and possibly penalties. Not only that, the information submitted by the faulty tax return software tweaked something that triggered an audit by your tax authority.

That would likely be considered a suboptimal customer experience, to say the least.

There are domains and circumstances where we can put up with plenty of defects and just keep going. If a game crashes, we likely just start over. If we really like the game, our tolerance of defects is much greater.

And there's always the good old standby of the air traffic control system. Not only does the customer experience have to be good, i.e. the air traffic controllers can easily understand what they see and make appropriate inputs, but the defect level of the software has to be extremely low.

So, have the discussion about the level of tolerance of defects within your domain. At the same time, strive for the best possible software. Even after my experience as a consumer, I would still advocate for better testing. The trick is finding the sweet spot between perfect software and what allows the business to grow sustainably. I still maintain that we can drive software defects to near-zero levels at a reasonable cost, but perhaps my lesson in this case has tempered the passion with which I convey that message.

Oh, and does anyone have any wine pairing suggestions for crow?