Testing in Production? Are you crazy?

Session Category :  Keynote 
March 17, 2021
13:00  -  13:45

Up until the last decade or so, it was thought that the best practice for introducing new software versions into prod was to first test them in a “full replica” of the production environment. The idea was, eliminate “all” chances of failures through exhaustive testing ahead of time. But if we are honest with ourselves, this testing environment was never completely the same as prod. Furthermore, maintaining such a test environment is extraordinarily expensive and a single such replica simply cannot not scale to today’s required rate of change (I once worked with a large enterprise that maintained three “exact” replicas of prod – at great cost).

In the last decade, a host of technology leaders – the FANG and many more – have taken a markedly different approach. Instead of employing operational practices that depend on problems being fully fettered out before software makes it to prod, they establish tools and
processes that allow them to deal with the inevitable troubles that will arise when it gets there. Don’t get me wrong, they still thoroughly test their code, but they make no attempt to do so in that mythical replica of prod. What they have done is embraced a simple truth:

You never really know if it’s going to work in prod until, well, you deploy into prod.

I’m going to ask you to embrace this mindset too. When you do, you will have to acknowledge that things will inevitably go wrong in production. But with proper preparation – with safety nets in place – you’ll be able to make corrections before any major problems
result. The moment that corrective actions no longer feel out of the ordinary, you will have transformed.

Yes, testing in production is for you.