AWS and Blue/Green deployments, A/B and Canary Testing

(How come the canary is so clean?!)

Once upon a time there were two canaries called Blue and Green who lived in a cage on the outside wall of a house. One night there was a big storm and the next morning the cage was found lying on the ground, folded completely flat, with no canaries in sight (i.e. flat inside the cage). The End. A True Story.  I also had other pets including tropical fish, a stick insect and a green tree frog (which only ate moving food, all I had was cockroaches, it would eat some pretty big ones!) The frog ended up in the Vice Chancellors garden pond at UNSW when I had to go home to NZ for Xmas one year I hope it was happier than the canaries. Since then my kids have had guinea pigs, rabbits, hermit frogs and we now have a proper pet the Hound Dog Gasher (who has his own facebook page), he's a Dalmatian, Labrador, Kelpie cross and is certainly not going to help out with any testing (apart from food testing which he is more than happy to oblige with).

One thing I wish the AWS solution architect certification course did differently is to provide some up front motivation and architectural design patterns to make sense of the individual services. In common use now are Blue/Green deployments, A/B testing, and Canary testing so I wondered how AWS does these, particularly in the context of continuous deployment and high availability.

Blue/Green deployment

The idea behind Blue/Green deployment is to have two identical environments, change the code on one at a time, and switch all the users to the new code. If there's a problem you can switch back quickly to the old code base, and you can hopefully test the code before release in a production identical environment to reduce the chance of problems. A lot has been written about this and a few different services can be used as these links suggest:

https://www.thoughtworks.com/insights/blog/implementing-blue-green-deployments-aws

https://d0.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf

https://aws.amazon.com/about-aws/whats-new/2017/01/aws-codedeploy-introduces-blue-green-deployments/

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html

https://www.slideshare.net/AmazonWebServices/dvo401-deep-dive-into-bluegreen-deployments-on-aws

A/B Testing (or Split  testing)

This is where two versions of an app or web page get tested at once with the same users and the best is picked to win. How do you measure which is best? Lots of metrics and data analytics I guess.

Not as much about A/B testing on AWS:

http://blog.gingerlime.com/2016/a-scaleable-ab-testing-backend-in-100-lines-of-code-and-for-free/ 

https://aws.amazon.com/blogs/devops/introducing-application-load-balancer-unlocking-and-optimizing-architectures/

https://techcrunch.com/2013/11/05/amazon-debuts-a-cross-platform-app-analytics-service-with-ab-testing-bundled-in/

http://deliveroo.engineering/2016/09/19/ab-testing-cdns.html (Deliveroo, not sure if they are running this AWS now or not?)

Canary Testing/Deployment (ha ha)

Cartoon link

Canary testing is what you do when you have no other way of finding out if the code changes work or not, so you try them out live with a subset of real users and get feedback. Facebook does this with the entire population of New Zealand (ha ha).   (Facebook also uses an approach called "fear of the chief reliability engineer" as she is reputed to chase anyone down who breaks a release, even if they are on holidays say in Alaska, and bring them back to fix it!) Maybe someone conducted "Moa testing" in New Zealand a long time ago, which explains how the Moa very large flightless birds) became extinct?!

There's not much that I could find on AWS canaries. However, Netflix relies heavily on automation of deployment and metrics analysis which are both probably essential:

https://engineering.klarna.com/simple-canary-releases-in-aws-how-and-why-bf051a47fb3f

https://medium.com/netflix-techblog/deploying-the-netflix-api-79b6176cc3f0

Ah, lots of people refer to this as Canary Deployment so a few more links:

http://engineering.plumbee.com/blog/2013/11/15/how-we-do-canary-deployments/

https://github.com/gruntwork-io/module-ecs-public/tree/master/modules/ecs-service 

PS
These all seem to be rather "binary", and I wonder what would happen if you try and do Blue/Green/Indigo/Violet deployments, A/B/C/D/.../Z testing, or combined them (e.g. Blue/Green/A/B/Canary testing? I.e. two versions of a new feature tested at once with ability to roll back to previous versions seamlessly for either or both).

How about support for continuous incremental DevOps/code/feature updates combined with any/all permutations of above and roll-back to any previous combination/state? Too complex?