Chaos Sloths? (or Chaos Sheep?) Move over Chaos Monkeys! A new type of chaos engineering "agent" for performance engineering?

I recently came across this post on Chaos Engineering which got me thinking again about cloud complexity.  And this article. The idea behind both is that current software systems are so complex you just can't test them before production, you have to test them during production (and try not to annoy too many paying or advert clicking customers, or just use Kiwis as Canaries).
Has anyone thought of and tried doing chaos engineering for performance engineering? I.e. introduce performance problems on purpose in production systems and see how quickly they are detected and if the system is self-healing enough to prevent any SLAs being violated? What would you call this? Chaos Sloths?! No one appears to have thought of this before, surprising!  You would need very good APM, baseline performance data (e.g. Blue/Green), and predictive analytics etc to detect any problems before they cause major issues (e.g. SLA violations).


And "why did the Sloth cross the road"? Because he had a few hours to kill? and lots of traffic to annoy.


In New Zealand we don't have Sloths we have Sheep, lots of them (on roads often). Hint: Don't honk your horn, it won't make them go any faster. And there's safety in numbers (the sheep that is).


So maybe "Chaos Sheep?

Not much has been written about performance monitoring and testing for Canay testing, odd.
Here's something from Hawkular (Open source APM?) 
An implementation of Open Tracing, a vendor-neutral open standard for distributed tracing.
A bit more, it seems to be API based.

We have some experience with this sort of APM approach with a client. Unfortunately it has it's limitations c.f. Agent based (e.g. Dynatrace, AppDynamics, etc), as all you get is timing data (elapsed time only, not detailed time breakdowns including say CPU, IO, suspension, wait, sync, throttled etc) from one point in the code to another (where developers have made the correct call), and if they have bothered to use it at all (i.e. gaps are common). Also doesn't work for UEM (e.g. browsers).

PS
Perhaps Chaos Sloths for introducing performance delays?
And Chaos Sheep for introducing unexpected extra load?

Comments

  1. Rekha Roy22 November 2021 at 07:15

    Want to change your career in Selenium? Red Prism Group is one of the best training coaching for Selenium in Noida. Now start your career for Selenium Automation with Red Prism Group. Join training institute for selenium in noida.

    ReplyDelete

Post a Comment

Popular posts from this blog

Chapter 11: AWS Directory Service, Cloud Directory

Image
"Name or Number please?" So the 1st telephones systems were point to point, and then hub and spoke. You rang the local exchange, asked to talk to Auntie Jill and the "operator" (think the Matrix) would ring Auntie Jill's phone and connect you (literally with a patch cord) once Auntie Jill had answered. If you wanted to talk to someone not on the local network the operator would call another operator (how did they know which one?) and ask to them to connect and you would eventually get patched through. This was even more fun if you wanted to ring someone overseas, as you would hear the patchwork of operators, some speaking foreign languages (what fun), the the sounds getting fainter and weirder (lots of clicks, "undersea" sound effects like talking down a tube, echos, etc) until finally you heard Auntie Jill's voice saying "hello?" and you could talk to them (which was expensive, and the operators would sometimes break in and say...
Read more

AWS Solution Architecture Certification Postscript

Image
So I sat the exam today and past ok yippee. Guess there's no need to continue blogging about AWS certification stuff!? The AWS Certified Solutions Architect – Associate exam is intended for individuals with experience designing distributed applications and systems on the AWS platform. Some general observations. The real exam is somewhat harder than the practice tests and practice exam (about 10% harder?) NONE of the practice test questions appear in the real exam. The focus is only what I would call the legacy IaaS Amazon services. Almost no questions on managed services, Machine Learning, Data Analytics, Lambda, etc. Lots of questions on what I would call systems and networks admin stuff.  I spent a few months trying to learn this stuff but actually I don't think it helped, I'm not sure I got any of these questions right. E.g. the sort of qns were if you have this sort of set up (VPCs, subnets, ACLs, groups, EC2, etc) and this doesn...
Read more

Chapter 2: Amazon Simple Storage Service (S3) and Amazon Glacier Storage (for storing your beer?)

Image
Chapter 2: Amazon Simple Storage Service (Amazon S3) and Amazon Glacier Storage The Amazon Glacier Storage Service is designed for cheap long term archival storage, so you want things to stay put in it for a long time. It is a slightly odd choice of name these days given that global warming is a significant threat to Glaciers (Glaciers are "shrinking" and large dangerous chunks are falling off them as in the above image of the warning signs at a famous New Zealand Glacier (Franz Josef Glacier).  Maybe the Amazon "Glaciers" will be the only ones still around in a few 100 years. I initially found it "odd" that the AWS Solutions Architect Certification book introduced storage services before compute services.  However, in hindsight this sort of makes sense as S3 is one of the original services, is core to other services, and introduces many of the AWS concepts, and is a "managed" service so it simpler and more typical in some respects. ...
Read more