Chaos Sloths? (or Chaos Sheep?) Move over Chaos Monkeys! A new type of chaos engineering "agent" for performance engineering?

I recently came across this post on Chaos Engineering which got me thinking again about cloud complexity.  And this article. The idea behind both is that current software systems are so complex you just can't test them before production, you have to test them during production (and try not to annoy too many paying or advert clicking customers, or just use Kiwis as Canaries).
Has anyone thought of and tried doing chaos engineering for performance engineering? I.e. introduce performance problems on purpose in production systems and see how quickly they are detected and if the system is self-healing enough to prevent any SLAs being violated? What would you call this? Chaos Sloths?! No one appears to have thought of this before, surprising!  You would need very good APM, baseline performance data (e.g. Blue/Green), and predictive analytics etc to detect any problems before they cause major issues (e.g. SLA violations).


And "why did the Sloth cross the road"? Because he had a few hours to kill? and lots of traffic to annoy.


In New Zealand we don't have Sloths we have Sheep, lots of them (on roads often). Hint: Don't honk your horn, it won't make them go any faster. And there's safety in numbers (the sheep that is).


So maybe "Chaos Sheep?

Not much has been written about performance monitoring and testing for Canay testing, odd.
Here's something from Hawkular (Open source APM?) 
An implementation of Open Tracing, a vendor-neutral open standard for distributed tracing.
A bit more, it seems to be API based.

We have some experience with this sort of APM approach with a client. Unfortunately it has it's limitations c.f. Agent based (e.g. Dynatrace, AppDynamics, etc), as all you get is timing data (elapsed time only, not detailed time breakdowns including say CPU, IO, suspension, wait, sync, throttled etc) from one point in the code to another (where developers have made the correct call), and if they have bothered to use it at all (i.e. gaps are common). Also doesn't work for UEM (e.g. browsers).

PS
Perhaps Chaos Sloths for introducing performance delays?
And Chaos Sheep for introducing unexpected extra load?

Comments

  1. Rekha Roy22 November 2021 at 07:15

    Want to change your career in Selenium? Red Prism Group is one of the best training coaching for Selenium in Noida. Now start your career for Selenium Automation with Red Prism Group. Join training institute for selenium in noida.

    ReplyDelete

Post a Comment

Popular posts from this blog

Which Amazon Web Services are Interoperable?

Image
Which Amazon Web Services are Interoperable? This is a question I asked when I started studying for AWS certification, asked a few AWS certified architects at Sydney Summit 201 (the answer typically was "all of them as they all use REST APIs", really?), and have attempted to answer from the documentation. They only way I could think to answer this so far is go through the documentation for the 3500 services (according to April AWS service update) and create a dependencies graph (which services interoperate with which other services).  I've also noticed that some of the certification questions ask which for AWS X, which other AWS services interoperate with it? So obviously it's important. Obviously AWS could answer this by monitoring (E.g. with X-Ray) which services call or are called with services and create a interoperability graph and frequency of calls. I've since come up with another approach to answer this.  I've used the 16 aws reference architecture
Read more

AWS Certification glossary quiz: IAM

Image
I also keep getting stuck on IAM and key related questions so here's a quiz on IAM. Qn1  One of two possible outcomes (the other is  deny ) when an  IAM  access  policy  is evaluated. When a user makes a request to AWS, AWS evaluates the request based on all permissions that apply to the user and then returns either deny or this one. alarm permit true allo w Qn2  A web service that enables  Amazon Web Services  (AWS)  customers to manage users and user permissions within AWS. AWS user service (AUS) AWS customer service (ACS) AWS access management (AAM) AWS Identity and Access Management  (IAM) Qn3:  An  IAM   managed policy  that is created and managed by AWS. AWS management AWS policy AWS IAM policy AWS managed policy Qn4:  A web service for requesting temporary, limited-privilege credentials for  AWS Identity and Access Management  (IAM)  users or for users that you authenticate ( federated users ). AWS credential service (AWS ACS) AWS token service
Read more

AWS SWF vs Lambda + step functions? Simple answer is use Lambda for all new applications.

Image
I do like my Lamb (daaaa)! What's the difference between SWF and Lambda + step functions? Well, for a start they can be used together. Benefits and Limitations of using Lambda Tasks There are a number of benefits of using Lambda tasks in place of a traditional Amazon SWF activity: Lambda tasks don’t need to be registered or versioned like Amazon SWF activity types. You can use any existing Lambda functions that you've already defined in your workflows. Lambda functions are called directly by Amazon SWF; there is no need for you to implement a worker program to execute them as you must do with traditional activities. Lambda provides you with metrics and logs for tracking and analyzing your function executions. There are also a number of limitations regarding Lambda tasks that you should be aware of: Lambda tasks can only be run in AWS regions that provide support for Lambda. See  Lambda Regions and Endpoints  in the  Amazon Web Services General
Read more