"Evil" next comes from where you are blind

Image result for mordor



My long term experience with large scale distributed systems monitoring, performance modelling and prediction etc is that "Evil next comes from where you are blind". I.e. The next major performance/scalability problem will always pop up from where you have the least visibility into your system (i.e. where you are blind). In that sense it is entirely predictable! If part of your system is a black box in terms of monitoring/visibility then inevitably that's where the next problem will occur. In Middle Earth the Evil will come from Mordor.

Gaps in monitoring can be tier related (e.g. client/browser, networks, web, application, database, 3rd party services, cloud managed services), due to incomplete monitoring (e.g. some systems may not be able to have agents installed so may be "invisible"),  lack of ability to correlate monitoring data across heterogeneous systems (e.g. lack of transactional monitoring), or lack of breakdown times inside a system (e.g. using only network monitoring so can't see CPU, IO, sync, wait, suspension times inside system).

I spent 2 years trying to build an end-to-end/top-to-bottom performance model for a Telco-like client a while back. They didn't have complete application level monitoring. We tried using logs, SPLUNK, and an APM PoC (AppDynamics) to get sufficient data. In the end we managed to exclude every part of the system as having problems except the part where we had limited monitoring visibility (which was "obviously" where the problem was at the start, and at the end that was the same conclusion except we could 100% exclude all the other parts of the system).

I recently read this interesting blog (by Netsil) on distributed/cloud monitoring approaches and they pick up on some the issues.  The problem is that any network monitoring only based approach will not have sufficient visibility inside the applications to diagnose and model/predict future issues.

I think AWS X-Ray has possibilities as it can give visibility inside the manged AWS services that other agent based solutions can't, can give a breakdown of times including "throttling" (but probably not CPU, IO, etc), and may not be able to monitor client/browser and network times. What would a hybrid (e.g. Dynatrace + X-Ray) solution look like?


Comments

Post a Comment

Popular posts from this blog

Chapter 11: AWS Directory Service, Cloud Directory

Image
"Name or Number please?" So the 1st telephones systems were point to point, and then hub and spoke. You rang the local exchange, asked to talk to Auntie Jill and the "operator" (think the Matrix) would ring Auntie Jill's phone and connect you (literally with a patch cord) once Auntie Jill had answered. If you wanted to talk to someone not on the local network the operator would call another operator (how did they know which one?) and ask to them to connect and you would eventually get patched through. This was even more fun if you wanted to ring someone overseas, as you would hear the patchwork of operators, some speaking foreign languages (what fun), the the sounds getting fainter and weirder (lots of clicks, "undersea" sound effects like talking down a tube, echos, etc) until finally you heard Auntie Jill's voice saying "hello?" and you could talk to them (which was expensive, and the operators would sometimes break in and say...
Read more

AWS Solution Architecture Certification Postscript

Image
So I sat the exam today and past ok yippee. Guess there's no need to continue blogging about AWS certification stuff!? The AWS Certified Solutions Architect – Associate exam is intended for individuals with experience designing distributed applications and systems on the AWS platform. Some general observations. The real exam is somewhat harder than the practice tests and practice exam (about 10% harder?) NONE of the practice test questions appear in the real exam. The focus is only what I would call the legacy IaaS Amazon services. Almost no questions on managed services, Machine Learning, Data Analytics, Lambda, etc. Lots of questions on what I would call systems and networks admin stuff.  I spent a few months trying to learn this stuff but actually I don't think it helped, I'm not sure I got any of these questions right. E.g. the sort of qns were if you have this sort of set up (VPCs, subnets, ACLs, groups, EC2, etc) and this doesn...
Read more

Chapter 2: Amazon Simple Storage Service (S3) and Amazon Glacier Storage (for storing your beer?)

Image
Chapter 2: Amazon Simple Storage Service (Amazon S3) and Amazon Glacier Storage The Amazon Glacier Storage Service is designed for cheap long term archival storage, so you want things to stay put in it for a long time. It is a slightly odd choice of name these days given that global warming is a significant threat to Glaciers (Glaciers are "shrinking" and large dangerous chunks are falling off them as in the above image of the warning signs at a famous New Zealand Glacier (Franz Josef Glacier).  Maybe the Amazon "Glaciers" will be the only ones still around in a few 100 years. I initially found it "odd" that the AWS Solutions Architect Certification book introduced storage services before compute services.  However, in hindsight this sort of makes sense as S3 is one of the original services, is core to other services, and introduces many of the AWS concepts, and is a "managed" service so it simpler and more typical in some respects. ...
Read more