Chapter 11: Is Kinesis really suitable for Real Time Stream Processing??? Is there anybody out there?

Is there anybody out there? Are polling and real-time stream processing compatible?





At the recent AWS Summit in Sydney in May I attended an interesting talk on Kinesis.

The speaker (Johnathon Meichtry) implied that the only way get data out of Kinesis (streams?) was by polling, which when I talked with him afterwards he didn't seem to understand the potential limitations with this for real time stream processing, or be familiar with other event/stream processing platforms that didn't have this limitation. My previous experience with CEP and stream processing was that for real-time requirements polling is "evil" and that the event processing rules/applications should be triggered immediately by new events coming into the stream (i.e. the semantics are that each new event fires the processing immediately and once per event).

So what are the actual requirements for streaming data processing? The 8 requirements of Real-Time Stream Processing paper and this blog make a good case for some basic features.  I agree with one of their observations, that Polling shouldn't be a feature. But SQL? No thanks. (So maybe you can pick and choose? Maybe polling is ok in some situations?)

And it does look as if polling is a given for Kinesis (given that it's distributed and RESTful there is perhaps no way around it). But maybe Lambda is a work around?

Based on this I would say that Kinesis is NOT A REAL TIME STREAM PROCESSING SYSTEM (pity). I would like to be wrong...?

This blog looks at Kinesis in production (c.f. Kaftka). They conclude that latency and throughput are ok for a distributed real-time stream processing system (but not perfect, particularly 5 reads per second limit).

Are other stream processing platforms "real-time" or not? E.g. Spark, Storm, Kaftka?

This blog on Spark says:

However, if you look at the architecture for the demonstration, there is no poll-wait at any stage. The closing piece of advice is thus: maintain a push model throughout the architecture.

 And polling problems with similar systems.

Not sure if Kaftka requires polling or not? Possibly supports long polling which is also how Kinesis works.

Comments

  1. Rekha Roy22 November 2021 at 07:20

    Want to change your career in Selenium? Red Prism Group is one of the best training coaching for Selenium in Noida. Now start your career for Selenium Automation with Red Prism Group. Join training institute for selenium in noida.

    ReplyDelete

Post a Comment

Popular posts from this blog

Chapter 11: AWS Directory Service, Cloud Directory

Image
"Name or Number please?" So the 1st telephones systems were point to point, and then hub and spoke. You rang the local exchange, asked to talk to Auntie Jill and the "operator" (think the Matrix) would ring Auntie Jill's phone and connect you (literally with a patch cord) once Auntie Jill had answered. If you wanted to talk to someone not on the local network the operator would call another operator (how did they know which one?) and ask to them to connect and you would eventually get patched through. This was even more fun if you wanted to ring someone overseas, as you would hear the patchwork of operators, some speaking foreign languages (what fun), the the sounds getting fainter and weirder (lots of clicks, "undersea" sound effects like talking down a tube, echos, etc) until finally you heard Auntie Jill's voice saying "hello?" and you could talk to them (which was expensive, and the operators would sometimes break in and say
Read more

AWS Solution Architecture Certification Postscript

Image
So I sat the exam today and past ok yippee. Guess there's no need to continue blogging about AWS certification stuff!? The AWS Certified Solutions Architect – Associate exam is intended for individuals with experience designing distributed applications and systems on the AWS platform. Some general observations. The real exam is somewhat harder than the practice tests and practice exam (about 10% harder?) NONE of the practice test questions appear in the real exam. The focus is only what I would call the legacy IaaS Amazon services. Almost no questions on managed services, Machine Learning, Data Analytics, Lambda, etc. Lots of questions on what I would call systems and networks admin stuff.  I spent a few months trying to learn this stuff but actually I don't think it helped, I'm not sure I got any of these questions right. E.g. the sort of qns were if you have this sort of set up (VPCs, subnets, ACLs, groups, EC2, etc) and this doesn
Read more

Chapter 2: Amazon Simple Storage Service (S3) and Amazon Glacier Storage (for storing your beer?)

Image
Chapter 2: Amazon Simple Storage Service (Amazon S3) and Amazon Glacier Storage The Amazon Glacier Storage Service is designed for cheap long term archival storage, so you want things to stay put in it for a long time. It is a slightly odd choice of name these days given that global warming is a significant threat to Glaciers (Glaciers are "shrinking" and large dangerous chunks are falling off them as in the above image of the warning signs at a famous New Zealand Glacier (Franz Josef Glacier).  Maybe the Amazon "Glaciers" will be the only ones still around in a few 100 years. I initially found it "odd" that the AWS Solutions Architect Certification book introduced storage services before compute services.  However, in hindsight this sort of makes sense as S3 is one of the original services, is core to other services, and introduces many of the AWS concepts, and is a "managed" service so it simpler and more typical in some respects.
Read more