Chapter 11: Is Kinesis really suitable for Real Time Stream Processing??? Is there anybody out there?

Is there anybody out there? Are polling and real-time stream processing compatible?





At the recent AWS Summit in Sydney in May I attended an interesting talk on Kinesis.

The speaker (Johnathon Meichtry) implied that the only way get data out of Kinesis (streams?) was by polling, which when I talked with him afterwards he didn't seem to understand the potential limitations with this for real time stream processing, or be familiar with other event/stream processing platforms that didn't have this limitation. My previous experience with CEP and stream processing was that for real-time requirements polling is "evil" and that the event processing rules/applications should be triggered immediately by new events coming into the stream (i.e. the semantics are that each new event fires the processing immediately and once per event).

So what are the actual requirements for streaming data processing? The 8 requirements of Real-Time Stream Processing paper and this blog make a good case for some basic features.  I agree with one of their observations, that Polling shouldn't be a feature. But SQL? No thanks. (So maybe you can pick and choose? Maybe polling is ok in some situations?)

And it does look as if polling is a given for Kinesis (given that it's distributed and RESTful there is perhaps no way around it). But maybe Lambda is a work around?

Based on this I would say that Kinesis is NOT A REAL TIME STREAM PROCESSING SYSTEM (pity). I would like to be wrong...?

This blog looks at Kinesis in production (c.f. Kaftka). They conclude that latency and throughput are ok for a distributed real-time stream processing system (but not perfect, particularly 5 reads per second limit).

Are other stream processing platforms "real-time" or not? E.g. Spark, Storm, Kaftka?

This blog on Spark says:

However, if you look at the architecture for the demonstration, there is no poll-wait at any stage. The closing piece of advice is thus: maintain a push model throughout the architecture.

 And polling problems with similar systems.

Not sure if Kaftka requires polling or not? Possibly supports long polling which is also how Kinesis works.

Comments

  1. Rekha Roy22 November 2021 at 07:20

    Want to change your career in Selenium? Red Prism Group is one of the best training coaching for Selenium in Noida. Now start your career for Selenium Automation with Red Prism Group. Join training institute for selenium in noida.

    ReplyDelete

Post a Comment

Popular posts from this blog

Which Amazon Web Services are Interoperable?

Image
Which Amazon Web Services are Interoperable? This is a question I asked when I started studying for AWS certification, asked a few AWS certified architects at Sydney Summit 201 (the answer typically was "all of them as they all use REST APIs", really?), and have attempted to answer from the documentation. They only way I could think to answer this so far is go through the documentation for the 3500 services (according to April AWS service update) and create a dependencies graph (which services interoperate with which other services).  I've also noticed that some of the certification questions ask which for AWS X, which other AWS services interoperate with it? So obviously it's important. Obviously AWS could answer this by monitoring (E.g. with X-Ray) which services call or are called with services and create a interoperability graph and frequency of calls. I've since come up with another approach to answer this.  I've used the 16 aws reference architecture
Read more

AWS Certification glossary quiz: IAM

Image
I also keep getting stuck on IAM and key related questions so here's a quiz on IAM. Qn1  One of two possible outcomes (the other is  deny ) when an  IAM  access  policy  is evaluated. When a user makes a request to AWS, AWS evaluates the request based on all permissions that apply to the user and then returns either deny or this one. alarm permit true allo w Qn2  A web service that enables  Amazon Web Services  (AWS)  customers to manage users and user permissions within AWS. AWS user service (AUS) AWS customer service (ACS) AWS access management (AAM) AWS Identity and Access Management  (IAM) Qn3:  An  IAM   managed policy  that is created and managed by AWS. AWS management AWS policy AWS IAM policy AWS managed policy Qn4:  A web service for requesting temporary, limited-privilege credentials for  AWS Identity and Access Management  (IAM)  users or for users that you authenticate ( federated users ). AWS credential service (AWS ACS) AWS token service
Read more

AWS SWF vs Lambda + step functions? Simple answer is use Lambda for all new applications.

Image
I do like my Lamb (daaaa)! What's the difference between SWF and Lambda + step functions? Well, for a start they can be used together. Benefits and Limitations of using Lambda Tasks There are a number of benefits of using Lambda tasks in place of a traditional Amazon SWF activity: Lambda tasks don’t need to be registered or versioned like Amazon SWF activity types. You can use any existing Lambda functions that you've already defined in your workflows. Lambda functions are called directly by Amazon SWF; there is no need for you to implement a worker program to execute them as you must do with traditional activities. Lambda provides you with metrics and logs for tracking and analyzing your function executions. There are also a number of limitations regarding Lambda tasks that you should be aware of: Lambda tasks can only be run in AWS regions that provide support for Lambda. See  Lambda Regions and Endpoints  in the  Amazon Web Services General
Read more