AWS Data Analytics Services: Amazon Kinesis

Data Analytics Services: Amazon Kinesis

Kinesis (in biology), a movement or activity of a cell or an organism in response to a stimulus

Chapter 11 "Additional Key Services" is an odd mixture of services including Analytics.
(Note to self: Perhaps it would be good idea to group all the services that could be used for analytics in one chapter?)

Amazon Kinesis is a complex stream/event processing system made up of Firehose, Streams, and Analytics.

I've had some experience with event/steam systems before (e.g. Esper, Coral8, OGC sensor web services, etc).  One of the key features is being able to run multiple complex temporal queries and use sliding windows to limit the event range. Kinesis appears to have a good range of temporal operators and sliding windows.

Here's a good explanation and visualisation of the temporal operators:
http://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-temporal-predicate.html 

(I wonder what the theoretical temporal logic model underpinning it is? I've used Allen's temporal logic before for machine learning and a temporal file system, is it equivalent? I also recall that one of the problems with complex temporal queries was that it gets complex very quickly to understand and debug and validate what's going on, I had an idea once for a GUI that would enable better visualization of temporal queries and results, I wonder if this is a thing yet? Here's an older paper, and a more recent paper).

Sliding (and Tumbling!) windows are described here:
http://docs.aws.amazon.com/kinesisanalytics/latest/dev/sliding-window-concepts.html

Unfortunately the Solutions Architect book only devotes just over 1 page to Kenesis and in practice based on past experience stream event processing is not a trivial concept to pick up quickly. One of the original works on the subject is "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, David Luckham, 2001". 

I'm also not 100% convinced that a SQL-like language is the best choice for stream processing (ok I admit I just don't like SQL).

I have had some past exposure to Kafka so I wondered how Kinesis compares?
One comparison also includes AWS SQS and concludes that in terms of cost Kafka may come out ahead, "Evaluating Message Brokers: Kafka vs. Kinesis vs. SQS"
https://www.opsclarity.com/evaluating-message-brokers-kafka-vs-kinesis-vs-sqs/

Of course you can run Kafka on AWS: https://aws.amazon.com/kafka/ 

And this is an interesting comparison of message queue architectures on aws:
http://tech.forter.com/comparing-message-queue-architectures-on-aws/

Another important question is "Do Kafka and Kinesis use the same query language?" If not there will some effort migrating from one to the other. Oddly enough I can't (yet) find the answer to this question.

What's the advantage of Kinesis? Probably that it's a managed service which scale indefinitely.

Also that's it's not TAXIS (another related but different biological term)