Chapter 2: Amazon Simple Storage Service (S3) and Amazon Glacier Storage (for storing your beer?)

April 13, 2017

Chapter 2: Amazon Simple Storage Service (Amazon S3) and Amazon Glacier Storage

The Amazon Glacier Storage Service is designed for cheap long term archival storage, so you want things to stay put in it for a long time. It is a slightly odd choice of name these days given that global warming is a significant threat to Glaciers (Glaciers are "shrinking" and large dangerous chunks are falling off them as in the above image of the warning signs at a famous New Zealand Glacier (Franz Josef Glacier). Maybe the Amazon "Glaciers" will be the only ones still around in a few 100 years.

I initially found it "odd" that the AWS Solutions Architect Certification book introduced storage services before compute services. However, in hindsight this sort of makes sense as S3 is one of the original services, is core to other services, and introduces many of the AWS concepts, and is a "managed" service so it simpler and more typical in some respects.

A good overview of S3 is here.

Some of my initial notes on S3:

AWS S3 is an object store. You can have multiple "buckets", each buckets contains multiple objects. It's a sort of key-value store as each key is the "bucket name" and the value is the data stored. Objects are stored within a single region only but in > 2 locations so the data can be retrieved even if you loose 2 locations.

It has a REST API using standard CRUD semantics: PUT (create), POST (change/update), DELETE (remove), and GET (retrieve) operations.

All object have a URL made up of: bucketname.s3.amazonaws.com/keyname

Data can be deleted but there is a versioning mechanism.

Some of my initial questions were about cost, speed, and if it's transactional.

There is a potential issue with eventual consistency as it's a distributed data store so you can't have high durability with distributed data and immediate consistency. I.e. PUT, READS,and DELETE to existing objects may give stale data. But is ok for new objects.

How long does eventually consistency take in the worst case? Is there a client side mechanism that can be used to get around this issue? Consistency is discussed in more detail here and may be region dependent? http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyMode

Netflix has a potential solution documented here: http://techblog.netflix.com/2014/01/s3mper-consistency-in-cloud.html

And here is a good introduction to internet scale conistency: http://cloudacademy.com/blog/consistency-models-of-amazon-cloud-services/

S3 can be used for static web hosting, this looks like an interested use case.

S3 supports multiple "storage classes". These are tradeoffs between availability, durability, read speed and cost.

S3 supports lifecycle configuration rules to allow automatic transition of objects from one class to another and eventual deletion.

Classes are standard, Infrequent Access (IA), Reduced Redundancy Storage (RRS), and Glacier.

S3 supports encryption of objects in flight and at rest.

S3 supports multi-part upload API which supports higher throughput data uploads. This should be used for data > 100MB and must be used for data > 5GB.

Cross-region replication is supported allowing async replication of objects from a bucket in one region to one other region. But this only replicates NEW objects for an existing bucket.

S3 also supports logging, event notification (to SNS, SQS and Lambda functions).

S3 supports caching, and indexing in DynamoDB.

Compared to other similar object stores, Cassandra has "tunable" consistency which may give more flexibility.

S3 doesn't appear to support "locking" (i.e. concurrent updates to the same object - latest update wins).

S3 objects have key, value, version id, metadata (set of key/value pairs), and AC information.

Performance issues? Need to randomise the key using a hash for better performance (this messes up the URL however). and CloudFront for heavy reads (how is the cache refreshed from S3?)

http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html

More on storage classes:

Infrequent Access (IA) has 99.99% availability, 128KB minimum data object size (Why?), 30 days minimum duration (what's the minimum duration for standard? Can you create lots of objects and delete them after a few minutes and only be charged for the time used?) and a fee per GB data read.

Reduced Redundancy Storage (RRS) provides "400 x disk durability" but you can only loose 1 location (i.e. >= 2 locations). NOTE: RRS appears to be "Redundant" now as the cost if > standard. This illustrates one of the architectural choices with AWS which is that price is a big factor and it's beyond your control so becomes a significant architectural "feature". How much would it cost to move from RRS to IA or back to standard at present?

Remember that it's free to move data in and out of the same region. This reduces the cost for applications that are just processing the data all in one region.

My standard question again is how do you architect S3 based applications taking into account Limitations and Pricing?

My suspicion is that there will be trade offs with pricing and in practice it may be necessary to build price/usage models and explore sensitivity to ensure that the solution is affordable and does't result in "bill shock" if something changes even by a small amount.

For example, an analysis of trade-off between standard and IA pricing: https://www.concurrencylabs.com/blog/save-money-using-s3-infrequent-access/

And here is a blog with some "issues" to watch out for: http://www.aws-simplified.com/aws/aws-s3/the-aws-s3-and-glacier-pitfalls/

A youtube video (few years old) on S3 performance.

Finally a quick look at cost. Over the last 10 years I've worked with lots of clients looking at migrating enterprise systems to the cloud. My approach has been to use APM data to build performance and cost models to explore potential issues before they arise. One example from a few years ago was a government application for managing workflows across whole of government focussed on document management. Typically documents are uploaded to the system and then found by other users who edit them and create new documents and the workflow keeps track of versions and all related documents. Workflows and documents can last for months. I've created a simplified cost model of a system based on this application deployed to a single S3 region assuming the number of users doesn't increase but that the number of documents does increase linearly over time. I've looked at 2 alternatives, one uses S3 standard only, the other uses standard + IA (documents are moved to IA after 1 month). Note that the costs are sensitive to a number of factors including read/write ratios, average document size, and number of transactions per second and load growth over time. Graphs show cost ($/month) for 12 months for the 2 options. Main observations are that the price per month naturally increases as the number of objects grows, that standard+ia is slightly cheaper than standard only, management and operations cost is small relative to storage and data transfer (which is highest percentage).

NOTE: What other costs may be incurred? E.g. Caching, security/encryption? Also note that this is storage only as I havn't looked at running the complete workflow/document management system in AWS (maybe next blog).

Postscript - a thought about AWS S3 API complexity and use...

I had a look at the aws documentation for AWS S3 API. For the REST API there is 1 operation for the s3 service itself, 49 operations for buckets, and 21 operations for objects, a total of 71 operations.
At one level the API documentation is really just a list of operations and detailed documentation for each. What would be useful (for me anyway) is something like a couple of UML diagrams (class and state diagrams etc) explaining the relationship between entities, operations, and what's valid/invalid to do depending on the state. I would imagine (and looking at some code examples) that it's relatively complicated to program anything useful for S3 on the client side (and how do you keep track of state and your objects and type and contents etc?).

Another way of looking at this is to examine the documentation for a single language client API for S3. I picked JavaScript at random as it's good for client side code development. Guess how long the documentation is? 100 pages? 10,000 words? 200 pages? 20,000 words? Nope, 238 pages and 40,000 words. About PhD thesis size.

Here it is if you have time read it! http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html

What's the easiest way of interacting with S3? Possibly using other AWS services. E.g. Lambda and Step Functions perhaps? https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-creating-lambda-state-machine.html

According the docs S3 is "deeply integrated" with 14+ other AWS services:

"Amazon S3 is deeply integrated with other AWS services to make it easier to build solutions that use a range of AWS services. Integrations include AWS Storage Gateway,Amazon CloudFront, Amazon CloudWatch, Amazon Kinesis, Amazon RDS, Amazon Glacier, Amazon EBS, Amazon DynamoDB, Amazon Redshift, Amazon Route 53, Amazon EMR, Amazon VPC, Amazon KMS, and AWS Lambda."

PPS
What's the difference between a NoSQL database and AWS S3? DynamoDB is the AWS NoSQL database. Here's a blog which explains the differences and how they turned S3 into a NoSQL database: http://www.s3nosql.com.s3.amazonaws.com/infinitedata.html

So here's a weird thing, AWS S3 also actually has SQL with the additon of AWS Athena (sort of, reads only): https://aws.amazon.com/blogs/aws/amazon-athena-interactive-sql-queries-for-data-in-amazon-s3/
Here's a bit more about how it works: https://aws.amazon.com/athena/faqs/
It appears to use Presto (which i thought only (?) worked on Hadoop? https://aws.amazon.com/emr/details/presto/ )
Presto was developed at Facebook and was also used by Netflix: https://en.wikipedia.org/wiki/Presto_(SQL_query_engine)
It was designed for large scale queries over relational data but not to support full relational database functionality.

What are some use cases? How does it compare to Redshift? https://www.alooma.com/blog/amazon-athena
A big data example: https://aws.amazon.com/blogs/big-data/analyzing-data-in-s3-using-amazon-athena/
And another analysis: http://blog.octo.com/en/i-have-tested-amazon-athena-and-have-gone-ballistic/
And a twitter use case:
http://northbaysolutions.com/blog/athena-beyond-the-basics-part-1/
http://northbaysolutions.com/blog/athena-beyond-the-basics-part-2/

I also wondered about which java persistence frameworks are supported for AWS databases and what an example looks like. Here's one nice example from a Java perspective (using EC2, S3 and RDS):
http://briansjavablog.blogspot.com.au/2016/05/spring-boot-angular-amazon-web-services.html

And yet another product (Heroku, a cloud PaaS) which can interoperate with S3:
https://devcenter.heroku.com/articles/using-amazon-s3-for-file-uploads-with-java-and-play-2
https://devcenter.heroku.com/articles/s3 (which has a warning about doing large file uploads to S3 from a single threaded client programming language).
How is Heroku similar/different to AWS? Here's a recent comparison: https://dzone.com/articles/heroku-or-amazon-web-services-which-is-best-for-your-startup

PPPS - patterns

I realised that you really can't make much sense of single AWS services or combinations without thinking in terms of architectural patterns. Looks like a few people have thought about this before me, including amazon and the book:

Cloud Computing Patterns
Fundamentals to Design, Build, and Manage Cloud Applications