Backtracking - aws architecture

I found these resources today which are at a higher level and probably provide a more general introduction to aws architecture.  I think I will backtrack and have a look at some. They include:

1
Whitepapers (a lot not sure anyone could read them all in a year)

2
Architecting for the cloud best practices.

This is probably where anyone with some background in s/w architecture, distributed systems, computer science etc should start with AWS :-)

It's a short introduction but uses concepts that I was already familiar with to explain some of the major AWS features/services/distinctions.

Some highlights (that would be good to follow up on) include (quotes from the above link):

Search

Search Applications that require sophisticated search functionality will typically outgrow the capabilities of relational or NoSQL databases. A search service can be used to index and search both structured and free text format and can support functionality that is not available in other databases, such as customizable result ranking, faceting for filtering, synonyms, stemming, etc. On AWS, you have the choice between Amazon CloudSearch and Amazon Elasticsearch Service (Amazon ES)

Redundancy

In standby redundancy when a resource fails, functionality is recovered on a secondary resource using a process called failover. The failover will typically require some time before it completes, and during that period the resource remains unavailable. The secondary resource can either be launched automatically only when needed (to reduce cost), or it can be already running idle (to accelerate failover and minimize disruption). Standby redundancy is often used for stateful components such as relational databases. In active redundancy, requests are distributed to multiple redundant compute resources, and when one of them fails, the rest can simply absorb a larger share of the workload. Compared to standby redundancy, it can achieve better utilization and affect a smaller population when there is a failure.

Detection Failure (for n-tier systems)

In a typical three-tier application, you configure health checks on the Elastic Load Balancing service. Design your health checks with the objective of reliably assessing the health of the back end nodes. A simple TCP health Amazon Web Services – Architecting for the Cloud: AWS Best Practices February 2016 Page 27 of 42 check would not detect the scenario where the instance itself is healthy, but the web server process has crashed. Instead, you should assess whether the web server can return a HTTP 200 response for some simple request.

At this layer, it might not be a good idea to configure what is called a deep health check, which is a test that depends on other layers of your application to be successful (this could result in false positives). For example, if your health check also assesses whether the instance can connect to a back end database, you risk marking all of your web servers as unhealthy when that database node becomes shortly unavailable. A layered approach is often the best. A deep health check might be appropriate to implement at the Amazon Route53 level. By running a more holistic check that determines if that environment is able to actually provide the required functionality, you can configure Amazon Route53 to failover to a static version of your website until your database is up and running again.

High Availability

For any production relational database, we recommend the use of the Amazon RDS Multi-AZ deployment feature, which creates a synchronously replicated standby instance in a different Availability Zone (AZ). In case of failure of the primary node, Amazon RDS performs an automatic failover to the standby without the need for manual administrative intervention. When a failover is performed, there is a short period during which the primary node is not accessible. Resilient applications can be designed for Graceful Failure by offering reduced functionality (e.g., read-only mode by utilizing read replicas). 

This is a really good (obvious in hindsight) example of how to reason about combinations of Amazon services. I.e. combination of a synchronous standby replica, and asynchronous read replicas to solve a specific problem and improved performance and availability. The thinking behind this is something like: Ok, what happens when one thing fails? (E.g. Master node) How long will it take and what happens? (The standby comes online eventually for both reads and writes). Ok,  is there anything that can be done to mitigate the failure? (e.g. reduce time to recovery, maintain a subset of functionality during the recovery period, etc). Yes, use read replicas while standby comes online to serve read only requests.

Durable data storage

A simple explanation of why some services provide immediate consistency vs. eventually consistency, and how Availability zones vs regions work. The simple story is that AZs have fast low latency networks which allow for synchronous updates to replicas, this gives immediate consistency but limited horizontal scalability.  i.e.

Synchronous replication only acknowledges a transaction after it has been durably stored in both the primary location and its replicas. It is ideal for protecting the integrity of data from the event of a failure of the primary node. Synchronous replication can also scale read capacity for queries that require the most up-to-date data (strong consistency). The drawback of synchronous replication is that the primary node is coupled to the replicas. A transaction can’t be acknowledged before all replicas have performed the write. This can compromise performance and availability (especially in topologies that run across unreliable or high-latency network connections). For the same reason it is not recommended to maintain many synchronous replicas.

Asynchronous replication decouples the primary node from its replicas at the expense of introducing replication lag. This means that changes performed on the primary node are not immediately reflected on its replicas. Asynchronous replicas are used to horizontally scale the system’s read capacity for queries that can tolerate that replication lag. It can also be used to increase data durability when some loss of recent transactions can be tolerated during a failover. For example, you can maintain an asynchronous replica of a database in a separate AWS region as a disaster recovery solution.

Quorum-based replication combines synchronous and asynchronous replication to overcome the challenges of large-scale distributed database systems. Replication to multiple nodes can be managed by defining a minimum number of nodes that must participate in a successful write operation. A detailed discussion of distributed data stores is beyond the scope of this document. You can refer to the Amazon Dynamo whitepaper to learn more about a core set of principles that can result in an ultra-scalable and highly reliable database system.

Quorum based approaches are well in known form distributed computing/database theory, it's a pity the certification course didn't introduce the relevant AWS services using these categories.
Actually I'm still which Amazon services use this approach? TODO

Also wondering when I first came across quorum protocols and in what contexts (possibly overlay networks?)  I suspect the idea if not implementations has been around for a while.

1995: http://web.mit.edu/6.033/2005/wwwdocs/quorum_note.html
Consensus protocols are related: http://www.cs.cornell.edu/ken/book/New%20514%20slide%20set/16-Quorums.ppt
http://www.inf.ufpr.br/aldri/disc/slides/SD712_lect12.pdf
https://www.cs.duke.edu/courses/fall07/cps212/consensus.pdf

But there's some potentially nasty problems related to types of failures (if nodes just go away, but also if they start misbehaving in potentially evil ways).

Two-phase commit is an example of a consensus protocol (I think).

This appears to be the fist reference to a consensus protocol (1979).
Robert H. Thomas. 1979. A Majority consensus approach to concurrency control for multiple copy databases. ACM Trans. Database Syst. 4, 2 (June 1979), 180-209. DOI=http://dx.doi.org/10.1145/320071.320076

A “majority consensus” algorithm which represents a new solution to the update synchronization problem for multiple copy databases is presented. The algorithm embodies distributed control and can function effectively in the presence of communication and database site outages. The correctness of the algorithm is demonstrated and the cost of using it is analyzed. Several examples that illustrate aspects of the algorithm operation are included in the Appendix.

And the 1st reference I could find to quorum protocols (also 1979):

David K. Gifford. 1979. Weighted voting for replicated data. In Proceedings of the seventh ACM symposium on Operating systems principles (SOSP '79). ACM, New York, NY, USA, 150-162. DOI=http://dx.doi.org/10.1145/800215.806583

In a new algorithm for maintaining replicated data, every copy of a replicated file is assigned some number of votes. Every transaction collects a read quorum of rvotes to read a file, and a write quorum of wvotes to write a file, such that r+w is greater than the total number of votes assigned to the file. This ensures that there is a non-null intersection between every read quorum and every write quorum. Version numbers make it possible to determine which copies are current. The reliability and performance characteristics of a replicated file can be controlled by appropriately choosing r, w, and the file's voting configuration. The algorithm guarantees serial consistency, admits temporary copies in a natural way by the introduction of copies with no votes, and has been implemented in the context of an application system called Violet.

I recall coming across these ideas before (but can't recall where, probably in OS lectures, and distributed systems conferences, and eventually in 2 phase commit. The concepts have certainly been around for a while.

A few references to consensus/quorums in AWS:

file:///C:/Users/brebs/Downloads/ConcensusAlgorithms.pdf

Every computer scientist trying to learn AWS should read this presentation as it actually explains the history behind the AWS computing theory and practice in terms of traditional distributed systems theory :-)

Under the Covers of AWS: Core Distributed Systems Primitives That Power Our Platform | AWS re:Invent 2014

1. 1. November 13, 2014 | Las Vegas, NV Al Vermeulen and Swami Sivasubramanian

Paxos and Gossip protocols used in AWS: https://www.slideshare.net/AmazonWebServices/spot302-under-the-covers-of-aws-core-distributed-systems-primitives-that-power-our-platform-aws-reinvent-2014

Even better, the video: https://www.youtube.com/watch?v=QVvFVwyElLY&feature=youtu.be 

Whereas, I hadn't heard ot shuffle sharding until reading this, fascinating!

Shuffle Sharding

One fault-isolating improvement you can make to traditional horizontal scaling is called sharding. Similar to the technique traditionally used with data storage systems, instead of spreading traffic from all customers across every node, you can group the instances into shards. For example, if you have eight instances for your service, you might create four shards of two instances each (two instances for some redundancy within each shard) and distribute each customer to a specific shard. In this way, you are able to reduce the impact on customers in direct proportion to the number of shards you have. However, there will still be affected customers, so the key is to make the client fault tolerant. If the client can try every endpoint in a set of sharded resources, until one succeeds, you get a dramatic improvement. This technique is called shuffle sharding and is described in more detail in the relevant blog post.

Spot blocks

https://aws.amazon.com/blogs/aws/new-ec2-spot-blocks-for-defined-duration-workloads/

You can now request Amazon EC2 Spot instances to run continuously, for up to six hours, at a flat rate that saves you up to 50% compared to On-Demand prices. This enables you to reduce costs when running finite duration tasks such as batch processing, encoding and rendering, modeling and analysis, and continuous integration jobs.

To get started, specify the duration you want your instance(s) to run – between one and six hours – when placing a Spot instances request. When Spot instance capacity is available for the requested duration, your instances will launch and run for that duration for a flat hourly price. Once the time block ends, your instance will be terminated automatically.

And finally how to pick NoSQL vs. Relational Database:

If your application primarily indexes and queries data with no need for joins or complex transactions (especially if you expect a write throughput beyond the constraints of a single instance) consider a NoSQL database instead. If you have large binary files (audio, video, and image), it will be more efficient to store the actual files in the Amazon Simple Storage Service (Amazon S3) and only hold the metadata for the files in your database. 

3
And the online course for well architected:

https://aws.amazon.com/architecture/well-architected/

Or maybe not, looks like you need to be a member of a secret society or something?

This training item is currently unavailable to you. Please select alternative training, or complete prior requirements before requesting this training.