AWS EBS revision: Watch out for the Mebibytes, Gibibytes and Tebibytes (or not)!!!

I keep getting caught out with AWS EBS questions.  One of the main things to remember is that they have different IOPS and throughput (MBps) rates. Docs are here.

The IOPS are in order from max of 20,000 IOPS for Provisioned SSD, 10,000 for general purpose SSD, 500 for Throughput optimised HDD, down to 250 for Cold HDD.

BUT!  MB/s per order is different. Order from max to min MB/s is:

Throughput optimised HDD: 500MB/s
Provisioned IOPS SSD: 320MB/s
Cold HDD: 250MB/s
General purpose SSD: 160MB/s

The other to remember is that Snapshots don't result in the volume being snapshotted being unavailable, and that data for new snapshots is available immediately (but lazily loaded).

AWS also recommends EBS-Optimized instances for improved latency and higher MB/s.

However, the max of 10,000 Mbps is actually only 1,250 MB/s. The table (below) says that 1,750 MB/s is the max throughput per instance. Which is actually correct???  Looks like some instance types do provide more than 10,000 Mbps. E.g f1.16xlarge has 14,000 Mbps (1,750 MB/s). Also a few more. See this page.

f1.16xlarge

Yes

14,000

1,750

75,000

Also note that some instances with 10-gigabit network interfaces do NOT OFFER EBS-optimization (shared b/w):

Note that some instances with 10-gigabit network interfaces, such as i2.8xlarge and r3.8xlarge do not offer EBS-optimization, and therefore do not have dedicated EBS bandwidth available and are not listed here. On these instances, network traffic and Amazon EBS traffic is shared on the same 10-gigabit network interface. 

Some other 10-gigabit network instances, such as c4.8xlarge and d2.8xlarge offer dedicated EBS bandwidth in addition to a 10-gigabit interface which is used exclusively for network traffic.

Also note that some instances are EBS-optimized by default.

Amazon EBS-Optimized Instances

For an additional low, hourly fee, customers can launch certain Amazon EC2 instance types as EBS-optimized instances. EBS-optimized instances enable EC2 instances to fully use the IOPS provisioned on an EBS volume.

EBS-optimized instances deliver dedicated throughput between Amazon EC2 and Amazon EBS, with options between 500 and 10,000 Megabits per second (Mbps) depending on the instance type used. The dedicated throughput minimizes contention between Amazon EBS I/O and other traffic from your EC2 instance, providing the best performance for your EBS volumes.

EBS-optimized instances are designed for use with all Amazon EBS volume types.

For more information about the instance types that can be launched as EBS-Optimized instances, see Amazon EC2 Instance Types.

Amazon EBS Volume Types

The following table shows use cases and performance characteristics of current generation EBS volumes:

	Solid State Drives (SSD)		Hard Disk Drives (HDD)
Volume Type	EBS Provisioned IOPS SSD (io1)	EBS General Purpose SSD (gp2)*	Throughput Optimized HDD (st1)	Cold HDD (sc1)
Short Description	Highest performance SSD volume designed for latency-sensitive transactional workloads	General Purpose SSD volume that balances price performance for a wide variety of transactional workloads	Low cost HDD volume designed for frequently accessed, throughput intensive workloads	Lowest cost HDD volume designed for less frequently accessed workloads
Use Cases	I/O-intensive NoSQL and relational databases	Boot volumes, low-latency interactive apps, dev & test	Big data, data warehouses, log processing	Colder data requiring fewer scans per day
API Name	io1	gp2	st1	sc1
Volume Size	4 GB - 16 TB	1 GB - 16 TB	500 GB - 16 TB	500 GB - 16 TB
Max IOPS**/Volume	20,000	10,000	500	250
Max Throughput/Volume	320 MB/s	160 MB/s	500 MB/s	250 MB/s
Max IOPS/Instance	75,000	75,000	75,000	75,000
Max Throughput/Instance	1,750 MB/s	1,750 MB/s	1,750 MB/s	1,750 MB/s
Price	$0.125/GB-month $0.065/provisioned IOPS	$0.10/GB-month	$0.045/GB-month	$0.025/GB-month
Dominant Performance Attribute	IOPS	IOPS	MB/s	MB/s

Note that the price drops from left to right. So max throughput (TP optimized HDD, 3rd column) is actually pretty cheap.

However, the other confusing thing is how much volumes can burst. This blog says EBS gp2 can burst to 3,000 IOPS.  But this is less than the max IOPS above (10,000). Why???



You can now monitor the burst-bucket performance and ensure you have the right volume size for Amazon EBS General Purpose SSD (gp2) volumes using Amazon CloudWatch. Amazon EBS gp2 volumes have the ability to burst to 3,000 IOPS for extended periods of time for workloads that need to burst beyond the baseline performance. Amazon EBS now publishes the Burst Balance metric into Amazon CloudWatch for all gp2 volumes. The vast majority of customers have sufficient burst for their workloads. However, if visibility into the Burst Balance is desired, you now have the ability to monitor and take appropriate action. You can automate Burst Balance monitoring by creating a CloudWatch alarm that notifies you when the Burst Balance falls below a user-specified threshold.

Ah, it's actually more complicated than that:

General Purpose SSD (gp2) volumes offer cost-effective storage that is ideal for a broad range of workloads. These volumes deliver single-digit millisecond latencies and the ability to burst to 3,000 IOPS for extended periods of time. Between a minimum of 100 IOPS (at 33.33 GiB and below) and a maximum of 10,000 IOPS (at 3,334 GiB and above), baseline performance scales linearly at 3 IOPS per GiB of volume size. A gp2 volume can range in size from 1 GiB to 16 TiB.

Max burst depends on volume size according to this graph:




Check out the other graphs for different volume types.

AWS also appears to have invented new units. E.g. MiB/s and GiB!!!

What are these? This appears to be an error as:

Mebibyte per second

The mebibyte per second (MiB/s) is a unit of data transfer rate equal to:

• 1,048,576 bytes per second
• 1,024 kibibytes per second
• 8 mebibits per second
• 8192 kibibits per second
• 8,388,608 bits per second

This pages says the max TP in MiB/s is as follows:

Max. Throughput/Volume†

160 MiB/s

320 MiB/s

500 MiB/s

250 MiB/s

Whereas the original page is definitely in MB/s (but different order).

Max Throughput/Volume

320 MB/s

160 MB/s

500 MB/s

250 MB/s

AWS docs appear to be riddled with MiB, GiB, TiB. This makes me suspect it's NOT an error, and that they really mean these units. And the difference to MB/s etc is small.

E.g. 500 MB/s is 500,000,000 B/s, c.f. 500 MiB/s = 524,288,000 B/s

i.e. only  about 5% difference.   BUT the difference increases with size and speed of devices!

See this article on the history and problems of these units.

But they are now ISO so I guess we have to put up with them!

Units of ISO/IEC 80000[edit]

The standard includes all SI units but is not limited to only SI units. Units that form part of the standard but not the SI include the units of information storage (bit and byte), units of entropy (shannon, natural unit of information and hartley), the erlang (a unit of traffic intensity) and units of level (neper and decibel). The standard includes all SI prefixes as well as the binary prefixes kibi-, mebi-, gibi-, etc., originally introduced by the International Electrotechnical Commission to standardise binary multiples of byte such as mebibyte (MiB), for 1024² bytes, to distinguish them from their decimal counterparts such as megabyte (MB), for precisely one million (1000²) bytes. In the standard, the application of the binary prefixes is not limited to units of information storage. 

And now I see why I'm even more confused than I thought as I've been around for a while and once upon a time a MB really did mean a MiB!!!

Which concludes with this helpful thought:

This is your brain, this is your brain on Kibibytes -- fried yet? (yep)