Evaluating solid-state drives (SSDs) for use in enterprise applications can be tricky business. In addition to selecting the right interface type, endurance level and capacity, decision makers must also look beyond product data sheets to determine the right SSD to accelerate their applications. Often the specifications on SSD vendor collateral are based on multiple and different benchmark tests, or other criteria that may not represent one’s unique environment. This paper will examine basic SSD differences and highlight key criteria that should be considered in choosing the right device for a given workload/application.
The industry typically classifies enterprise-class SSDs by interface type. The main interfaces to evaluate are SATA, SAS and PCIe. From here, it is easy to qualify the devices based on factors such as price, performance, capacity, endurance and form factor. SATA is usually the least expensive of the device types, but also brings up the rear in terms of performance due to the speed limitations of the 6Gb/s SATA bus. SAS SSDs are mostly deployed inside SAN or NAS storage arrays. With dual-port interfaces, they can be configured with multiple paths to multiple array controllers for high availability. SAS drives deliver nearly double the performance of SATA devices, thanks to the 12Gb/s SAS interface. SATA and SAS SSDs are the most widely deployed interface types today, with capacities that can reach more than 4TB.
At the high-end of the performance spectrum are PCIe SSDs. These devices connect directly into the PCIe bus and are able to deliver much higher speeds. By implementing specialized controllers that closely resemble memory architectures, PCIe products eliminate traditional storage protocol overhead, thereby reducing latencies and access times when compared to SATA or SAS. Given the importance of latency in enterprise applications, PCIe is often preferred as it reduces IO wait times, improves CPU utilization and enables more users or threads per SSD. SATA and SAS SSDs typically have higher latency than PCIe SSDs. This is primarily due to the software stack that must be traversed to read and write data. The diagram in Figure 1 above illustrates the layers of this stack.
More on PCIe
A PCIe connection consists of one or more data transmission lanes, connected serially. Each lane consists of two pairs of wires, one for receiving and one for transmitting. The PCIe specification supports one, four, eight or sixteen lanes in a single PCIe slot typically denoted as x1, x4, x8 or x16. Each lane is an independent connection between the PCI controller and the SSD, and bandwidth scales linearly, so an x8 connection will have twice the bandwidth of a x4 connection. PCIe transfer rates will depend on the generation of the base specification. Base 2.0 or Gen-2 provides 4Gb/s per lane, so a Gen-2 with 16 lanes can deliver an aggregate bandwidth of 64Gb/s. Gen- 3 provides 8Gb/s per lane, so accordingly a Gen-3 with 8 lanes provides 64Gb/s.
When evaluating PCIe devices, it is important to look for references to generation and number of lanes (Gen -2 x4 or Gen-3 x8 and so on). While PCIe devices tend to be the most expensive per GB, they can deliver more than 10 times the performance of SATA. Table 1 on the following page shows a quick summary of the distinctions between the three SSD interfaces.
Why NVMe Matters for PCIe
Another recent PCIe development is a standards-based device driver called NVM Express or NVMe. Most operating systems are shipping the standard NVMe driver today, which eliminates the hassles of deploying proprietary drivers from multiple vendors. Compared to the SATA interface, NVMe is designed to work with pipeline-rich, random access, memory-based storage. As such, it requires only a single message for 4K transfers (compared to 2 for SATA) and has the ability to process multiple queues instead of only one. In fact, NVMe allows for up to 65,536 simultaneous queues. Highlighted in Table 2 to the right are more details comparing SATA (which follows the Advanced Host Controller Interface standard) to NVMe-compliant PCIe.
The NVM Express Work Group has not stopped at just a driver. Coming soon are standards for monitoring and managing multi-vendor PCIe installations under a single pane of glass, as well as common methods for creating low-latency networks between NVMe devices for clustering and high availability (NVMe-over-Fabrics).
|Specification||SATA SSD||SAS SSD||PCIe SSD|
|IOPS||75k /||130k /||740k /|
|Sequential||500MB /||1.1GB /||3GB /|
Table 1. Typical Specifications for SATA, SAS and PCIe SSDs
|Maximum Queue||One command||65,536 queues;|
|Depth||per queue; 32||65,536|
|commands per||commands per|
|Uncacheable||Six per non-queued||Two per|
|Register Accesses||command; Nine per||command|
|(2000 cycles||queued command|
|Message Signal||A single interrupt;||2048 MSI-X|
|Interrupts (MSI-X)||No steering||interrupts|
|Parallelism and||Requires||No locking|
|Multiple Threads||synchronization lock|
|to issue a command|
Table 2. Features Comparison of SATA/AHCI vs. PCIe/NVMe