Report on Optane and IMDT

Last update: 04/07/2018

Info on Optane SSD DC P4800X

Intel® Optane™ SSD DC P4800X Series

375 GB Capacity
HHHL (CEM3.0) Form Factor
PCIe NVMe 3.0 x4 Interface

Product brief: https://www.intel.com/content/www/us/en/solid-state-drives/optane-ssd-dc-p4800x-brief.html

Technical specifications from the vendor's website: https://www.intel.com/content/www/us/en/products/memory-storage/solid-state-drives/data-center-ssds/optane-dc-p4800x-series/p4800x-375gb-aic-20nm.html

Host systems

The nodes it-proc-ssd-testbench-01 and olsky03 were used for benchmarking the drives.


- Intel Optane(TM) SSD DC P4800X Series PHKS73350091375AGN -

Bootloader : EB3B0306
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : E2010324
FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.
Index : 0
ModelNumber : INTEL SSDPED1K375GA
ProductFamily : Intel Optane(TM) SSD DC P4800X Series
SerialNumber : PHKS73350091375AGN

Full log from `isdct` (Click to expand)
- Intel Optane(TM) SSD DC P4800X Series PHKS73350091375AGN -

AggregationThreshold : 0
AggregationTime : 0
ArbitrationBurst : 0
Bootloader : EB3B0306
CoalescingDisable : 1
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
DirectivesSupported : False
DynamicMMIOEnabled : The selected drive does not support this feature.
EndToEndDataProtCapabilities : 17
EnduranceAnalyzer : Workload did not induce any wear on the drive. Expected life calculation is invalid. Ensure that workloads run for an hour or more for valid calculations.
ErrorString : 
Firmware : E2010324
FirmwareUpdateAvailable : The selected Intel SSD contains current firmware as of this tool release.
HighPriorityWeightArbitration : 0
IOCompletionQueuesRequested : 30
IOSubmissionQueuesRequested : 30
Index : 0
Intel : True
IntelGen3SATA : False
IntelNVMe : True
InterruptVector : 0
IsDualPort : False
LBAFormat : 0
LatencyTrackingEnabled : The selected drive does not support this feature.
LowPriorityWeightArbitration : 0
MaximumLBA : 732585167
MediumPriorityWeightArbitration : 0
MetadataSetting : 0
MetadataSize : 0
ModelNumber : INTEL SSDPED1K375GA
NVMeControllerID : 0
NVMeMajorVersion : 1
NVMeMinorVersion : 0
NVMePowerState : 0
NVMeTertiaryVersion : 0
NamespaceId : 1
NamespaceManagementSupported : False
NativeMaxLBA : 732585167
NumErrorLogPageEntries : 63
NumLBAFormats : 6
NumberOfNamespacesSupported : 1
OEM : Generic
PCILinkGenSpeed : 3
PCILinkWidth : 4
PLITestTimeInterval : The selected drive does not support this feature.
PhySpeed : The selected drive does not support this feature.
PhysicalSectorSize : The selected drive does not support this feature.
PhysicalSize : 375083606016
PowerGovernorAveragePower : The desired feature is not supported.
PowerGovernorBurstPower : The desired feature is not supported.
PowerGovernorMode : 0
Product : Coldstream
ProductFamily : Intel Optane(TM) SSD DC P4800X Series
ProductProtocol : NVME
ProtectionInformation : 0
ProtectionInformationLocation : 0
ReadErrorRecoveryTimer : Device does not support this command set.
SMARTEnabled : True
SMARTHealthCriticalWarningsConfiguration : 0

    

Known issues

With no clear pattern, we were able to observe that the drive sometimes enters in a frozen state after two consecutive low-level format commands launched through the isdct (Intel SSD) tool. Intel is helping us investigating the issue.

Optane benchmarks

Sustained Multi-Threaded Random 4KB Mixed (30% write / 70% read) using 100% Capacity
  • Average IOPS: 513053
  • Coefficient of variation (StdDev): 0.2%

Sustained Multi-Threaded Random Read Tests by Block Size

Sustained 4KB Random Mixed 70% read / 30% write performance by number of threads

Sustained 4KB Random Read Latencies by # of Threads

Sustained 4KB Random Write Latencies by # of Threads

IMDT tests

Intel® Memory Drive Technology (IMDT) is a software that extends system memory transparently. Combined with an Intel® Optane™ Solid State Drive (SSD), Intel® Memory Drive Technology transparently integrates the SSD into the memory subsystem and makes it appear like DRAM to the OS and applications. Intel® Memory Drive Technology increases memory capacity beyond DRAM limitations and delivers DRAM-like performance in a completely transparent manner to the operating system and application. In addition, no changes are required to the OS or applications.

Enabling IMDT on olsky03.cern.ch, we were able to see ~500GB of DRAM from the operating system, compared to the 64GB physically installed. IMDT does not allow to use the full capacity of the drives as system memory, since some of the capacity is kept internally by the software to implement pre-fetching and caching algorithms.

GeneROOT test

In order to test the IMDT capabilities, we performed a test using code from the GeneROOT project, a collaboration aiming at adopting high-energy physics data analysis tools on large genomics datasets. In particular, we ran a software with the goal of realigning genomic files in another format, a task which is very memory and storage consuming.

We compared the execution on olsky03.cern.ch with IMDT enabled with a normal execution on a quad-socket Haswell server with 512GB of DRAM. All the files were read from local SSD storage.

The times reported below concern the realignment of a 75GB genomic file. Two runs were performed, limiting the amount of memory consumption to 200GB, 400GB, as well as limiting the number of threads:

Memory limit (GB) # threads Wall time
IMDT 200 8 17h 39m 26s
DRAM 200 8 15h 03m 10s
IMDT 400 16 9h 26m 55s
DRAM 400 16 8h 31m 05s

Although the DRAM still performs better (as expected) for this usecase, is very interesting to notice how the execution times are comparable. This is relevant, considering that the price/GB for the Optane drives might be much lower in the near future than the price/GB of a classic DDR4 memory DIMM.

Also, no optimizations were applied for the IMDT tests, leading to the possibility of a performance improvement.

Other material:

Interesting presentation by Intel/Moscow State University: https://software.intel.com/sites/default/files/managed/ca/71/132980_hpcdevcon2017_benefits-of-intel-memory-drive-technology-for-scientific-applications.pdf

Luca Canali (IT/DB): still waiting for feedback to openlab, but he already discussed with Intel directly.

Fabrice Le Goff (ATLAS, CERN) Slides about Optane tests: https://docs.google.com/presentation/d/1uwKi5AAU380HWrYZxfJ9zmQAzP7wwU864AZ9Modrp6I/edit?usp=sharing