https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...
https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-...
In an era of RAM shortages and quarterly price increases, Optane remains viable for swap and CPU/GPU cache.
Isn't that actually crazy good, even insane value for the performance and DWPD you get with Optane, especially with DRAM being ~$15/GB or so? I don't think ~$1/GB NAND is anywhere that good on durability, even if the raw performance is quite possibly higher.
They suck for large sequential file access, but incredible for small random access: databases.
Looking at those charts, besides the DWPD it feels like normal NVMe has mostly caught up. I occassionally wonder where a gen 7/8(?) optane would be today if it caught on, it'd probably be nuts.
This showed up as amazing numbers on a 50%-read, 50%-write mix. Which, guess what, a lot of real workloads have, but benchmarks don't often cover well. This is why it's a great OS boot drive: there's so much cruddy logging going on (writes) at the same time as reads to actually load the OS. So Optane was king there.
So what you mean is that on the most important metric of them all for many workloads, Flash-based NVMe has not caught up at all. When you run a write heavy workload on storage with a limited DWPD (including heavy swapping from RAM) higher performance actually hurts your durability.
Late last year I switched from a 1.5tb Optane 905P to a 4tb WD Blue SN5000 NVMe drive in a gaming machine and saw improved load times, which makes sense given the read and write speeds are ~double. No observable difference otherwise.
I'm sure that's not the use case you were looking for. I could probably tease out the difference in latency with benchmarks but that's not how I use the computer.
The 905P is now in service as an SSD cache for a large media server and that came with a big performance boost but the baseline I'm comparing to is just spinning drives.
https://pcpartpicker.com/forums/topic/425127-benchmarking-op...
You can compare their benchmarks with the other almost 400 SSDs we've benchmarked. Most impressive is that three years later they are still the top random read QD1 performers, with no traditional flash SSD coming anywhere close:
https://pcpartpicker.com/products/internal-hard-drive/benchm...
They are amazing for how consistent and boring their performance is. Bit level access means no need for TRIM or garbage collection, performance doesn't degrade over time, latency is great, and random IO is not problematic.
It's so incredibly fast and responsive that the LuCI interface completely loads the moment I hit enter on the login form.
It seems like there's a very small window, commercially, for new persistent memories. Flash throughput scales really cost-efficiently, and a lot is already built around dealing with the tens-of-microseconds latencies (or worse--networked block storage!). Read latencies you can cache your way out of, and writers can either accept commit latency or play it a little fast and loose (count a replicated write as safe enough or...just not be safe). You have to improve on Flash by enough to make it worth the leap while remaining cheaper than other approaches to the same problem, and you have to be confident enough in pulling it off to invest a ton up front. Not easy!
For databases, where you do lots of small scattered writes, and lots of small overwrites to the tail of the log, modern SSDs coalesce writes in that buffer, greatly reducing write wear, and allowing the effective write bandwidth to exceed the media write bandwidth.
These schemes are much less expensive than optane.
That was never going to work out. Adding an entirely new kind of memory to your storage stack was never going to be easier or cheaper than adding a few large capacitors to the drive so it could save the contents of the DRAM that the SSD still needed whether or not there was Optane in the picture.
Flash is no bueno for write-heavy workloads, and the random-access R/W performance is meh compared to Optane. MLC and SLC have better durability and performance, but still very mid.
And if no shrink was possible, is that because it was (a) possible but too hard; (b) known blocks to a die shrink; or (c) execs didn't want to pay to find out?
The SSD form factor wasn’t any faster at writes than NAND + capacitor-backed power loss protection. The read path was faster, but only in time to first byte. NAND had comparable / better throughput. I forget where the cutoff was, but I think it was less than 4-16KB, which are typical database read sizes.
So, the DIMMs were unprogrammable, and the SSDs had a “sometimes faster, but it depends” performance story.
The SSDs were never going to be dominant at straight read or write workloads, but they were absolutely king of the hill at mixed workloads because, as you note, time to first byte was so low that they switched between read and write faster than anything short of DRAM. This was really, really useful for a lot of workloads, but benchmarkers rarely bothered to look at this corner... despite it being, say, the exact workload of an OS boot drive.
For years there was nothing that could touch them in that corner (OS drive, swap drive, etc) and to this day it's unclear if the best modern drives still can or can't compete.
The read path is sort of a wash, but writes are still unequalled. NAND writes feel like you're mailing a letter to the floating gate...
Worth noting though that Optane is also power-hungry for writes compared to NAND. Even when it was current, people noticed this. It's a blocker for many otherwise-plausible use cases, especially re: modern large-scale AI where power is a key consideration.
You're looking at the entirely wrong kind of shrinking. Hard drives are still (gradually) improving storage density: the physical size of a byte on a platter does go down over time.
Optane's memory cells had little or no room for shrinking, and Optane lacked 3D NAND's ability to add more layers with only a small cost increase.
https://goughlui.com/2024/07/28/tech-flashback-intel-optane-...
1. “Optane” in DIMM form factor. This targeted (I think) two markets. First, use as slower but cheaper and higher density volatile RAM. There was actual demand — various caching workloads, for example, wanted hundreds of GB or even multiple TB in one server, and Optane was a route to get there. But the machines and DIMMs never really became available. Then there was the idea of using Optane DIMMs as persistent storage. This was always tricky because the DDR interface wasn’t meant for this, and Intel also seems to have a lot of legacy tech in the way (their caching system and memory controller) and, for whatever reason, they seem to be barely capable of improving their own technology. They had multiple serious false starts in the space (a power-supply-early-warning scheme using NMI or MCE to idle the system, a horrible platform-specific register to poke to ask the memory controller to kindly flush itself, and the stillborn PCOMMIT instruction).
2. Very nice NVMe devices. I think this was more of a failure of marketing. If they had marketed a line of SSDs that, coupled with an appropriate filesystem, could give 99% fsync latency of 5 microseconds and they had marketed this, I bet people would have paid. But they did nothing of the sort — instead they just threw around the term “Optane” inconsistently.
These days one could build a PCM-backed CXL-connected memory mapped drive, and the performance might be awesome. Heck, I bet it wouldn’t be too hard to get a GPU to stream weights directly off such a device at NVLink-like speeds. Maybe Intel should try it.
Which “Optane memory”? The NVMe product always worked on non-Intel. The NVDIMM products that I played with only ever worked on a very small set of rather specialized Intel platforms. I bet AMD could have supported them about as easily as Intel, and Intel barely ever managed to support them.
Yes, the pure-Optane consumer "Optane memory" products were at a hardware level just small, fast NVMe drives that could be use anywhere, but they were never marketed that way.
That uncertainty couldn't have done the market any favors.
Also… were those weird hybrid SSDs even implemented by actual hardware, or were they part of the giant series of massive kludges in the “Rapid Storage” family where some secret sauce in the PCIe host lied to the OS about what was actually connected so an Intel driver could replace the OS’s native storage driver (NVMe, AHCI, or perhaps something worse depending on generation) to implement all the actual logic in software?
It didn’t help Intel that some major storage companies started selling very, very nice flash SSDs in the mean time.
They were definitely part of the series of massive kludges. But aside from the Intel platforms they were marketed for, I never found a PCIe host that could see both of the NVMe devices on the drive. Some hosts would bring up the x2 link to the Optane half of the drive, some hosts would bring up the x2 link to the QLC half of the drive, but I couldn't find any way to get both links active even when the drive was connected downstream of a PCIe switch that definitely had hardware support for bifurcation down to x2 links. I suspect that with appropriate firmware hacking on the host side, it may have been possible to get those drives fully operational on a non-Intel host.
It isn't weird at all. I would be surprised if it ever succeed in the first place.
Cost was way too high. Intel not sharing the tech with others other than Micron. Micron wasn't committed to it either, and since unused capacity at the Fab was paid by Intel regardless they dont care. No long term solution or strategy to bring cost down. Neither Intel or Micron have a vision on this. No one wanted another Intel only tech lock in. And despite the high price, it barely made any profits per unit compared to NAND and DRAM which was at the time making historic high profits. Once the NAND and DRAM cycle went down again cost / performance on Optane wasn't as attractive. Samsung even made some form of SLC NAND that performs similar to Optane but cheaper, and even they end up stopped developing for it due to lack of interest.
There was certainly a time when it seemed they were shopping for engineers opinions of what to do with it, but I think they quickly determined it would be a much smaller market anyway from ssds and didn’t end up pushing on it too hard. I could be wrong though, it’s a big company and my corner was manufacturing and not product development.
There were/are often projects that come down from management that nobody thinks are worth pursuing. When i say nobody, it might not just be engineers but even say 1 or 2 people in management who just do a shit roll out. There are a lot of layers of Intel and if even one layer in the Intel Sandwich drag their feet it can kill an entire project. I saw it happen a few times in my time there. That one specific node that intel dropped the ball on kind of came back to 2-3 people in one specific department, as an example.
Optane was a minute before I got there, but having been excited about it at the time and somewhat following it, that's the vibe I get from Optane. It had a lot of potential but someone screwed it up and it killed the momentum.
Of course it works exceptionally well when the instinct turns out to be right. But can end companies if it isn’t.
For a lot of bulk storage, yes, you don't have frequently changing data. But for databases or caches, that are under heavy load, optane was not only far faster, but if looking at life-cycle costs, way way less.
The niche that could actually make use of Optane's endurance was small and shrinking, and Intel had no roadmap to significantly improve Optane's $/GB which was unquestionably the technology's biggest weakness.
Intel's got an amazing record of axing projects as soon as they've done the hard work of building an ecosystem.
The newest fully E-core based Xeon CPUs have reached that figure by now, at least in dual-socket configs.
Power failure can happen in between any of "1 byte updates with crazy latencies." However small latency is, power failure is still faster. Usually, there is a write ahead or some other log that alleviates the problem, this log is usually written in streaming fashion.
What is good, though, is that "blast radius" [1] of failure is smaller than usual - failed one byte write rarely corrupts more that one byte or cache line. SQLite has to deal with 512 (and even more) bytes long possible corruptions on most disks, with Optane it is not necessarily so. So, less data to copy, scan, etc.
[1] https://sqlite.org/psow.html
"Just make it a faster SSD" was never a business. The DIMMs were weird, sure, but the bigger issue was that Optane made the most sense when software treated storage and memory as one tier, and almost nobody was going to rewrite kernels, DBs, and apps for a product that cost more than flash and solved pain most buyers barely felt.
If CXL was around at the time it would have been such a nice fit, allowing for much lower latency access.
It also seems like in spite of the bad fit, there were enough regular options drives, and they were indeed pretty incredible. Good endurance, reasonable price (and cheap as dirt if you consider that endurance/lifecycle cost!), some just fantastic performance figures. My conclusion is that alas there just aren't many people in the world who are serious about storage performance.
I believe Optane retained a performance advantage (and I think even today it's still faster than the best SSDs) but SSDs remain good enough and fast enough while being a lot cheaper.
The ideal usage of optane was as a ZIL in ZFS.
And their whole deal was making RAM persistent anyway, which isn't exactly what I want.
It was also the best boot drive money could buy. Still is, I think, though other comments in the thread ask how it compares against today's best, which I'd also love to see.
If they had been cheaper, I think they'd have been really, really popular.
There are very few applications that benefit from such low latency, and if one has to go off the standard path of easy, but slow and expensive and automatically backup up, people will pick the ease.
Having the best technology performance is not enough to have product market fit. The execution required from the side of executives at Intel is far far beyond their capability. They developed a platform and wanted others to do the work of building all the applications. Without that starting killer app, there's not enough adoption to build an ecosystem.
Basically any RDBMS? MySQL and Postgres both benefit from high performance storage, but too many customers have moved into the cloud where you can’t get NVMe-like performance for durable storage for anything remotely close to a worthwhile price.