The "ephemeral" term is a legacy. Unfortunately it is part of the EC2 API for the block device mapping [1] of the "classic" instance store interfaces on the Xen platform. I don't know exactly when we stopped using "ephemeral" in our documentation, but I think it was with the introduction of EBS around 2008.
The "ephemeral" term confuses a lot of customers, and that's why we stopped using it. Data written to local storage is not transient, fleeting, or short lived. By 2010 we had transitioned to using "instance storage" in the documentation [2], which included a big note about how the data remains if an instance reboots for any reason (planned or unplanned).
Still, there is a misconception that data on local instance store volumes (both the more "classic" HDD or SSD volumes that are virtualized by Xen, as well as the new generation of local NVMe storage) could vanish due to this vestigial term that lingers in the API. Many customers, as well as services like Amazon Aurora [3], build highly durable and available systems on local instance storage.
Post author here. It's ephemeral, yes. It survives reboots, so that's not a problem. It doesn't survive instance-stop, so if a machine is being decommissioned by AWS we do indeed lose its data. As for how we protect against it, the main thing is replication: the data is stored on more than one machine. If we lose a machine for whatever reason, the shards from that machine are copied from a replica to another DB instance.
As local NVMe storage does not have any interaction with the "classic" block device mapping APIs (the storage shows up as a PCI device, the same way that a GPU or FPGA does, and it doesn't matter in any way how the block device mapping is set up), there is no reason to use "ephemeral" to describe it.
Said more directly: no, it is not ephemeral. It is local storage that is tied to the life cycle of the instance.
Reboots are actually fine as ephemeral data will persist through a reboot on an EC2 instance. Your question is still valid though in case of halting and how you deal with it is specific to your application, but you have to be able to handle all the data on that ephemeral disk disappearing without warning.
One way in the case of a database could be a second EC2 instance configured as a read replica in a different AZ.
Back when we ran the Citus cluster on EBS, we lost some EBS volumes as well. This manifested as disk not responding, followed several days later by an email from AWS with the subject Your Amazon EBS Volume vol-123456789abcdef telling you the disk was lost irrecoverably.
But yeah, you need to be ready for your disks to go away no matter where they are: ephemeral, EBS, physical, whatever.
