SD card and SSD's have a latency on the order of 500us, while even a Raspberry Pi 3 has a memory latency[1] of around 200ns. That's three orders of magnitude difference in latency, which would destroy performance.
In addition, memory bandwidth is still at least an order of magnitude higher.
Old computers were vastly weaker in a lot of ways, but memory latency isn't one of those things. A 6502 was also hitting memory every few hundred nanoseconds.
You could make it work with Optane except oops that just got discontinued.
SSD speeds will probably continue increasing, so removing RAM will take us about 20 years back to the past, with the difference that these will have seemingly "unlimited" memory. Imagine Windows 2000 never complaining about being out of memory.
> SSD speeds will probably continue increasing, so removing RAM will take us about 20 years back to the past
I think you significantly overestimate how much they can reduce latency for SSDs. Keep in mind, even a 80286 from 1982 had a memory latency in the 200ns range[1].
Getting rid of DRAM would take us back a lot farther than 20 years. A typical desktop PC circa 2000 using PC133 RAM would have had a memory access latency in the tens of nanoseconds.
In addition, memory bandwidth is still at least an order of magnitude higher.
[1]: https://browser.geekbench.com/v4/cpu/11296737