Uh I don't think vms or containers are cluster management in itself. They are just technologies. What you use to orchestrate them is entirely different, and yes that has changed over the years many times.
For example? Most early cluster management was found in products doing general fleet provisioning, HPC/HTC in academia, proprietary commercials offerings like relational databases, or proprietary in-house solutions like Google's Borg.
I'd describe cluster management as four things - provisioning + configuration, resource management (CPU+RAM+Disk), scheduling (when & where to put things based on availability/resources), and deployment (running things). At least, these are the things I'm concerned about when managing a product that requires a cluster (besides monitoring).
Early cluster management tools were often just doing provisioning and configuration (CFEngine). We see Puppet, Chef, and eventually Ansible further refine the solution as we enter the "Bronze Era" where standing up new servers take a fraction of the time. Now we didn't even bother to name each server by hand when we were going through the installation process after booting up the OS - servers had become cattle, and they were tagged appropriately.
Around the same time (2003-2006) we see virtual machines begin to catch on, culminating in the 2006 debut of Amazon Elastic Compute Cloud (EC2) and the birth of modern cloud computing. We now had a general purpose mechanism (VMs) to spin up isolated compute units that could be provisioned, configured, and managed using tools like CFEngine & Puppet. IT departments begin spending less on the SANs that dominated the early aughts and shift budgets to AWS or VMWare ESXi.
Then in 2006 we see Hadoop spin off from Nutch, and the MapReduce paradigm popularized by Google leads to an optimization of the resource and scheduling problems thanks to YARN and Mesos and related tools. Non-trivial programs and workloads can now be scheduled across a cluster with vastly improved confidence. Distributed programs are now in reach of a much larger audience.
Suddenly new Hadoop clusters are springing up everywhere and the "Silver Era" begins. Tools enabling greater efficiency hit the scene like RPC serialization frameworks (Thrift, Protobuf), improved on-disk storage (O/RCFile, Parquet), distributed queues (RabbitMQ, SQS), clustered databases (Cassandra, HBase), and event streaming and processing (Storm, Kafka).
Coordination becomes more complicated and essential, so we get ZooKeeper and etcd and consul. Managing passwords and other secure data leads to tools like Vault. Logstash and Graylog make log management less of a nightmare. Solr leads to Elasticsearch leads Kibana and we now have logging and visualization for both server and application monitoring.
Developers also begin to take advantage of VMs and it's not long before tools like Vagrant help bridge the gap between development, staging, and production. Our profession takes a collective sigh of relief as getting our distributed programs started on a new developers machine went from three days of trial-and-error to minutes thanks to well-maintained Vagrantfiles.
Still, deployment is a big pain point. A single cluster could be home to Hadoop (which benefits from YARN) and thirty other things being run by teams across the organization. One day you discover that your mission critical web app is sharing resources with your BI team after your CEO calls you in a panic to tell you the website is down. Turns out a DBA transferred a 20TB backup across the same switch shared with your customer-facing website because somebody forgot to isolate a VLAN to prevent backups from interfering with other traffic.
This doesn't even take into the consideration the abominations we call deployment scripts that developers and DevOps wrangle together to get our programs deployed.
Then Docker and containers become a thing. Developers are now able to setup incredibly complex distributed programs with little more than text files. No more coordinating with DevOps to make sure your servers have the latest version of libpng, or spending months fighting to get Java upgraded to the latest version. I shout in glee as I delete Chef from my machine because progress. Then the beer goggles dissipate and the love affair ends and we realize things are shitty in a different way.
Docker Swarm and Kubernetes emerge, and that brings us to today which I'll call the "Golden Era". We now have incredible tooling that deals with provisioning and configuration, resource management, scheduling, and deployment. But like any new era there are rough spots, but I'm positive incredible new tech will popup.
Throughout all of this, virtual machines and containers were the fundamental building blocks that enabled improved clustering and cluster management. They're inextricably tied together. But all-in-all, things have changed VERY little (couhg LXC cough) in 20 years compared to the rest of the landscape. We're solid for another 10 years before anything like this is going to happen again.
