I believe a part of this has to do with some software products wanting to control the look-and-feel of their applications as part of brand recognition, which runs counter to the idea that GUI applications should conform to the UI guidelines set in place by the operating system or desktop environment. Some people blame this on modern Web trends, but this phenomenon is not new. A lot of video games for Windows or the Mac never conformed to their platforms' UI guidelines. I also remember when Apple ported iTunes and Safari to Windows during the Windows XP era and the controversy caused by these programs refusal to conform to Windows' UI guidelines (interestingly enough, the Windows port of ClarisWorks conformed to Windows' UI guidelines). Even Microsoft doesn't always adhere to Windows' guidelines; Microsoft Office 97 introduced the Tahoma font, flat toolbars, and other elements, which did not adhere to Windows 95's look-and-feel guidelines, and although Office 97 could run on Windows NT 3.51, it didn't even bother conforming to the look-and-feel of that operating system, which still followed Windows 3.1-style semantics. (To be fair, though, the styling of Microsoft Office 97 was a foretaste of the styling of Windows 2000, which used Tahoma as the system font. Office 97 blends in perfectly with Windows 2000.)

This "look-and-feel-as-branding" philosophy has now entered parts of the Linux community, as evidenced in this public letter to the GNOME community titled "Please don't theme our apps" asking Linux distributions to not apply custom themes to GNOME applications (https://stopthemingmy.app).

Personally I'm a major proponent of native GUI applications conforming to their platform's UI standards, and I'm also a proponent of users being able to theme their environments. In my opinion the purpose of personal computing is to empower the user. Users should be able to control their work environments and their workflows as they see fit. Unfortunately I feel that this philosophy of user empowerment has been slowly challenged, where the user experience is being controlled.

I used to run the application skins section on deviantart way back in the day. What is now marketed as "dark mode" is what every kid would put up as their first skin. "I turned white black and black white!" It always amuses me to see people absolutely freaking out when programs add this feature today.

To be fair, the feature today is not so much "dark mode" per se, as the ability to centralize the setting so that everything can follow it - especially the websites. I remember using dark OS themes in early 00s, and it was always annoying when you opened a browser or a PDF, and it drowned out everything else with a field of white.

I think it has a lot of potential answers on a case by case basis.

Often people don't want to maintain an old code path. Sometimes people get into that line of thinking without clear understanding of just how many people like what the old code path is doing for them and why. Sometimes they also over-estimate the maintenance cost of the old thing.

In the case of win8, there was a strong drive for "we're doing a tablet now", very little intuition about how their thing was going to be received in the market, and the company culture and talent pool was already markedly different from the one that produced their 1990s greatest hits, so it's not like they were automatically going to reproduce prior success.

Also notable that in vista and win7, turning on the classic look disables dwm.exe, the compositing engine, and some GPU-based optimizations.

Part of it is laziness, part of it is this lurking impression that users should be coddled and are too dumb to hold the reigns on a machine they own outright.