But why discontinue the old method before the new one is ready? This leads to an awkward UX and disinterest by developers, during the time there is no option.

GNOME probably imagined it had enough market power to persuade app developers to change their design. Also GNOME likes to emphasize "distraction-free" computing so the designers probably thought status icons are too distracting for users.

A dedicated tray is a concept tied to windows 95 style UX. I hope the new designs are more about standardizing ways to indicate various statuses and actions of "the app" broadly, without dedicated status icons. For example download progress is still only an Ubuntu Unity protocol that KDE also supports because they can but it's all not freedesktop standard.

It's not just Linux that suffers from outdated tray APIs. It is the same on Windows. To create a status tray icon button, you need to call the Win32 Systray API (the oldest Windows subsystem, forget about using WinUI or UWP). The way apps like Google Drive and Dropbox do their fancier systray popups is that they get the x,y coordinates via the systray API (NOTIFYICONDATA passed to ShellNotifyIconA) and then render a chromeless window at the location with a slide up animation via something like Electron. I don't think the default systray menu can be styled to any reasonable extent.

And it works pretty well, at least Win32 is less susceptible to GUI civil war going on since Windows 8.

How about applications that are controlled entirely from their status icon, like dropbox or vpn apps? Fedora, which normally champions GNOME, is beginning to acknowledge that status icons are here to stay whether GNOME designers like them or not: https://pagure.io/fedora-workstation/issue/246

How about desktops that don't have anywhere to put status icons? i.e. what if I don't want to have any panels? Working as a normal window should be an option in all applications.

With "old" method under X11 the icon was just a tiny window the application created that was then embedded in the tray window and you could do basically anything with it. This caused problems with transparency though so for the sake of a prettier desktop (and because this approach would not work with Wayland) we got a more limited D-Bus protocol where you only get an icon and a menu.

Very interesting. The current state of tray icons is one of the most painful aspects of desktop Linux, in my opinion. Where are these discussions happening, and what kinds of plans are in the works?

The discussion is happening on the Freedesktop GitLab instance.

That's great to hear! It does bring this to mind though https://xkcd.com/927/

Is there a similar reason GNOME doesn't have a convenient task list?

Plank is probably the dock you want.