I don’t think libadwaita is a problem in the direction you’re saying at all—it’s a comparatively slim layer mostly of additional components that should generally not be all that difficult to make optional if you desire. And you can easily use GTK without it if you want.
Rather, I think the problem is that libadwaita hasn’t gone far enough. The advertised theory is that GTK is agnostic, and libadwaita gets you GNOME HIG stuff, but the fact of the matter is that GTK is still heavily GNOME. Their attitude to overlay scrollbars is a good example (though hardly the only one): they deliberately removed the admittedly-clumsy system-wide configurability that was present in GTK 3 (not sure if themes were able to control it before, but I think not, which was itself a regression from GTK+ 2), and have declared that overlay scrollbars are a feature of GTK 4 and there will be no attempt to comply to system conventions, or any way for users to change the behaviour; individual apps can still control it, but there will be no conventional way of switching this, which de facto means that almost no apps will provide any switch for it, and so users that want real scrollbars or want platform convention compliance are left high and dry. They also deliberately dismantled the module loading technique, which was a way of fixing things GTK made a mess of and allowing improved platform conformance support, including things like global menus (it was admittedly technically unsound, but it worked).
The issue (as I see it) is that libadwaita is GNOME's bartering chip. For years they've been anal-retentive about custom theming, be it their infinite diatribes about how XDG only needs to respect 2 themes, the "Don't Theme My App" disaster or their ongoing attempts to put custom stylesheets further and further away from the user's control. GTK4, libadwaita and Flatpak are their leverage here. Each of these systems, when taken separately, make it difficult to theme your app (though still technically possible). When incorporated though, it becomes outright unreasonable to theme things. This is wholly their fault, and this notion that "they didn't know what they were doing!" makes me roll my eyes. They know exactly what they did, and their behavior has been insanely harmful to the rest of the community.
As a matter of fact, I think the Linux desktop might just be permanently split as a result of their actions now. It's a more extreme version of the old GTK2 vs GTK3 argument, but this time the divide is more extreme. Wayland is borderline unusable on anything but GNOME. GTK4 refuses to integrate with any desktop but GNOME. libadwaita is dividing the development of applications to "GNOME app" and "not GNOME app". Flatpak is a broken olive branch that is in a comical state of neglect (and only integrates well with GNOME anyways).
This segregation has to stop, or it will kill the Linux community along with it. In their attempts to make Linux competitive with Windows and MacOS, the GNOME maintainers have completely quit listening to their users and sabotaged everything that people genuinely liked about Linux.
Rather, I think the problem is that libadwaita hasn’t gone far enough. The advertised theory is that GTK is agnostic, and libadwaita gets you GNOME HIG stuff, but the fact of the matter is that GTK is still heavily GNOME. Their attitude to overlay scrollbars is a good example (though hardly the only one): they deliberately removed the admittedly-clumsy system-wide configurability that was present in GTK 3 (not sure if themes were able to control it before, but I think not, which was itself a regression from GTK+ 2), and have declared that overlay scrollbars are a feature of GTK 4 and there will be no attempt to comply to system conventions, or any way for users to change the behaviour; individual apps can still control it, but there will be no conventional way of switching this, which de facto means that almost no apps will provide any switch for it, and so users that want real scrollbars or want platform convention compliance are left high and dry. They also deliberately dismantled the module loading technique, which was a way of fixing things GTK made a mess of and allowing improved platform conformance support, including things like global menus (it was admittedly technically unsound, but it worked).