Seems the setting screens are loaded on-demand, maybe using some plug-in architecture that results in overhead. Also, some panels require network communication which makes them even slower.
Each settings panel is run in a separate xpc process, that is launched on demand. So the UI is rendered out of process too.
Open Activity Monitor and see how many processes are run when navigating around in System Settings.
Some parts of the panes are also webviews too, which definitely doesn't help. From memory you can go in and inspect them from the Safari web inspector.
GPU is not a bottleneck. CPU should not be a bottleneck.
Then why?