Sizeable chunk in the source tree. The binary driver built from that is only loaded when the hardware is present...

"Or all in, Linux 5.9 comes in at roughly 27.81 million lines distributed among some fifty-nine thousand source files.

...

For a while now the AMDGPU kernel graphics driver has been around 2+ million lines of code making it the largest in-tree kernel driver. With Linux 5.9, it comes in at 2.16 million lines of code plus another 247k lines of code comments and another 109k blank lines... Or up to 2.51 million lines of code is the AMD DRM driver code including AMDKFD, PowerPlay, DC, and all the kernel code ultimately making up the AMD Radeon support on that driver (but not the older Radeon DRM driver -- that older Radeon driver is at around 157k lines of code). "

You can also build the kernel yourself. It's fun.

If you build your own kernel and you can choose between hundreds of options including building many components as optionally loaded modules. Its all configured through an ncurses interface.

Want a kernel that doesn’t support USB? Sure.