Is it really helpful to have different components of the same project in separate repositories? Are they really that independent? Four independent components for a single window manager seems a bit high to me.
Regarding modularization, I tend to prefer Fowler's criterium that a component should contain code which is usually changed together. So, if you have one component where 95% change slowly and 5% change often, you should split it. However, if you have two components that you almost always have to change at the same time, you should merge them.
You could take advantage of having a single git repo for multiple cargo packages. This has worked extremely well for me. It was a lot easier having a single git repo to manage and control but still have separate packages for each of the different pieces. You could have the main package become the executable and have the core and various other packages in there, too.
Regarding modularization, I tend to prefer Fowler's criterium that a component should contain code which is usually changed together. So, if you have one component where 95% change slowly and 5% change often, you should split it. However, if you have two components that you almost always have to change at the same time, you should merge them.