Larger brain by itself is a very strong evolutionary disadvantage - it consumes a lot of calories, and in humans it hampers childbirth which in out natural state is very deadly already.

There is a strong pressure for us to have a brain that's "barely just enough" capable for the functions that increase survivability.