Because it's too big, it's very slow to start, it's controlled by a company that is not very lovely, it's not installed by default on our system, it's free and opensource without in fact being very libre, it has frequent security flaws that don't seem to be addressed seriously... Some of this is probably only partially true, but it gives you an idea why people don't want to use it, whether they're right or wrong.
So it's a good thing you're propagating partial truths then, right?
The JVM isn't easy to beat for software that's more complicated than "Hello World". Sure you can beat it, but you're going to have to work very, very hard.
The security flaws exist in the area few people care about (and shouldn't even really be installed any more) -- the sandboxing and web start code.
Did you include gcc and friends in your "too big" calculations? Hard to program without them.