>We’re limited in other ways, weirdly. For example, LORRI, our high resolution imager, has an 8-inch (20cm) aperture. The diffraction limit (how much an 8” telescope can magnify) is 3.05 mircorad. which is just over half the size of single pixel 4.95 microrad. So if we swapped out the current sensor with a higher res one, we couldn’t do much better because of the laws of physics. A bigger telescope would solve that problem, but then it would make the spacecraft heavier, which require more fuel to send to Pluto AND a longer time to get there, because the spacecraft is more massive. We launched Pluto on the largest, most powerful rocket available at the time (the Atlas V, with extra boosters), so again we’re limited by physics: “At the time” doesn’t mean best ever. The Saturn V rocket, which sent astronauts to the moon, was actually more powerful.
>More megapixels also means more memory. For example, LORRI images are made up of a header and then the 1024x1024 array of numbers that make up our image and go from 0 to 65535 (216). There’s not really a way to make that info smaller if we went to 2048x2048. We could downlink a compressed version, but we want the full info eventually.
tl;dr
1. Between optical physics and balancing different costs to launch mass, it was the sound engineering choice.
2. Higher resolution would take even longer to retrieve the captured data.
I'm not an engineer, but it's probably a combination of mission specs being drawn in 2001 and the fact that adapting technology for use in space is a multi-year process.
The Curiosity rover, which was commissioned in 2004 and launched in 2011, has a primary camera with 1600x1200 resolution.