Except you're not. This is a popular talking point in 35mm circles, but the amount of light on a given area on the sensor remains unchanged, which is what actually matters.
The smaller sensors are challenged to keep their noise down, given they typically have a smaller pixel pitch to keep up the megapickel count.
> the amount of light on a given area on the sensor remains unchanged
This isn't the only thing that matters. It also matters that you're amplifying the noise more by enlarging it 4x as much as you would have on the larger sensor. It's analogous to film in that a larger format will exhibit less obtrusive grain than a smaller one despite using the exact same film speed. You're using the same exposure over a larger area, and thus gathering more light to drown out the noise.
You can either think of it in terms of total light gathered compared with the underlying noise of the sensor (signal to noise ratio) or in terms of enlargement of the image. The result is the same, smaller sensors have "more noise" simply because they're gathering less total light and amplifying everything more.
And don't be fooled by the old "higher megapixel = more noise" fallacy. All else being equal, it's the same amount of noise. However each pixel is gathering less light (signal) to offset the noise, so the signal to noise ratio is worse when comparing pixel for pixel. However, if you were to downscale the output of the higher res sensor to match the lower resolution sensor, you'd find roughly the same amount of noise in both images.
The smaller sensors are challenged to keep their noise down, given they typically have a smaller pixel pitch to keep up the megapickel count.