You could, but then you may as well just run the game on a tablet.
For playing games, especially action games you really want tactile feedback (so that it's easy to judge the angle you are holding the joystick at without looking at it etc). It also helps that good controllers fit well into your hands for long playing sessions.
TV is a better fit for same-room multiplayer. Could even do multi screen games like with the Wii U. Trouble is the GPU is a bit on the weak side, it's fine for the 4" Nexus but probably won't scale up to 1080p
So is the Galaxy Nexus (which this shares a SoC with), and it's considered weak on there. It's unlikely blowing the output up from 4.7" to 47" will be very pretty.
For playing games, especially action games you really want tactile feedback (so that it's easy to judge the angle you are holding the joystick at without looking at it etc). It also helps that good controllers fit well into your hands for long playing sessions.