My fear is that people would assume I was sacked/forced-out for being crap though, since there is such a mystique around the company and people dream of getting a job there. Perhaps in a few more years things will tarnish a bit more.
Nah, don't worry about it. If anything people will respect you for following your passions, but more likely they just won't care. Strong developers are rare, no-one will question how they arrived in their interview room, they'll just be glad you're there.
Don't know how you got this idea. No-one will care.