It was just a hobby, it was really just a cut-down Mac mini with a trimmed version of OS X that would just boot into Front Row. It wasn't really more than that.

It only got serious when Apple got into the ARM market with iOS.

I guess the market wasn't ready either. At the time when the first Apple TV was around, many people still had very slow DSL.