Probably Arch. You can build really small images with Gentoo, Slackware, etc. but Arch is probably the simplest for you to manage.

TinyCore looks cool but not like something I would want to maintain as a server.