At a bare minimum you should have Google Analytics integrated into your site before beta starts. Its simply too powerful to avoid doing. (If you hate the idea of the borg having your data, I suggest getclicky.com , but it is grossly inferior for tracking conversions currently.)

I'd suggest everyone read the Startup Metrics for Pirates presentation, which is a work of awe-inspiring genius. Seriously. Read it, make money. It took me probably 18 months to learn as much doing thing the long way as this presentation will teach you.

http://www.slideshare.net/Startonomics/startup-metrics-for-p...

Anyhow:

Identify your key conversions early. Track the heck out of them. Hint: does money change hands? If yes, its a key conversion.

Then build backwards to things which are necessary prerequisites for your money conversions. In my case, I sell software. The majority of sales happen as a result of free trials. Accordingly, I track the free trials. I also track successful installs of the free trials. Those numbers should be roughly proportional and periodically increasing or I'm doing it wrong.

Incidentally, the problem with data is that there is too much of it. Analytics will drown you in data if you let it. Get in the habit of not looking at anything that doesn't have the prospect of an actionable decision you can use to increase business value from it. For example, I could tell you what percentage of German users use Firefox, but knowing that doesn't help me sell software, so I don't slice data that way.

Re: "hating the borg having all your data". If the budget has an extra 4-5K in it, the GA can be replaced with its standalone version - Urchin.

http://www.google.com/urchin/index.html

I'm a big fan of Mint - no, not the financial app.

http://haveamint.com

$30 and tons of plugins... does a great job giving us the type of data we really only have time to care about. I run both GA and Mint, but Mint provides real-time data and better referrer links so we can dig deeper into sites and find comments about our products/us.

Page tracking analytic services would not be useful with respect to what this article is about--they do not yield very insightful information to help learn about customers.

Common sense says to track absolutely everything. Flickr (care of Cal Henderson) have written extensively about metrics and analysis.

A framework that lets you track the results of A/B variate (or bucket) testing helps for trying new things, too.

as a startup you shouldn't track anything beyond the basics(i.e. # of users)...finish the website first...add the features....then when you have nothing to do, you can focus on doing all that a/b testing. If you spend the time building that prior to launch, you are wasting your time.

vaksel,

We're talking about startups that are launched and continuously iterating to find the right model. We only spoke about A/B testing once, when starting out tracking user engagement and responses is more crucial then building out every feature that would pop in your head with no empirical evidence to back it up.

I would never consider learning about your customers "a waste of time."

Two can play at that game!

As a startup you should build the smallest test case for your product, test it, and iterate as the data comes in. That means data collection and analysis from day 1.

++jfarmer, btw you need to write more, I miss it =)

I think at the beginning stage you should store the data. Maybe that's what he's asking. What data to collect.

Close, if you use our analytics service mixpanel.com you only need to ping us data with a single line of code in which we graph, trend, and store all the data. We try to make life even easier =)

Wow, that's neat.