"So the point of my keeping a notebook has never been, nor is it now, to have an accurate factual
record of what I have been doing or thinking. That would be a different impulse entirely, an instinct for
reality which I sometimes envy but do not possess. At no point have I ever been able successfully to
keep a diary; my approach to daily life ranges from the grossly negligent to the merely absent, and on
those few occasions when I have tried dutifully to record a day's events, boredom has so overcome me
that the results are mysterious at best. What is this business about "shopping, typing piece, dinner with
E, depressed"? Shopping for what? Typing what piece? Who is E? Was this "E" depressed, or was I
depressed? Who cares?"
Joan Didion, "On Keeping a Notebook"
Apologies for the long quote but this is pretty much how I feel about all attempts at "personal analytics" so far. As other posters have mentioned, analytics are great for businesses who have their FOMs and TPMs and KPPs and whatever to maximize already, but on a personal level the best use of diaries and journals is to force yourself to realize what your goals are, not to help achieve them.
I liked "Where I was From," the book she wrote about how crummy California is and how she had disowned it as her childhood home. I was feeling down on California at that time, and it was a great book to read.
Personal analytics software is something I would never use. It must be nice to have a life worth quantifying.
I've built this and have used it for years. Initially I decided never to make a product out of it so that I could allow myself to experiment and really just solve my own problems.
Its exactly focused on insight and de-cluttering the mind and finding focus. Being happier and more creative.
It really works and lead me to significant periods of peace and contentment and working on meaningful things.
In the last year I did a lot of work to turn it into a proper business, but got caught up in another startup which took up all my time.
I am planning on making it into a product with great design.
That's really the primary work now: crafting a design that is communicates the data in a clear way.
Some messy bells and whistles thing with all the data and graphs and features.... nope. Not interested. I want clarity.
I'm building a mobile app, called DidSum, that focuses on your #2 "Avoid Data Entry, but Make it Easy" (and putting an API around the data). It's almost ready for broader use.
I use DidSum to track running, blogging, eating healthy, sex, coffee consumption, and more.
DidSum allows you to define the actions you want to improve at. You can track publicly with friends and family and encourage/compare/compete with each other. You can also track privately (some actions I only track with my wife). It has basic analytics, which will get more powerful with time.
We're in a closed beta right now, if you're interested in helping test I'd love anyone interested in this thread's feedback: http://didsum.com/auth/register.html
You need to separate the signal from the noise, and ramping up the noise isn't the solution. Quality takes precedence over quantity. Benjamin Franklin identified thirteen 'virtues', and those signals seemed to work for him.
> I propose a simple data entry framework: click a button, and that event gets tracked.
The problem with data entry is that it requires the motivation and initiative of the user, which is precisely what the user lacks. That's the reason people get personal trainers and life coaches: you need someone else to keep you accountable and motivated.
> Machine Learning Matters
Here's where I think almost all of these personal-analytic type applications go wrong: it's trying to solve a very subjective human problem with a very objective machine program.
Here's an alternative startup idea: a web application for life coaches and their clients, focused on providing them the tools they need to communicate better with one another. I'm betting that life coaches use email, text messages, phone calls, a calendar, and maybe a spreadsheet. They probably could use some help integrating all those tools and all the data they are collecting.
You don't think there are correlations in behavior? I'm pretty sure when I drink I don't sleep as well. What if you could show that with actual personal behavior instead of "some study"?
And as far a motivation, the mobile component is essential because it will literally buzz in your pocket to remind you to do things.
> If you consider the number of people that want to lose weight, quit smoking, or get better at something, I’m probably wrong about the market size. It’s huge.
Everyone wants to improve something (or everything!) about themselves. Most people have a sense of what that is, and what they should be doing. Stop smoking, go to the gym, get more sleep, eat better, spend less money, make more money, etc. A personal analytics system may help someone optimize their life from seeing the hidden details, but I claim that the steps to self-improvement for most people are fairly obvious. The problem in self-improvement is not lack of data, it's how to execute on the obvious known fixes (how to stop smoking, etc).
What I'm finding from using and talking to beta users of DidSum (referenced in another comment on this post) is that the act of physically tracking a behavior has value in building habits and improving toward goals (or being extra aware of it, when you're not).
This is doubly true in a social setting where your friends/spouse/mom/coach can _see_ when you are or are not taking actions toward what you want to improve at.
When a friend updates its movies everything is recalculated like in a spreadsheet. The core idea behind this is to end with emergent behaviour from the small scripts of millions of people.
This is exactly the problem we're trying to solve with http://exist.io.
For the moment we're starting small and just trying to pull in activity tracker data (Fitbit, Jawbone, Withings etc) and some secondary services that provide useful quantifiable data - things like Foursquare checkins, Gmail, task management tools, productivity trackers, etc. It's hard to interface with every service when you're writing custom API clients for each of them. And on the other hand, if you try to dictate a common standard without any clout to back it up, you just end up with http://xkcd.com/927/. We're working on an API to export all your data too, so people can build visualisations and apps on the aggregates of their data.
Tracking things has to be as frictionless as possible, so I'm wary of manual tracking unless you're already quite committed, or it takes mere seconds - the interface in #2 looks like a good way to approach this. But we're avoiding this for now.
The other thing we strongly agree on is #4 - you need insights, not just data! Numbers alone are of limited value, and a lot of people already into Quantified Self end up with "number fatigue" where they're no longer motivated to put in the effort to track things, let alone work towards a goal, just so they can see the number go up. People need to qualify their data and make it part of a narrative about their life. They also need actionable feedback, which is where we're trying to focus on providing value - if you're trying to be more productive, on which days is that working, and what are the factors behind it? We'll try and tell you so you can focus on that. Same process for becoming more active or anything else.
It's a big problem to solve, but so many possibilities too.
I think the oversimplified nutrition analytics is actually killing one of the biggest potential sources of insight from personal analytics. The key then is having a large, accurate database of nutritional information so you can see which fundamental ingredients a users is constantly ingesting.
Other than that, I totally agree. Someone please make this app-- then let me do the machine learning for you. :)
Users won't input that data. I care a lot about this, and found using apps like LoseIt to track food way too much overhead.
I know some people are trying food tracking via taking a photo. Taking a photo per meal is a lot of overhead! That said, 4 Hour Body recommends it to stick to a nutrition goal.
Also, I disagree with your premise. I know what I shouldn't eat, and the main questions I want answered are what correlates to my making bad decisions. How does sleep relate? Alcohol? Being around friends? Tracking specifically what I eat seems irrelevant.
I've been taking a similar approach to a side/hobby project that relates to personal spending/budgeting.
I am seeking to apply the "easy data entry" + "subjective good/ok/bad" measures to help folks discover patterns of bad spending decisions. While most every budgeting/spending app will show you "how much you have left" or "safe to spend" the real measure of improvement is whether you made good decisions or bad decisions given your life circumstances. This also has the effect of softening the penalty for missing data. If you miss some data, you just pick up where you left off. You're not trying to balance your budget. You're tracking decisions. Although I'm not (yet) biting off the notion of correlating it with other personal analytics, that's an interesting extension of the idea.
I'm really convinced that a behavioral approach to spending/budgeting would lead to longer lasting positive changes than fretting about exceeding your grocery budget by $19 due to unexpected company joining you for lunch.
The issue is that of avoidance. If I have to tell an app I've been bad, I'm more likely to avoid telling it at all. (Like not getting on the scale when one knows the weight won't be good news.)
Sure, highly motivated users will track everything, but it doesn't take much to leave the highly motivated state. There's also the angle that knowing you have to report in with bad news might stop you doing the bad thing in the first place, but when the penalty for not reporting in is nil, it loses some of its power. Perhaps approaches that use human accountability or other reward systems (eg. GymPact) could help here.
In some circles I've described it as "a money tracking tool built for couples" and that's true. This all emerged from a chart my wife and I had been using on our fridge. We wanted to address the decision making progress as a couple and not beat ourselves up over specific failures (impulsive spending, false bargains, etc), but try to improve our process for budgeting and spending money.
Sites like MyFitnessPal and MyNetDiary would likely disagree. They have hundreds of thousands of users. However, I agree that if you're in generally good health then it probably is too high of a burden.
> I know what I shouldn't eat
I'd argue you probably do not know what you should or should not eat. Sure, in general you know you should avoid eating candy for every meal, but I suspect you don't really know which particular ingredients are resulting in a change of your state.
For instance, many people are gluten intolerant for years before realizing it. If you were able to track foods to the approximate level of knowing the level of gluten in your body at a given time, you could identify gluten as a cause of some symptoms. I would also make similar arguments about the long-term effects of certain diets (semi-vegetarian, low-carb, etc.).
I guess there is a divide between the kind of myopic analytic insight you're looking for and the more long-term ones I'm interested in. Both are valuable, but they require different data sources.
>>Sites like MyFitnessPal and MyNetDiary would likely disagree. They have hundreds of thousands of users.
That doesn't mean much. I used to use Fitocracy to track my workouts, and while the app itself was very useful, the data entry part absolutely sucked and it was ultimately the reason I gave up on it and went back to pen and paper tracking.
What companies have been at least mildly successful in Personal Analytics? From where I am standing I can't think of any.
Yes Fitbit is a great pedometer that gets people to walk more but last time I checked they weren't recording that much data and linking to connected scales is useful but its barely scratching the surface.
One issue with Personal Analytics is we need more data and it needs to be EFFORTLESS to record. The ROI is only high enough when the effort is very low. This is a challenging problem that requires "true innovation". I don't think a tracking app on my mobile phone is going to be the answer.
Another issue is the ability to improve some facet of my life with this data(weight, sleep, etc). Even if I can correlate drinking coffee in the evening is causing me sleep problems unless I can change my habits its not useful information. We need to be able to change our habits and processes based on this data. There are scientific strategies to get people to change habits and simply telling them they need to walk more or eat less do not pass scientific muster. Trigger -> Habit -> Reward. If we need to change Habits we must replace them something. If I need to stop drinking coffee then I need to replace it with something.
Well the basis watch is pretty good in the passive recording department. It automatically tracks your sleep and activity levels with many sensors and can give you a pretty good estimate of your daily calorie burn. You never have to press any buttons to track sleep like with the fitbit, making it purely passive. It's also in a watch form factor, preventing the device loss you can get from the fitbit form factor.
Basis although is lagging in the software department. Their mobile apps are not very useful and their API is not open so people can't grab their data without doing unofficial hacks that accesses the web apps json data directly.
I think although people are going to be using something like the iPhone 5S's M7 motion tracker more than something like a fitbit, because they carry their phones everywhere and don't have to buy or manage a separate device.
This would be the best current combo if basis opened it's data silo:
1. A basis watch for sleep and general activity.
2. A wifi weight scale for weight & easy to measure BF%
3. Runkeeper, etc to track your cardio activities. Basis & fitbit doesn't track actual fitness activities (like biking or hiking) too well compared to runkeeper.
4. Also lets you manually input personal measurements such as your navel, hips, waist, etc from the mobile app. Fitbit does not do this.
All of these items in one cloud-managed website & mobile app will be pretty powerful. The closest you have for this is fitbit right now.
In the future, a credit card / payment system that automatically tracks nutrition purchasing info will be very powerful. This needs to be combined with more nutrition estimates in restaurants although.
Passive 24/7 location tracking can also help co-relate events. It can track the effects of vacations, ski trips, trips to the pool, more grocery shopping less restaurants, etc to put more meaning into your graphs. It can co-relate you going to the gym frequently or not going to burger king anymore with you losing weight.
I think just awareness that your slowly gaining weight can help snip weight gain trends in the bud, along with warnings your eating a lot of high calorie crap that you might not be aware just how high calorie it actually is.
Having the right data be effortlessly recorded is key. If I have to manually input data, I'll forget to enter things and my data will not be completely accurate. Plus, if I don't get immediate value out of the system, I'll stop entering data all together because it's not worth my time.
But, with the new wave of smart watches, glasses and other smart technology there is an opportunity for that data to be easily imported into a personal analytics application.
4 months after our 2nd child was born my wife still wasn't getting enough sleep. Part of that was the child's sleeping pattern and part of it was her not being disciplined with sleep (not taking naps or staying up even later). Of course my telling her "you need more sleep" fell on deaf ears. So I recommended she get a fitbit since it claims to track one's sleep. It was a big reality check to see on her computer in a big red circle that she was only getting on average 4hrs of sleep each day. Just seeing it on the computer meant she could no longer ignore the truth. It's the same as a doctor's prognosis vs. a friends opinion. The former carries a lot more weight. She started getting more sleep the following week.
If someone can create a system that nearly automates the tracking of what you eat you'll make gazillions and help a lot of people along the way. Maybe a bluetooth food scale in conjunction with a tablet that can do "facial recognition" on a plate of food to estimate what I'm eating and the carb/protein/fat/calorie breakdown. Then based on my physical activity for the day and calorie consumption it could recommend I skip desert or have a carrot instead.
How would tracking all this be useful to you? Why would you pay for it? The reason why tracking is more common in business is that knowing the metrics gives you guidance on how to be profitable. Therefore you can optimize all your metrics to paint that picture. With personal analytics, there isn't one thing that delivers that kind of ROI, even if we're talking about the investment from inputing the data alone.
People talk about "getting to an a-ha moment". I think this is really relevant here. It's one reason services focus on visualization: it looks like you have something right away.
I would focus things around a goal, and make clear that the service will help you stick to it over time. The benefit is over time and not immediate, so I think you're point is really valid.
It also depends on the quality of the insight. Maybe you can tell from my email and social media patterns when I'm sleeping, and make recommendations a few moments after you auth.
The idea is that we're simple stimulus-response machines running along the treadmill of life, with our day to day consciousness, identity & lifestyle choices dictated by our fuzzy pattern-matching ability; we are subconsciously conditioned by rewards unconscious to us, and we live our life by unconscious heuristics designed to maximize progress towards such rewards.
If we could better understand what we actually enjoy then we could design a lifestyle that maximizes periods of real well-being and enjoyment. Happiness is a loaded word but wouldn't you contend that the modernity, and all the alienation contrived therein with life reduced to private experiences, could be improved for all by an optimization of private experience?
Yep. I created a system that addressed 3 or 4 out of these 6 points a few months ago, but struggled with adoption and monetization simply because it's hard to find a tangible use/benefit from seeing these trends as an individual. One thing I am/was considering is the value of comparing your data to that of others.
Personal analytics can help you decide e.g. if those expensive supplements you started taking last month do in fact improve your sleep, or if you burn more calories at the gym than when you go for a free walk. Sometimes the effects are obvious enough, but often life is too messy (and your thinking to wishful) to reach a correct conclusion without some help.
I wonder if you can really reach a worthy conclusion. Because life is so messy, it seems hard to control for the thousands of other factors that could affect what you measure.
I feel that the most important part of this is having clear data ownership rules to assure users that they actually own their data. That is, they can take it in a machine-readable format whenever they wish, and with it remove the right of the company to continue holding the data. I am very much in favour of stepping up personal analytics, but there's no way I'm going to just hand over some of my most personal data and hope that the company doesn't use it as the ultimate lock-in tool.
If a web platform is developed to provide storage and an API, the only way I would trust it is if it were completely open source and users were encouraged to run their own nodes (as well as supporting easy data migration between nodes).
Considering your early adopter audience is probably going to be highly technical, I think this matters. Having a clear benefit also makes the privacy cost more appetizing.
But frankly, requiring users to run their own nodes is a terrible idea and would doom a startup that followed this advice to failure. It would please some early adopters, but make growth essentially impossible.
I don't think it is necessary to require users to run their own nodes, but simply allow the option and encourage a community to form around doing so.
Basically, set up a scenario where the startup is service-oriented and must stay honest because it competes with its own community for users on the official node. Most users will prefer to trust the startup with their data for ease of use, and there will be an enormous amount of credibility gained by having a mechanism to keep the startup's greed in check.
I didn't mention this, but the value and the revenue will come from solving people's problems. I think things are really well aligned here. You have a really cynical view that most people don't share. You're probably closer to reality, but what users actually care about matters more than reality.
I should probably mention that I have put quite a bit of work into developing a similar system myself. I even prototyped and pitched it a couple years ago, but basically got shut down by judges who were offended by the kinds of data I suggested tracking (i.e., everything).
I am quite cynical about companies that extract revenue from personal data (LinkedIn, Facebook, Google, etc.), and I recognize that most people probably aren't. I suppose I am more of an idealist and would have trouble sleeping if I inadvertently created Big Brother, which is why I would risk hamstringing such a startup to try to ensure that it stays honest.
I've written a response, but I suppose I am hellbanned for new posts. No clue why unfortunately.
The basic idea: prompting is of great importance in this field. You can't get GREAT data through passive measures, but you can get much more consistent data if you consistently prompt. (This is my personal method, at least.)
[edit] It's not so much of a response as it is related - a way to capture data about self. Not so much related to the startup viability discussed in OP's article.
"Every piece of content created: mobile camera roll, instagram, facebook, vine. Every piece of content consumed: amazon, itunes, netflix, spotify, rdio. Every place you go. Every dollar you spend. You won’t start with 100 services, but you’ll get there soon enough. Some people will be obsessed with adding more data"
And therein lies a potential headache. What if the API for those 100 services break? What if just 10 of them change their API every month? Are you going to update them? And oh btw, you're gonna keep doing this before you actually have paying customers.
I'm slowly creating something similar to learn rails.
I want to track how much spend, consume electricity or gas, what are my goals.
The problem I found that I forget to enter the data into the app so it is fairly sparse. I can only get meaningful data on a monthly base.
I imagined it would be capable to compare gas (heating) consumption with the weather on the same day but it is not really working (without some automatic data input, that I don't have).
Among others I though about adding mood tracking, attaching 1-3 picture/day so when you want to watch back on your vacation you are not overwhelmed with 5000 photos, tracking locations. I would like to think about it like a social network where I am alone or something like this. (Codename: Track My Sheet :-) )
Showing trends, adding watchers like "you now need 1 year 3 months on your Y Account to reach Z amount at your current rate +Q USD / day, but these kind of things are faster with an excel sheet. Here is a dashboard mockup: http://snag.gy/OozTi.jpg
It is similar to the reportr.io app, actually I started mine on the same week before I first saw it on HN. :-)
I wrote a chrome extension to analyze the websites I visit on certain days and track how long I spend looking at each one and graphs it. It's called melytics and it is pretty buggy but it does the job.
Honestly I think this is too much data to make use of. I'm just one person. How much meaning can you get out of what I do? If it is a lot, do I really want that platform to exist on the web?
I worked on a concept for this for a while. I ran into the following contradiction I couldn't solve:
If the data entry is quick and easy, then I'm not gathering empirical usable data. If I try to gather good data the user will become annoyed.
I think data gathering apps work for a small scope when the user is engaged enough to provide the data. Fitness or body building for example. Or sports car races etc.
This was making me think that maybe a generic statistics app with user generated values/measures/timers might be interesting. But once again, non technical folks probably aren't interested in statics too much.
Maybe gamification and a great UI is the answer here or maybe one should start with a specific problem (workout, drinking, sex or whatever) and broaden the scope as he analyzes user interaction.
I would give such an app a shot, but I'm not sure if I wouldn't forget about it too quickly. Just as I forget about using Lift (the app) and other interesting stuff.
Completely agree that "graphs without the ability to manipulate data are worse than nothing"--it's hard to gain much insight without being able to drill down and filter the data. Simple questions such as "how strong is the correlation between room temperature and sleep" end up having to be refined e.g. with "ignore weekends" and "correct for different amounts of physical activity".
Being able to "get as much data as possible" sounds good, but garbage in, garbage out applies...
Finally, I don't think it's possible to create a single mobile app that suits everyone, which is why Zenobase has a flexible, generic data model and an API.
The biggest challenge imo is privacy and security: How do you safely collect and process everything (from emails to texts)? While I could implement something for my personal use, would anyone give access to all their communications to a third-party service? I wouldn't.
Maybe there is some better way to collect this information (just metadata? do some analytics and store results instead of raw messages?).
My biggest issue with personal analytics is storage. I want to track everything, from browser history to keystrokes, but I don't want to share this information with a third party. I think solving the data problem is the first hurdle.
The way it is now, each front-end service has to do a ton of work for each data source they want to integrate with. There should be services that translate data into a shared format. And then front-end services can focus on their idea instead of having to parse all the data.
I think social media platforms are slowly building towards generating massive personal data. I thought with the advent of Facebook timeline, user events (both macro and micro) would be well documented in an aggregate consumable form.
Where do you get your revenue for personal analytics like this? Therein comes super targeted ads with pinpoint accuracy, using predictive analytics at a personal level. If you buy a brand of milk only on Saturdays, rivals have a higher incentive to strut their wares on next Saturday.
Sorry for the following unsolicited ad. I work for a company in the Washington, DC area that is making almost this exact product. Reading this article was like hearing a stranger talk about what I do at my day job.
In addition to the Quantified Self crowd, we are also (for now, primarily) focusing on the educational space. We are currently hiring developers. Most of the code is Python, but we need front end help also. Contact me at personalanalyticsdc@drbacon.33mail.com if you are interested in learning more.
I've been thinking about this for a while. It seems almost impossibly easy and yet no one has really done it.
Maybe it's because the consumer startup crowd and the data science crowd don't overlap enough. If http://prediction.io supported time series data you might see a few of these products pop up overnight.
> It seems almost impossibly easy and yet no one has really done it.
How does it seem easy? If you just want to track one or two data points and put them on a graph sure, it's easy. If you want to track 100 sources of data, each in a different format with different types of data it becomes quite hard. Then try taking that data and analyzing it. Try looking for correlation between your manually recorded caffeine consumption and your sleep patterns logged with FitBit / BodyMedia / another service.
It's a hard problem with a limited market and limited options for monitization.
>It's a hard problem with a limited market and limited options for monitization.
I disagree; I think there is enormous potential in this market; having a set of concise and insightful information that shows you are making progress with a particular goal is something a large number of people would be interested in.
This sucks.
We don't need some new "exciting" personal helper shit, what we need on hacker news is exciting object recognition algorithms, better nature language processing module, high accuracy text detection and recognition in natural scenes.
Why are there so many people building crappy stupid things that no one really need?
you web dev people just won't get the big picture. that's why some black smith can learn to build a website/app in a couple of month but he can't do machine learning, numerical optimization or natural language processing or what have you. because it not the same thing. anyone could build websites, wake up!
I remember starting a company several years ago focused on using advanced analytics to improve personal decision-making. This idea morphed over time to a specific use: budgeting. Visualization and making the process quick and super easy are of the utmost importance for "personal analytics".
Visualization with insight in budgeting is pretty easy because you're trying to answer "where do I spend my money?". Categorization needs to be good, but then you just need a distribution.
Mint is surprisingly bad at automatic categorization (failing on things like large transfers that trump all my other data), so I think there is an opportunity here.
I’ve enjoyed reading your post but I’d be more interested in seeing how you go about fleshing your ideas out. I personally keep a similar list but have trouble taking the ideas from rough works to completed business plans. I’d love to see what processes/frameworks other people use.
Maybe the slow carb diet is a bad choice since it doesn't work... always results in bouncebacks (2 data points for you; 1 for me). Maybe only useful for a photo shoot or weeks leading up to a beach trip..
My startup sympho.me is working on this challenge. Trying to help people get a holistic view of their lives though quantified self.
Please send feedback:D
Joan Didion, "On Keeping a Notebook"
Apologies for the long quote but this is pretty much how I feel about all attempts at "personal analytics" so far. As other posters have mentioned, analytics are great for businesses who have their FOMs and TPMs and KPPs and whatever to maximize already, but on a personal level the best use of diaries and journals is to force yourself to realize what your goals are, not to help achieve them.