Holy moly, my 3 year old post on the front page! The power of blogging. Cool.
Quick update: Yup, I am still doing this. I enhanced my tooling in VIM quite a bit, and logging is super fast now. Earlier today I entered log line number 30558.
I also wrote a bunch of software to visualize and analyze cause and effect in this type of log. Some of the many potential cause/effect relationships look promising.
Since the data in this type of free-form log does not fit into the typical statistical tests very well, I have no "significant" results yet. That's why I have not posted an update so far. I first want to improve my statistical abilities.
If anyone knows of established tools to analyze such a log, or has an interest in building new ones, get in touch!
for unstructured message analysis -- it's not a silver bullet, but you may want to try BERT or another newish embedding tool
(you can do this on your laptop, you don't need to upload the data anywhere)
then use like sklearn OPTICS to cluster (clustering method that doesn't require you to know the number of clusters in advance)
If there are obvious clusters, you may be able to label a few useful categories. If you get lucky and most entries have a cluster, you can potentially manually label the outliers.
I think I saw an example where you said you use it to answer questions like 'How many curls did I do with these weights last time' [1]. That was from a couple of years ago, and I'm assuming you are the commenter there. If that's correct, do you have a way to make this system portable? I feel like the place I'm most likely to need to answer such questions would typically only have my phone available.
I greatly appreciate the data you're collecting and the overall method of doing so. However, I'm curious to learn about the portability of this information, and how you access it when you're away from a computer with vim.
> If that's correct, do you have a way to make this system portable? I feel like the place I'm most likely to need to answer such questions would typically only have my phone available.
If your phone runs Android, you can just install Termux and then do everything exactly the same. Stick the files in a git repo, edit with vim. If the ergonomics bother you, I think it's possible to have termux open files in an android native text editor app, but I unironically read and edit my files in (neo?)vim and it works fine.
Interesting. I wonder whether they have learned anything from their observations.
I would probably let numbers be part of the tracked event, because it might be important for things like. "Took 500mg of painkillers" or something.
I’ve been doing a similar thing for over 1,000 days using an app called Daylio. It’s been both useful and rewarding to have a large archive of daily activity to look back on. I try to review old entries on an interval, usually at the start of the year.
https://rickcarlino.com/2020/journal-apps-with-reminders-hel...
Some other posters have noted there are apps for broad personal data, but this is a whole other level that I think could be super valuable for those who want to build a scheme as they go. The blocker for me is that it’s not an app - if I’m going to do this kind of logging, it needs to be something I can put on my phone.
Apple Shortcuts or similar can handle this efficiently. Can even pump it straight into other apps like Apple Health to visualize a variety of metrics that you want to log. Multiple apps too, one tap can enter a metric into Health, log it into another app, and then into a text file.
Takes a bit of upfront work to personalize it, but it's mostly just tedious not actually strenuous.
What works for me is that when I'm on the go, I just write events into a text file on my phone.
2023-06-27 16:00 Drink_Coffee? Yes
And type them into my log when I'm at home. Sounds cumbersome, but it's in fact quite easy. I also log the time I spend on the self-tracking project and while it's one of my main projects, it only takes 10 to 15 minutes per day.
As always, I’d recommend ISO-8601 style timestls so which would only require one additional character (space swapped with T and a trailing Z) for the same semantic meaning, but could be extended to support time one offsets, fractional seconds, etc. without changing the parser.
Nice. I do something like this except less complex. I use Blitzmail (there's an iPhone equivalent, but I don't know what it's calle) -- basically, it's an ultra simple app that pops a window to email only one person (usually, yourself) and have replicated a script on my desktop to do the same.
Then I have another script which just makes them ALL "todo" items in the journal of http://zim-wiki.org, and I just "check" (triage?) those every day.
okay, every few days, but i'm trying to better about it.
Either way it's replace a LOT of individual "apps" for me.
I started down the path of building a narrower version of this to track my personal activity. The syntax was a similar date log format (time optional) but it was unit aware and it used commas for labels. Example:
The idea was to eventually generate a static site with automatic graphs and tables but I eventually just switched back to a spreadsheet. It is hard to beat a spreadsheet for flexibility.
Makes me think of https://www.conventionalcommits.org/en/v1.0.0/ for your life.
Any recommendations for tracking while away from your computer? I love ease of use but I think I would lose large parts of my day & weekends when not at my desk.
I would just use a cloud product like Google drive or Google notes for this. Then at the end of the day paste it into the correct location in the log.
I've thought about this type of tracking in the past, but I haven't been confident enough that it would actually find useful patterns to put in the work to implement it.
If you're using a phone, you could email yourself with a journal specific subject then use an API to automatically add those observations to a log. (Might be able to do that with Google Drive or notes too, I'm not sure if they have API hooks).
If you don't want to keep a phone, a small journal is also a possibility.
This is cool. Would love a system like home assistant w/ little blueprints that you could add. For example maybe you want to track your location and have that get defined automatically, i.e. anytime I visit the coffee shop, I’m working, so define that as work.
If you're on iOS, Life Cycle[1] can do this kind of thing for you. You can assign locations to categories (and it often does a good job of guessing them anyway) and it'll tally them up on a weekly/monthly/yearly basis.
Since it is just a text file, it is easy to merge it with other data at any point in the future.
So you could log your position on a different device. And only when you want to analyze the data, you write a script which converts the positions to lines like:
I have tried some similar things. I like your approach and write-up a lot!
You asked about tools. Well, from the "when you have a hammer" department..
you could use plain text accounting tools for some quick reporting wins.
Some examples:
If you have a.dat:
2020-05-28 18:41 Eat Pizza
2020-05-29 09:00 Slept with the window open
2020-05-29 09:00 Headaches
This is close enough to hledger's timedot format to do some reporting.
Each line is interpreted as an empty transaction,
which you could query by date or description:
You could transform your data to a plain text accounting format, with quantities. Eg, make it TSV:
$ perl -pe '$c=0; $c++ while $c < 2 && s/ /\t/' a.dat > c.tsv
$ cat c.tsv
2020-05-28 18:41 Eat Pizza
2020-05-29 09:00 Slept with the window open
2020-05-29 09:00 Headaches
and use hledger CSV conversion rules to customise and enrich it:
$ cat c.tsv.rules
fields date, time, description
# save the time as a tag
comment time:%time
# count each item as one "event" by default
account1 (events)
amount1 1
# special cases
if pizza
account1 (food)
amount1 200 cal
Now you have a (single entry) accounting journal:
$ hledger -f c.tsv print
2020-05-28 Eat Pizza ; time:18:41
(food) 200 cal
2020-05-29 Slept with the window open ; time:09:00
(events) 1
2020-05-29 Headaches ; time:09:00
(events) 1
Allowing quantity reports:
$ hledger -f c.tsv balance -MATS cur:cal
Balance changes in 2020-05:
|| May Total Average
======++===========================
food || 200 cal 200 cal 200 cal
------++---------------------------
|| 200 cal 200 cal 200 cal
$ hledger -f c.tsv activity -D desc:headache
2020-05-28
2020-05-29 *
$ hledger-bar -v -f c.tsv cur:cal
2020-05 200 ++
$ hledger-ui --all -f c.tsv # explore with a TUI
With "tools" I meant statistical tools, aka statistical tests.
Because the log is a time series where events happen at random times, the typical tests like the t-test are not readily applicable.
One way I am currently generating p-values is to look at questions like "When I did X / did not do X - how often did Y happen within the next 24 hours?". And put those numbers into statistical tests like the t-test.
But since "Y" often is not just something that happened but something that has a value (For example "High energy / normal energy / a little tired / very tired") this discards some data. This and many other specifics in a random-interval-event-log probably allow for better tests than I came up with so far.
The first 4 lines have the same timestamp because I use a vim macro to ask me about these 4 values. The macro then logs those 4 values at the wakeup time.
Quick update: Yup, I am still doing this. I enhanced my tooling in VIM quite a bit, and logging is super fast now. Earlier today I entered log line number 30558.
I also wrote a bunch of software to visualize and analyze cause and effect in this type of log. Some of the many potential cause/effect relationships look promising.
Since the data in this type of free-form log does not fit into the typical statistical tests very well, I have no "significant" results yet. That's why I have not posted an update so far. I first want to improve my statistical abilities.
If anyone knows of established tools to analyze such a log, or has an interest in building new ones, get in touch!