Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Ask HN: Are Binary Logs a Thing?
2 points by shortrounddev2 on Aug 22, 2023 | hide | past | favorite | 2 comments
I was wondering if there are any logging frameworks which write data in binary rather than utf-8 encoded text. For example, the datestamp might be a 64 bit integer rather than a formatted ISO string. It would probably make it harder to work with *nix-like tools (sed, grep), but would reduce storage usage and perhaps make it easier to do structured queries on the data using shells like powershell.

I know there are services which store this type of data as columns in a logging database, but I was wondering if there were frameworks which used binary files to do so



I'm not familiar with any off the top of my head, but this reminded me of the book Designing Data Intensive Applications, here's one somewhat relevant snippet from it

"JSON is less verbose than XML, but both still use a lot of space compared to binary formats. This observation led to the development of a profusion of binary encodings for JSON (MessagePack, BSON, BJSON, UBJSON, BISON, and Smile, to name a few) and for XML (WBXML and Fast Infoset, for example). These formats have been adopted in various niches, but none of them are as widely adopted as the textual versions of JSON and XML."


Doesn't the journald component of systemd do this? I seem to recall it was much reviled for exactly the problems you point out - that it is harder to work with than plaintext logs.




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: