"What kept me from [writing my own tool] was a calm voice in my head telling me that I’m here to write a book, not a preprocessor. [...] Now I have written two books and zero tools, which I consider a success."
It's a crazy important quote and everyone should really keep it in mind when they set out to do something.
Many people have problems focusing on the task they set out to do. It's a type of procrastination. Insert that famous gif from Malcolm in the Middle.
This is relevant to startups as well (and business in general): Being able to decide whether something is in scope of a project or not. You are encountering a roadblock, how do you react? Do you set out to solve it perfectly or do you work around it (or even, do you give up)?
The quote you pasted is the perfect way to put it into words. Another common theme I hear about is people asking: "How do I become a successful [profession here]?" (eg. "streamer", "youtuber", "musician", "programmer", "writer", ...). The answer is always the same: "Start".
I think there is a time in a programmer's life when writing tools to make yourself more effective is finally worth it.
For me, that time started about two years ago. At 31, a full 25 years since first learning how to program[0] I finally felt like I could be trusted to make the right calls on when to invest the time to abstract or create generation or code parsing tools; and when to do things the old fashion way.
After two years of spending about 20 to 40% of my time on tooling I'd wager I'm about 10x faster than I was at 30 and about 20x faster than I was at 25.
The problem now isn't writing the code. The problem now is that the code is so fast that I don't have the same amount of passive time to think about features / strategy. I have to intentionally set aside time for it and I find it unnatural. Also, I know too much about cybersecurity which is good for my end users, but bad for my feature productivity.
[0] Poorly. My mum taught me how to code QBASIC. I altered video games.
> I think there is a time in a programmer's life when writing tools to make yourself more effective is finally worth it.
Indeed yak shaving should only be done when one has the breathing room to do so. It's just good common sense, except there are exceptions. Slack was a yak shaving effort. The most valuable things I've shipped have all been yak shaving efforts. I guess I'm just better at building tools than anything else. Perhaps I enjoy it more. shrug
To each their own, but if you're trying to ship a product instead of a bald yak you're going to have to do it the ugly way, with tools you could fix, etc...
This worldview does not take into account network effects, which consolidate the aggregate investment of many people onto a few bad projects. For example Common Lisp is a more effective language than Javascript and always the wrong choice. Corollary: ClojureScript is ecosystem-compatible with Javascript, and TBD if it is a long run better choice!
One thing that pervasively infects every decision is the principle of least privilege. Basically it’s the idea that someone/something should only have the minimum access necessary to do what they/it is supposed to do. It applies to infrastructure access control, network communication, application access control, everything. Once it becomes a default part your thinking you sometimes have to make conscious tradeoffs so you don’t spend forever perfecting things and not delivering any value. Ideally a lot of that thought and effort is common or reusable though.
I'll get my clients to give me access to an S3 bucket that I can push their data to.
Now:
I'll make sure I get my clients to give me a password or public key to encrypt their data before I push it to the S3 bucket because many people fuck up security on those things.
To my end user, it's obviously more secure, but it isn't lining my pocketbook and most end users don't care about security, even if they should. Same thing with security headers or anything, really. Security slows you down at some point.
> It's a crazy important quote and everyone should really keep it in mind when they set out to do something.
I don't agree at all. It seems to me that the most interesting advances of mankind have happened precisely when people were trying to do something else, and had to invent new tools. The tools turn out to be more important than the final result because they can be used by other people.
> when people were trying to do something else, and had to invent new tools.
(Author here.) My point was not that you shouldn't invent tools if you need them. By all means, go ahead. My heart beats for creating tools (especially tools to create other tools...).
What I wanted to express was this: if you want to write _a book_, first write the book and only then worry about the tools. If you then notice that you need a custom toolchain to get the book in exactly the shape you need it to be, do it.
I've had people tell me that they would love to write a book, but, well, the existing tools are not sufficient enough. And I think that's the wrong approach to writing a book: write something and you'll quickly realize that the tooling is not your biggest hurdle.
> What I wanted to express was this: if you want to write _a book_, first write the book and only then worry about the tools. If you then notice that you need a custom toolchain to get the book in exactly the shape you need it to be, do it.
I agree 100% with this. Following the case of Knuth/TAoCP/TeX, in fact the first edition of TAoCP was not written in TeX, he made TeX to improve his already written book.
> I've had people tell me that they would love to write a book, but, well, the existing tools are not sufficient enough.
That’s hilarious. Humans were hand writing books on paper and giving them to other humans to hand write copies. A good craftsman never blames his tools.
> Humans were hand writing books on paper and giving them to other humans to hand write copies.
Not very many were, and a big reason for that was the available tooling.
> A good craftsman never blames his tools
Someone who writes a book by blames the available tooling support for the low quality of its content is a craftsman (but, per the saying, not a good one) blaming his tools.
A person who chooses not to write a book at all because the anticipated opportunity cost given the existing tooling support and that person’s personal utility function is not.
Yes! I use Scrivener. It exists because the developer needed a better tool to write novels. I end up too disorganized trying to write with other tools like word processors or flat files.
I managed to write a 25,000 word story in a word processor once, but it was such a jumble that I just closed the file and forgot about writing for a few years. I know now there are ways to organize with folders and spreadsheets, but now I do most of my writing on the phone. Scrivener's binder makes that easy on a small screen.
Thank you. I've thought about making this exact thing multiple times. Word docs and dropbox just weren't cutting it for my organizing needs. Combined with the scrapple product oh man. The only thing they are missing is a wiki feature for character/show bibles
"Most interesting" seems like a pretty subjective metric. The advances that come to my mind are things like agriculture, calculus, the scientific method, Newton's theories, representative democracy, post-modernism... those all resulted from decades of effort by groups of people who focused on that task specifically.
Tools are super valuable; it's arguable that many of the things I list above are sort of tools. I just don't think that a tool that is quickly slapped together in the service of another project is likely to be a game-changer.
In fact, I'd say that one of the keys to growing your personal impact is to find the best work that's already been done, and use it to further your goal. Even Newton said he was standing on the shoulders of giants.
Calculus of all things was not an end by itself, but a set of secret tricks by Newton to prove results about the orbits of planets (that then he translated into a more cumbersome geometrical language)
Then there are the three branches of modern mathematics, according to V.I. Arnold:
1) modern hydrodynamics and PDE, are just guys procrastinating while they are paid by the manufacturers of nuclear submarines
2) celestial mechanics, control theory and ODE, are just guys procrastinating while they are paid by the manufacturers of rockets and missiles
3) number theory, are just guys procrastinating while they are paid by the CIA to develop cryptographic tools
agriculture : this kid is just playing with small seeds instead of collecting nuts for winter
computers : just tools to speed-up the important computations of nuclear reactors
TeX : the guy just had the write a damn book about algorithms but spent 10 years procrastinating
Bach : the guy just wanted to glorify God
I stand by my point that nearly all important advances of mankind are tools developed to solve a long forgotten goal.
The idea that calculus was some sort of side project for Newton both diminishes the level of focused effort he put into it, and ignores the contributions of Leibniz and the many other mathematicians whose work was known and incorporated by Newton and Leibniz.
If we just open up the idea of "tool" wide enough, everything becomes a tool. Writing a book isn't actually a goal, it's just a tool for acquiring money and fame. Calculating the orbit of planets was just a tool for Newton to establish social credit and historical significance. Agriculture is just a tool for eating, but eating is just a tool for living, but living is just a tool for procreating, but procreating is just a tool for perpetuating genes.
Why are we perpetuating genes? Really, no one knows. Solve that problem, I guess, and finally we'll have the one project enriquto can't easily dismiss as tool development.
"I stand by my point that nearly all important advances of mankind are tools developed to solve a long forgotten goal."
That is indeed true.
Of course, almost all of the goals I have accomplished for myself have been accomplished by actively engaging in pursuing that goal rather than pursuing some side channel, while the side projects I've created have generally been a lot less useful to myself and humanity than, say Bach. :D
[citation needed] -- but addressing your point without a citation, even if what you say is true, it doesn't change anything at an individual level.
If you're trying to write a book, and you want to write a book, write the damn book. There's an infinitesimal chance that the tool you're writing instead of your book will change the world.
If you're trying to write that tool, and want to write that tool, then power to you, but don't lie to yourself and tell yourself it's "for the book". Just admit you're switching focus.
A close friend of mine has been trying to create a game for .. a decade or so. Every time he starts, there's always something. Instead of using, say, Unity, he tried his hand at writing a game engine. A physics engine. A programming language. Because none of the tools are "quite right". Of course, none of those get completed either because they're a shitton of work. And his work didn't change the world, because it's never completed. At the end of the day, much like many in the software engineering world, he's a brilliant mind busy optimizing ads. And he doesn't have a working game engine, a working programming language, or a game.
I don't want to sound demeaning, because it's still very good experience to acquire. And in general I don't want to tell people "don't write your own a programming language". I think these things are super cool. But then if that's what you set out to do, make that your project instead, it'll have a much higher chance to actually complete, instead of sitting in an incomplete state, in a hidden unfinished repo with nobody to share it with.
The common thread between all of mankind's most interesting advances: They're finished. How do you finish a project? Well... start.
> [citation needed] -- but addressing your point without a citation, even if what you say is true, it doesn't change anything at an individual level.
> If you're trying to write a book, and you want to write a book, write the damn book. There's an infinitesimal chance that the tool you're writing instead of your book will change the world.
Not the person you were replying to, but their point certainly rings true to me, from all the history I've read or otherwise heard of over the years. They could just as easily ask you for citations.
"if what you say is true, it doesn't change anything at an individual level" -- yes it does. It's usually individuals that come up with the important tools and inventions. Those development are very significant because they make systemic advances.
"There's an infinitesimal chance that the tool you're writing instead of your book will change the world" - it doesn't matter how small the odds are for any one individual.
There's two points here. One is that the effects can be cumulative -- even small advances and bits of knowledge gained contribute, over a longer period of time, to large benefits.
The other is that the historical reality is that tools that were developed as side effects of other goals have made a very significant overall effect, and this can't be denied by a blinkered focus on the probabilities concerning individual cases.
And the point is that troff is actually an excellent tool with very beautiful typesetting. The fact that Knuth didn't consider it good enough says a lot about him and his ambitious level of exigence.
Actually, troff was simply not an option for Knuth, for least a couple of reasons.
For one thing, at the time, Unix and all its associated tools were really not prevalent outside of Bell Labs and a few places — troff would not have been available to run on the computer systems Knuth had available to him in 1977. (The ones at SAIL, which were PDP-10 machines running either TOPS-10 or WAITS, I forget what I read.) He didn't have the source code of troff either; all he had was published descriptions: and those he did use, e.g. the paper by Kernighan and Cherry describing `eqn` came out in 1975 and he used ideas from it while designing TeX's syntax for math.
For another thing, troff was designed to control line printers and other such devices that were typically attached to computers; it simply wasn't designed to drive a digital typesetter of the kind Knuth was thinking of (Alphatype CRS). So he'd need a new program anyway. In fact, he says that until he saw Patrick Winston's book on AI in 1977 (which was one of the early books produced on a digital typesetter, i.e. where the actual technology for getting ink onto the page was based on 0s and 1s, instead of being based on either pieces of lead (hot metal typesetting) or light and lenses (phototypesetting)), he had simply not associated output from computers with “real books”. Things like troff were designed to control the simplistic devices that came with computers, that could move by discrete (large!) amounts on the paper (e.g. some could move by full “lines” only), and strike down specific symbols (from a limited font repertoire that came with the device) onto the page. This wouldn't have seemed relevant to Knuth, who primarily wanted to create his own digital fonts (as he did with METAFONT), and TeX was among other things simply a way to use those fonts.
So, regardless of whether one considers troff to have very beautiful typesetting or not, it simply wouldn't have seemed applicable to the problem at the time, to be rejected as "not good enough" or not. Now of course with output going to generic page-description formats like PDF (even TeX's DVI was not Knuth's idea—he was thinking TeX would just write code for the device directly—and he had to be convinced to do it by Fuchs: https://tug.org/interviews/fuchs.html), and high-resolution printers (fairly high DPI, even if not 5000+ as with the CRS then) available to typical consumers, we don't appreciate all this. :-)
I think your main point is correct, that phototypesetting at the time was really bad and that's what drove Knuth to write the tools. (He had been happy to publish with Addison Wesley because of the quality of their typesetting, and now if they were going to move to such an inferior process, it was very upsetting.) The details are off though: phototypesetting is precisely not digital typesetting — in phototypesetting, the decision of where ink goes on the page is controlled via light and lenses and a fixed set of fonts that have been made (physically) for them; it is not controlled by arbitrary 0s and 1s, which was the appeal of digital typesetting. About troff, see my other comment in this thread.
Donald Knuth seems to have got it right. On the way to writing a book, he wrote all the tooling including a typesetting system, a programming paradigm, a font, a font authoring tool, and so on.
I do think obsessing about trivial things is harmful, but finding an editor you really like working with is very valuable. I was exposed to vim and had a really hard time working with it, but I saw the value that lay beneath the arcane surface. I knew how much of a pain it was for me to copy/paste/rename/switch between windows when editing; it took me out of my flow. Becoming minimally proficient (I don't use any of vim's really advanced features) was a huge boost in comfort, translating to increased productivity for my attention-deficit brain. This feeling was compounded even more when I started using a tiling window manager: now my hands never needed to leave my keyboard during a programming session unless I was looking something up (I played around with vimperator or the like for a while, never 100% clicked for me).
Of course, that's the ideal. I can't use my preferred setup when I'm working on our legacy Windows codebase. I also won't pretend that I'm immune to chasing the new shiny thing to distract me from doing real work, it happens with some frequency. I just think that editors (and perhaps a good tiling window manager) are the exception(s) to the rule.
I am glad that you brought this up. I tend to spend a good chunk of time cleaning up and optimizing my .emacs right before I start coding. Absolutely useless waste of time!
"What kept me from [writing my own tool] was a calm voice in my head telling me that I’m here to write a book, not a preprocessor. [...] Now I have written two books and zero tools, which I consider a success."