I struggle with this constantly. I think there are two problems:
1. I like interesting puzzles. A lot of code - especially commercial code - is pretty boring if you do it right. I find myself subconsciously pushing for features that will be fun to implement. And by "fun", I mean, features that will overcomplicate everything.
2. While I'm in the middle of programming something, all the choices that I make seem straightforward and necessary. Its only later when I step away from my code and then try to understand it with fresh eyes do I notice what a mess I've made of everything.
I also think that a lot of the time the most obvious solution to most problems is quite complex. It takes wisdom, skill and domain knowledge to know where to look for simple solutions to any given problem. Simple, clean solutions are rarely obvious.
"Oh, it looks like we're slowly implementing a halfbaked message queue. Lets use a standard message queue instead." "Oh, we're slowly building up a custom, buggy, binary framing protocol. Lets just use protobuf/msgpack". "How about instead of writing a custom RPC protocol to fetch data, we just use REST over HTTP. And then we can put nginx in the middle to cache our backend responses, and we can throw out our custom cache."
This is it. There's real skill in creating simple solutions to complex problems. Knowing the general landscape of what's out there, and easily available off the shelf, really does help.
Developers grow. It starts with simple code that doesn't work. The next step is, complicated code that solves the problem, in messy unmaintainable ways. The next step is writing super clean, almost boring code, that's highly readable and "dumb" and does exactly what it's supposed to do.
The other thing to realize, a lot of great code doesn't just spring from the developer's hands in its final form -- it's extensively edited and rewritten into its final, good, form.
> There's real skill in creating simple solutions to complex problems.
Not entirely facetiously, I think that, for engineers, there's real skill in creating simple solutions to simple problems—not, for example, finding the general instance of the problem and solving that, when the problem is unlikely to recur and crafting the perfect general solution delays delivery on what's actually in front of you.
(I know Perl's not fashionable any more, but I've always liked its design philosophy of "make easy things easy, and hard things possible." It seems like a slogan that can be adaptable to how to solve problems, though I'm not sure of the absolutely perfect analogue. Hmm, maybe I'm trying to solve the general instance of a problem ….)
I actually think this is the opposite of the case, for some definition of “generic”: the more generic problem has fewer possible solutions (there is only one pure function of type x=>x) so, if you hit on the right general problem to solve, your code will almost always be simpler. The problem is this is one of those “$1 to solve the problem/$99 to know which problem to solve” situations.
> I actually think this is the opposite of the case, for some definition of “generic”: the more generic problem has fewer possible solutions (there is only one pure function of type x=>x) so, if you hit on the right general problem to solve, your code will almost always be simpler. The problem is this is one of those “$1 to solve the problem/$99 to know which problem to solve” situations.
This is true, but I think also illustrates the phenomenon. It's tempting not to solve the specific problem in front of you, because a more general problem might be easier and more elegant to solve—and this mindset can easily lead one into, at worst, never solving the real problem; or, at less worst, solving a problem that's so far generalized that no one else looking at your code can tell why it's doing what it's doing.
(It can also happen that the more general problem doesn't have a simpler solution. If I want to print a string that has a few hard-coded values in it, formatted a particular way, I could develop a formatting spec and write a formatting library to process it, which is surely the right solution to the general problem—but, if the specific problem is likely only going to arise once, then it may be both easier to understand and a better use of time just to put in the hard-coded values.)
> "Oh, we're slowly building up a custom, buggy, binary framing protocol. Lets just use protobuf/msgpack"
By using protobuf/msgpack you lose the ability to precisely control the layout and encoding of your data on the wire. Most applications don't care, but this results in your wire representation being defined by "whatever protobuf says".
Say I want to transmit an unsigned 16 bit integer with protobuf. How do I do that? The documentation doesn't include 16 bit integers as a datatype, so I'd probably have to wrap it in 32 bits and/or use some varint stuff. It would be simpler to just write a big endian 16 bit int though.
I wish there was a simpler alternative to protobuf that gives more control to end users and doesn't try to be smart. Until then, making your own binary protocol is not over-engineering.
It might be different if you have to talk to an ASIC that cannot understand protobuf, or send billions of values at line speed. But generally I don’t have to care anymore whether a number is sent in exactly sixteen bits, for the same reason I long since stopped caring about message framing or parity or run-length limits or 0-5 vs ±12 V busses. Expressing any of those constraints takes more effort than letting the machine use the commonly-supported default.
It wouldn't be ad hoc per se, basically you would have a set of guidelines on how to transmit data and that by itself would be a standard.
Something like "use fixed length 8/16/32/64 bit signed/unsigned integers in big endian, length prefix can be 8/16/32 bits, bool is 1 byte (00 = false, 01 = true)" etc, without extra stuff like varints or bit packing, which a lot of current formats are doing.
In short, just use the most straightforward way of encoding while also using the least amount of data. Big endian for ints is very common, simple and relatively compact if you only use the bit width that you need.
I agree; sometimes writing your own binary format is the right call. To my point upthread, the trick is knowing when that’s the right choice and when it’s better to use protobuf or something standard. (Or, when to just stick to json).
Developing good instincts for this stuff takes a lifetime.
> 1. I like interesting puzzles. A lot of code - especially commercial code - is pretty boring if you do it right. I find myself subconsciously pushing for features that will be fun to implement. And by "fun", I mean, features that will overcomplicate everything.
1. I like interesting puzzles. A lot of code - especially commercial code - is pretty boring if you do it right. I find myself subconsciously pushing for features that will be fun to implement. And by "fun", I mean, features that will overcomplicate everything.
2. While I'm in the middle of programming something, all the choices that I make seem straightforward and necessary. Its only later when I step away from my code and then try to understand it with fresh eyes do I notice what a mess I've made of everything.
I also think that a lot of the time the most obvious solution to most problems is quite complex. It takes wisdom, skill and domain knowledge to know where to look for simple solutions to any given problem. Simple, clean solutions are rarely obvious.
"Oh, it looks like we're slowly implementing a halfbaked message queue. Lets use a standard message queue instead." "Oh, we're slowly building up a custom, buggy, binary framing protocol. Lets just use protobuf/msgpack". "How about instead of writing a custom RPC protocol to fetch data, we just use REST over HTTP. And then we can put nginx in the middle to cache our backend responses, and we can throw out our custom cache."