From a lifetime of coding I indeed came to a similar conclusion: "Micro-abstractions" are more useful in the longer run than large-scale abstractions. Nobody can predict the future accurately: I've seen many Grand Abstractions byte the dust face first. BUT, with micro-abstractions you can often re-use a decent portion of them when the future doesn't go as planned. Unlike large abstractions, you don't have to marry micro-abstractions: you just date them. If any one doesn't work out, you can ignore and replace it without much penalty: no messy divorce. I've built up a library of "handy" functions and mini-classes that I can borrow as needed per project. I've translated the commonly re-used ones into different programming languages.
Further, one can experiment with different micro-abstractions without investing too much time. They are micro-experiments. If one fails to provide re-use or flexibility, so be it: just stop using it. It's like picking stocks: some stocks will wilt, but if you buy enough, you'll roughly approximate the general market at about 10% annual growth.
I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out. I just haven't cracked that nut yet. Maybe smarter people can.
Beautifully put. It resonated with me about not over abstracting too early.
Abstractions usually come earlier in the design cycle than optimization for me and the wrong one makes that later stage harder.
> I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out.
I go back and forth. I mostly develop in a mix of Python and Java. Python may give me a lot of room to explore ideas, but I find refactoring a fragile nightmare. Java's type system is fairly simple and lets me quickly pull up all use sites and trivaly catches misspellings.
It lets me back out of and refector the wrong abstraction much more simply.
I don't know why there are not more code analysis and refactoring engines for dynamic languages. Sure, it takes more guessing on part of the tool, but I see no reason why they can't identify and help with a large portion of such tasks. It shouldn't be an all-or-nothing thing. Sure, the more "meta" your programming, the harder it is to automate analysis, but only a portion of an application typically requires high meta.
Because the refactorings are generally impossible. At best the automated tool could tell you everywhere it thinks a change might be necessary, but has to ask you want you want to do there. And it would likely greatly over or underestimate the number of changes needed. It's rather akin to saying "why can't the system implement my functions for based on their signatures?"
Yes, and ORM's are often a painful failure point when they don't work as expected unless you really master their guts. In my opinion it's best to code your ORM queries such that if the ORM is not doing something right, you leave your options open such as using direct SQL, stored procedures, or views. In other words, don't overly rely on ORM's both for the total application stack and individual queries. Have and test a Plan B.
I've built up a series of "SQL helper" functions to do a lot of the SQL syntactic and Prepared Statement grunt-work to avoid ORM's. They are NOT intended to entirely wrap or hide the RDBMS, but rather help with the app's interaction with the RDBMS. Helpers rather than wrappers. If the helpers don't work in a particular case, you can ignore them with minimal code rework. If you depend heavily on an ORM, such is often not the case.
On the more general point, I think that the larger scale the abstraction is, the larger the number of actually encountered use cases needs to be. You shouldn't write an ORM because you feel like existing ones don't serve your specific needs. You should write SQL, and then replace it with an ORM once it's clear you will benefit from the extra abstraction.
There seems to be a fairly consistent trade-off between the scale of the abstraction, the degree of over-engineering, and the leakiness of the abstraction. So if you're going for a large scale abstraction, then I'd want lots of proof that it's not a leaky one.
I'm kind of bothered by the choice of direct SQL versus ORM. I'd prefer something in between that assists with SQL but doesn't try to completely hide you from the RDBMS: helper API's. I know many developers "hate" the RDBMS (perhaps because it's unfamiliar), but shifting the burden from potentially tricky RDBMS's to potentially tricky ORM's is not problem solving, just problem shifting. But maybe the ORM debate is another topic in itself.
I think I phrased my opinion badly. I'm generally in favor of using ORM's because they simplify 99% of the stuff you need to do in the database. There will always be stuff that doesn't fit your choice of abstraction well... that's just a fact of life.
I'm mostly wailing against writing your own ORM (and also other forms of "not invented here" syndrome), which is something I've seen done way, way too many times, because you have to reinvent a ton of things you probably don't have much experience with, will probably get wrong in ways you don't anticipate, and will miss scaling requirements you don't know about yet. Because of this, I think there should be a heavy burden of experience on writing "core" libraries.
ORM's "simplify" things until something goes wrong, then they can be a pain to troubleshoot. A "light" set of utilities can also simplify a good portion of the database interaction. However, they don't give you a false sense of power such that you tie yourself into a knot that bites you 3 years down the road. But, it depends on the shop: If your shop always has a couple of good ORM experts around, then they can deal with and/or avoid such snafus as they happen. But, it's hard to "casually" use ORM's right.
I suspect our experiences differ, or maybe it's our philosophies. I'm of the opinion that if your self-tied knot bites you 3 years down the road, that's better than the status quo. :P
That's situational, of course. My current projects are tech giant things, so some choices actually do have a >3 year window, but then there's a tech giant sized stack of use cases to design against.
When my projects were startup problems though, I'd be happy if that component still worked well after 3 years of growing and pivoting. In that kind of environment, getting more for free is by itself valuable for your ability to pivot quickly.
The environment does indeed matter. With startups, the short term is critical: if you don't grow fast enough, there will be NO "3 years from now" to fix. A big bank will care about glitches 3 years from now.
I'm not sure what you mean in the middle paragraph, per "stack of use cases to design against". You mean getting it done "now" overrides problems 3 years from now? Your org making that trade-off is fine as long as it's clear to everyone and perhaps documented.
We start to see ORMs like Diesel (https://diesel.rs) or GRDB (http://github.com/groue/GRDB.swift) that radically simplify their domain by removing traditional ORM features like implicit uniquing, lazy-loading, or auto-updates. This of course gives more work to do on the client side. But this work is positive and often simpler than expected. With traditional ORMs, there is still some work to do, which often looks like fights: against misunderstandings, against leaky abstractions, against configuration levels that don't match the needs of application, and generally speaking against the framework.
> I do find dynamic languages make such mixing and matching easier
Yes! I think all of the people I've met who advocate the article's position were using static languages that make it harder to re-use what they've written. This article sounds like it's about a C++ project, which doesn't surprise me.
Where are all the articles about this happening to dynamic data-driven projects? In contrast, people seem to go to incredible lengths to keep old Lisp code running. A lot of people (even non-Lispers!) would apparently rather write a new Lisp interpreter than port their Lisp code to another language -- because the Lisp code is largely a declarative description of the problem and solution.
Data itself is already a Grand Abstraction that has survived the test of time. I don't need a library of mini-classes.
The boundary between data and behavior is fuzzy, and one can often be remapped into the other. A common example is a "field", such as a screen field. We can represent it as a structure similar to: field(dbName="socSecNo", screenName="Social Security No.", type="string", mask="999-99-9999", required=true...); Whether that's a method call or "data" doesn't make much difference. Ideally we want to define it once, and that's our abstraction.
Lisp does make "swapping" between data and behavior easier, but many say Lisp code is just hard to read in practice. I've been in Yuuuge debates about Lisp, but my summary conclusion is that many just find existing Lisp code hard to read. Perhaps its too much abstraction. Short of eugenics, we are stuck with that problem. Lisp has had 50+ years to try mainstreaming; time to give up.
This resonates with me. For my long-running Python projects I usually have a file called small.py which has these, as you put it, micro-abstractions. Over time some stuff from the file gets into the larger projects ... and some stuff doesn't. But I like the comfort of "cognitive offloading" this file affords me: I don't have to keep worrying about increasing the scope of my abstractions.
I'll pick a couple that are easy to describe. First a simple string function: "softEquals(a,b)" that compares ignoring case and leading and trailing white-space, similar to "return trim(upper(a))==trim(upper(b)). The second is to "draw" an HTML button and/or hyperlink because often the user or designer change their mind about whether it's a link, button, or icon: link(title, url, httpType=[hyperlink,submit,get,reset], dispalyType=[hyperlink,button,icon],target="",classes="",otherAttribs="",buttonImg=""); Those with "=" are optional named parameters or optional object values, depending on the language of implementation. I'll usually tune it to app or shop conventions, which can shorten the range of variations or likely defaults, and thus simplify the interface. But the implementation is mostly the same: just the interface needs trimming/tuning. Or just leave the fuller one intact for edge cases, but create short-cut versions better fitting local conventions.
Further, one can experiment with different micro-abstractions without investing too much time. They are micro-experiments. If one fails to provide re-use or flexibility, so be it: just stop using it. It's like picking stocks: some stocks will wilt, but if you buy enough, you'll roughly approximate the general market at about 10% annual growth.
I do find dynamic languages make such mixing and matching easier, though, because they are less hard-wired to a specific type or iteration interface: less converting needed to re-glue. I've yet to get as much re-use in type-heavy languages, but not ruling it out. I just haven't cracked that nut yet. Maybe smarter people can.