Ten years after that I was thinking "great, I've spent 10 years validating email addresses on contact us forms" (the reality was quite different, of course, but I was feeling like I'd lost the thread). But then I started doing a little more with ecommerce and got good at mentoring people and so on.

Now, it's ten years after that. I am managing a bunch of programmers and still write a little code. And I'm wondering how I will stay relevant for the next twenty or so years but so far, so good.

And I keep learning more and more. Not about compilers and Docker (this does come up, I guess) but about projects and people, business, domain knowledge about the area we build software for, and so on.

On paper, yes, I'm nothing. Just some average dude. But I really like my job, my team, and my skills.

I'm not saying this will happen to you but what I am saying is you shouldn't feel inadequate because you can't build a C compiler in assembly or something. That's the kind of thing only someone truly, truly interested would do. It's not something you do because you should or because it will make you a "real programmer".

Find an industry or business that fascinates you and figure out what they need to see to hire you and go for that.

I came upon this book/course: https://mitpress.mit.edu/books/elements-computing-systems-se...

From a Hacker News post IIRC, and it helped more than I can express. At least for me it filled in all the little gaps in the computing stack and really made me "get it". I could relate to the old timers complaining about how everything is so bloated now.

The course is very hands on and you build a "full stack" from logic gates to an operating system that can run programs. It's truly amazing, and I think I learned more from this book than most of my college courses.

I believe it's also offered for free on coursera and has a website: https://www.nand2tetris.org/

But if you really want to know how a full computer works from the ground up, I think this book would be well worth the time.

It's very easy to get stuck in the world of CRUD because it pays the bills, but as you noted there are a lot of technologies.

Shifting out of that world requires risk tolerance because there will be failure after failure. That wall you speak of is the nature of hard things and the time you can dedicate bashing your head against that wall.

My advice for climbing/breaking the wall:

Pick a small project that you find interesting: build an OS, game, programming language, VM, etc... and just do it. The key is to cross the hump of understanding and get something simple working, and the good news is that conceptual understanding isn't that far away. The tool to help you get unblock is writing. Write the comments of what you are trying to do, and break it down until it becomes code-like. Read books on the subject and bounce back to writing what you read.

However, going from conceptual understanding to professional environment is different because the concepts start to inter-relate and the complexity manifests.

Prior writings on this:

http://jeffrey.io/writings/thoughts-for-young-hackers-and-so...

http://jeffrey.io/writings/build-first-buy-second.html

Most of the world, however, does not work this way. As an example, take Restbucks (https://www.infoq.com/articles/webber-rest-workflow/). You can see how there's a web component to this, though it's much more complicated than your normal CRUD view. I've discovered new challenges in dealing with these problems ranging from building a rich domain to handling side effects.

Guess what? I'm in the older generation, I have a CS degree from the late 1980s. I'm comfortable with hardware, compilers, run-times, drivers, operating systems, etc. but all that stuff you listed is like a mystery. I don't know what it is, why or when I would use it, etc other than at a very high level.

I mostly write CRUD apps.

That's where the volume is. There's a big demand for programmers to do web stuff. That demand may or may not continue. It's so stylized that it should have been automated by now. Have an exit strategy.

There are many things in computing that are hard and more interesting, but they tend to be low-volume. From self-driving cars to industrial control, there are many areas of specialization. But not many people do those things. You can get stuck in a niche where there are no jobs near you.

If you're stuck doing web apps, at least learn to do them in Go. The performance is pretty good and there are fewer layers.

Someone recommended Rust. I write a lot of Rust, but for non-web stuff. It's useful for things where you'd otherwise have to suffer through debugging C++. Don't write web apps in Rust; it's the wrong tool for the job.

A user-level understanding of machine learning is useful. At least learn how to start up TensorFlow.

How compilers work, and especially how they optimize, is fascinating, but there's not much demand for people to do that. Distributed database internals are similar. As is networking. These areas are very important, but most of the big problems have been solved and they don't need many people.

Operating systems are also interesting, but things seem to have settled down to either Linux or Windows and mods to them.

The blockchain/distributed finance/cryptocurrency world has a sizable following, but it's mostly the make-money-fast crowd, some of whom are crooks and some of whom are suckers. Be cautious.

This is highly variable. The more rote aspects of web development are indeed automatable: the parts that feel like repeating boilerplate because that's what they are. Creating duplicative controller actions, state updates, etc.

But then there are the parts that would be extremely difficult to automate. Creating rich, unique UI elements with lots of custom logic. Anticipating the user's needs before they realize they even have them. These take a lot of skill and experience, and are truly fun to work on – assuming they're your jam. Lucky for me, they are.

I thought that's what coding bootcamps were for?

Regarding VMs and OSes:

Since it seems that you have been working as a web developer, I think a soft transition for you would be to get into modern devops practices:

- start with vagrant to spin up a linux virtual machine and start playing with it. E.g., how to setup ssh keys, how to install packages in an automated an idempotent way (e.g., using Ansible), etc.

- while you are dealing with virtual machines, you'll learn also a bit about networking (e.g., what's a private network? what's NAT?)

- when you know enough you could start using Terraform to spin up virtual machines in the cloud. There's some magic the moment you run one command and you have a linux machine up and running in some cloud provider! Even better: you run another command and the machine is totally destroyed :)

In such a journey you'll be exposed to different and potential new concepts for you: vpcs, virtualization, a bit of security, firewall rules, ssh, tcp/ip, etc.

After around 6 years of crud I decided to try something different, and went to yandex (it is like google, but in Russia). First time it was just the same crud (lot of rest api), but after some time I managed to land on some infra projects - kafka, streams, microservices. And it became much much more fun. You really build stuff from the scratch, you design it, test it under high load. Anyway my tips:

1. Don't go for buzz word, find job where you can actually choose technologies. Soon you will understand how to see through the hype. And cost of choosing wrong tech.

2. Stop trying to learn what 'everyone think is cool'. Learning compilers won't make you better developer, esp if you are not into it. And Dragon book is not the best intro to compilers anyway. If you want to learn something fun, find what it is fun for you in the first place.

3. If you want to become better developer, try to improve things you do at your day job. Not something like 'we should use k8s or we die'. More like 'we will have to make a lot of these crud in near future, lets spend some time to build framework that makes building and support them 2x faster and cheaper'. Look at small and achievable improvements first.

4. Invest time to understand how things work. And build your own opinion about technological decisions.

Yes, no. Do this and then leave the company. Then I or some other poor sucker gets to maintain your framework that makes you 2x faster but everybody else 2x slower. Yay fun.

Build a framework only if its reeeeeaaaaallly necessary. Whis is most likely never. If you feel the urge to write one, by all means build one for your own side projects, but dont shove it down everybody else's throat

I really don’t think most modern CS programs are glorified bootcamps. Not sure where you’re getting that from.

Some of the worst offenders were:

* A graduate-level parallel computing class taught in sync with a 1000-person online class aimed at OMCS students. For the level of depth we went into, I may as well have just taken a Coursera course.

* A PhD seminar on compilers, led by a tenured professor who was reusing slides from 2002 and seemed not to know anything about modern compiler theory or practice. "Huh? What do you mean a 'lattice'?" he said. Thankfully his TA cared enough to develop some projects around LLVM, which is the only useful bit of knowledge I took away from the course.

* A natural language processing course with a severely outdated bibliography, where the projects required us to make changes to convoluted "starter code" riddled with bugs. Half of the final exam was manually computing the probability table (by hand, on paper, without a calculator) for a quite complicated probabilistic context-free grammar.

* A machine learning course where the lecture slides were written by the TA, and where the professor was often seeing the slides for the very first time five minutes before lecture.

And that's just the courses I was able to take.

* For my entire four years as an undergraduate at a large public university, there were no classes offered on the topics of programming language theory or computer graphics. They simply couldn't attract research faculty in these areas, and instead of hiring a lecturer, they thought it better not to offer the subjects at all.

* When I became a graduate student at another large public university, I was excited at the chance to finally take some PL + graphics courses. But nope, during my first year, all of the PL + graphics faculty left to join other universities / industry.

Everything newer you will learn on the job.

I think these days, a proper compilers course should also be a comparative study of programming languages. There is a lot to learn from comparing how C++ vs Python vs Haskell go from text -> machine code. There are both practical and theoretical aspects to this, and I would expect a graduate-level research-focused course to cover the more theoretical side.

This may be true for graduate programs, but I wholeheartedly believe that undergrad Comp Sci programs should be completed at a 4-year school mostly because the course work is less specialized, much easier to teach and generally attracts pretty decent pool of instructors at most schools.

Additionally, bootcamps are not online courses and shouldn't be conflated. A 6 to 9-months bootcamp goes through material at such a high pace that it often becomes counter-productive. I have a friend who paid $10K for a 6-months bootcamp, got burned out and learned very little in the end. The entire experience basically turned him away. I have another friend who started Comp Sci program at a community college and he's been absolutely happy with overall experience and pace. I have helped both of them and would generally recommend staying the hell away from bootcamps, especially with no prior programming experience.

Teaches you how computers, languages and operating systems work from basic principles

* Little Schemer, Seasoned Schemer

Teaches you about the power of functions, recursion, various CS topics and lisp's syntax

* K&R C Programming Language

Teaches you about lower-level programming

I also got a lot out of Bob Nystrom's "Crafting Interpreters" book

My personal path hasnt shown that to be true, but maybe I’m the odd duck.

Initially I built a simple crud app to track water changes for my aquarium. For a while there I had it online as a free SaaS app too. It was using Ruby on Rails, and I had no idea how to use JavaScript to make the UX better.

Fast forward a few years and I was getting fed up with voicemail, so I read up on APIs and figured out how to use GCPs transcription service, pointing to audio files in twilio (I diverted my mobile to twilio to act as my voicemail). Actually pretty simple even though it might sound complicated.

Fast forward to last year and I built a free directory to raise the profile of primary producers who were missing out on foot traffic as a result of COVID, closely followed by another project I’m close to launching. These last two projects were in nodejs as I figured I had better finally figure out JavaScript.

When I was completely stuck on something I did a lot of googleing and watching YouTube - that happened a lot, particularly when trying to get my head around promises.

Moral of the story, don’t be put off with CRUD apps, they are very rewarding. Just build stuff you enjoy when you have spare time (for me evenings after child in bed).

Depending on your field / industry, having good knowledge of accounting is magnitude times better than compiler knowledge.

Learning about optimization (both db and app), write good / clean code, debugging are useful skills wherever you go.

Unless you want to change course, it's perfectly fine.

Disagree. Most JS frameworks require toolchains these days, particularly in the form of Babel (parsing/compilation) and Webpack (linking). If you understand the way compilers and linkers work, then these web tools are more useful.

Sadly, a lot of web devs just think it's too complicated and so copypasta their configs without understanding, but lots of frustration.

I'm interested with it. Got any examples at what can they be used further if you know compilers and linkers? So far without understanding it I can use it for compiling react.

However I didn't said that those skills aren't useful. They just aren't requirement for CRUD / web devs. And I'd argue that learning business side is far more useful than technical side (after you've learnt up to certain level)

This is an excellent perspective, and a recommendation for OP I hadn't thought of. Learning how accounting or finance works, how sales works and then how to think strategically about what each faction involved in a business is interested in (including customers and various types of employees) can be very helpful. Appreciating and starting to understand design can also be helpful.

> I'm interested with it. Got any examples at what can they be used further if you know compilers and linkers? So far without understanding it I can use it for compiling react.

I'd take a look at the sources on GitHub at first. Maybe clone them, set breakpoints and start debugging.

In React, the compiler (based on Babel) is going to translate JSX into (virtual) DOM operations, plus handle some of the JS features that aren't supported in the vanilla browser implementations. Webpack is going to read in these translated sources as well as the React runtime (which has the logic handling virtual DOM operations) and any other assets you use (images, CSS, etc.) as represented in JavaScript. With everything in the same namespace, Webpack can then begin "linking" the "objects" and resolving references. It will then begin writing them all out as you specify in the webpack config.

This is highly analogous to e.g. `gcc` (C compiler) and `ld` (linker). The compiler takes sources and translates them to something the machine understands, the "object" files. But the references to anything outside of each object file are all broken. The linker takes all of the object files and resolves the references, including to runtime dependencies like "shared libraries"/DLLs, and then it organizes the final outputs in a way the programmer specifies (usually there's a default that just works, but for some types of outputs, you write "linker script").

Also like with webpack, there are all sorts of kind of lousy tools that aim to automate the creation of the build scripts, like Automake or CMake. These tools do save time in most cases, but they are leaky abstractions that are quite complex to understand vs. the relatively simple (though also difficult) compiler and linker, which are fairly easy to reason about with some experience. React-scripts is I guess the latest analogous tool I've seen that tries to hide away webpack, though I've also seen create-react-app and razzle.

The beauty is, if you understand how people were dealing with some of these issues in compiling C or FORTRAN or whatever 50 years ago, it's easier to see what the pitfalls and strengths of the current web toolchains, despite the complete dissimilarity on the surface. The patterns are quite similar.

EDIT: it might be useful to find a tutorial you like on regex and parsing. (Strict) regex are equivalent to "deterministic finite automata," which are conceptual, abstract virtual machines that transition between states based on inputs. Parsers are more related to a class of automata called "pushdown" automata (though there's a lot more variation, and parsers are not actually implemented as PDAs despite some loose equivalence). Understanding the hierarchy of automata (DFAs up to Turing machines and then the purely imaginary ones like "oracles"), as well as the distinction between deterministic and non-deterministic and how those can be made equivalent (for some trade-off in real world efficiency) is also useful. It might also be good to study and explore BNF-type grammars and parser generators. Some of these exist in JavaScript, like Ohm. Ohm is particularly useful because it has a visual browser-based editor that explains exactly how a parse is being performed given some test input and grammar. Once you kind of understand that, you can experiment with other kinds of parsers (Earley, LALR, recursive descent, etc.).

Writing a small calculator from scratch or building your own JSON parser could be useful tools.

I understand the 'mastery' bit is less about whether you can do it (clearly you rate yourself well at that) so much as the feeling that you're improving at it.

"just CRUD apps" suggests that you're not feeling you're mastering it, and that you don't feel like it's the best fit for what you're interested in doing.

I think any professional task you can competently accomplish without having to learn anything is going to be boring. I think e.g. Scala, K8s are complicated enough that there's room for learning more about it. Not necessarily "do hard stuff for the sake of being interesting", so much as "being able to evaluate whether this is the best tool for the job".

One thing to consider is that the older generation has simply been at it for longer than you have, and this is an industry where the vast majority of experience is gained on the job.

Another consideration is that due to a number of timing factors, a certain demographic (the elder Millenial/younger Gen X cohort) came of age at a time when computers were just recently cheap enough to become a mass-market good and viable for home use, but before they were commoditised and standardised, and before there was an internet full of helpful information, so if you wanted to learn how they work under the hood, you needed to get your hands dirty.

We are at the stage of lacking a lot of software help. We have an assembler, but that's it.

We're expecting some initial chips back in a few months from using the Google/Skywater/Efabless shuttle program: https://efabless.com/open_shuttle_program

Currently we are simulating the CPU in verilog simulators, as well as fpga's. There's a lot of fun very low-level work to be done.

I'd love to help.

Do you need someone to write embedded tools? I am getting into FPGA for RISCV with the Arty board and was wondering if you need any help regarding this.

Of course, it's not for everyone, but I'd say it's worth giving it a shot. Definitely start with C/C++ if you haven't already, and try not to stay with existing game engines like Unreal or Unity for too long (unless you're specifically trying to start a career in game development; a bad idea)

Have you tried joining one of these projects you mentioned as open source contributor? I know at least some of them are always happy to get more people to join and help. And not just with code contributions. They also have (CRUD) websites that need maintenance, docs to be written, events to be organized, and training/marketing material to be produced. You might be surprised, how these seemingly smarter people end up being super happy about your contribution, as usually what is easy for you will be hard for them, just as you might see it the other way around.

And because you said you have wife+kids, your time might be limited: there are also ways to get paid to participate, at least around the bigger projects.

A few things have helped me. The first is that you need to embrace what you do and love what you do. It doesn't sound like you hate being a web developer, more that you feel you should be doing "harder" things. Web development is far more than CRUD apps, so it might be worth looking for another job that will challenge you more as a web developer.

The second thing is that you need to understand what you do _deeply_ instead of _broadly_. I have for a long time considered myself a generalist, but this has its drawbacks. When you really understand a tool set deeply and become an expert a lot of the lower level internals really reveal themselves. You end up learning things that you've always thought you wanted to learn in isolation, but instead there is now a useful context in which to learn them. Not to mention that this helps with the feeling of being "not so smart" because you will form informed opinions as a result of your deep level of understanding.

The last thing is that you do need to feed your curiosity and the only way to do that is through projects. You want to build a compiler, build an emulator, a game engine, a database, or an OS? You really just need to do it. I have dabbled in each one of these things. Some of which I got very deep into, others not so deep. Each time I learned a little bit more. The hardest part is picking a project and running with it, especially when there are so many different options, but it is the only way.

Sometimes it's important to take a step back and appreciate what you do. It's very easy to beat yourself up and compare yourself to others. If you really are unhappy with what you do, then you need to start deeply diving into something that you do find interesting. Nothing in this field is easy and it all takes time to gain any level of proficiency at.

[0]: https://www.indiehackers.com/

[1]: https://microconf.com/

[2]: https://www.startupschool.org/

(Note: There is nothing wrong with vocational schools for those who need to learn specific technologies. In fact, many people with CS degrees could benefit from courses in specific technologies. However, the purpose of a bachelor's degree in computer science is to teach students the fundamentals of computer science, and I believe undergraduates students are done a disservice when they are not taught the fundamentals.)

I highly recommend going through the book Structure and Interpretation of Computer Programs (https://mitpress.mit.edu/sites/default/files/sicp/full-text/...). This may be your first exposure to functional programming in Scheme. Even though it was originally designed as the textbook to MIT's introductory CS course (before switching to Python in the late 2000's), this book is useful for anyone interested in computer science, and it's easier to read for those who already know how to program. By the time you get through the end of this book, you will have written a Scheme interpreter, a logic programming interpreter that can handle unification (don't worry about what that means right now), and a Scheme compiler.

Once you finish that book, then it should be easier for your to delve deeper into the topics of your choosing. Want to learn more about computer architecture? I highly recommend getting the latest edition of Patterson and Hennessy's Computer Organization and Design (https://www.elsevier.com/books/computer-organization-and-des...). Want to learn about operating systems? I highly recommend Remzi and Andrea Arpaci-Dusseau's free Three Easy Pieces (https://pages.cs.wisc.edu/~remzi/OSTEP/). For compilers, you already have the dragon book, which should be easier to read once completing SICP.

- Take Nand2Tetris

- Take Casey Muratori's upcoming course StarCode Galaxy.

- Get some inspiration from Jonathan Blow and Casey Muratori on YouTube, and start the rabbit hole there.

- Be prepared to question a lot of the dogma that you have already absorbed.

Secondly, I'm not sure if you've spent time in different industries? While you may still be building what amounts to CRUD apps in different industries, your work can be very different and you can have very different learnings. For example, I worked in social games (i.e. FB) and mobile games a decade+ ago, for several years. Engineering was all about high traffic, high performance backends; squeezing every bit of speed possible, handling large amount of concurrents and DAU while keeping systems up. Over the next couple of jobs, eventually I landed in real estate transaction tech, which is almost a polar opposite -- very small amount of traffic relatively, but needing to be very robust and well-designed at every level from API to data store to schemas to ETL. Design patterns and concerns etc. are completely different; to me it's a whole different world of software engineering. I imagine if you move the opposite direction, you would see just as much drastic differences.

I would look for jobs in software companies who make stuff you want to learn to make, but for some parts of their products they also need these Docker/Scala/Kafka/etc. from your resume. If you are good at these things, they gonna pay you well i.e. enough to support a family. I think over time you’ll inevitably learn their lower-level stuff you have to integrate with. You can then change position within the company should you want to.

That's different though than "Computer Science in general," which is far more theoretical, mathematical, and logical. These things, while they can be learned by doing, are much easier to learn by being taught. Paying someone to teach you C/C++ isn't going to make you better at Computer Science in General, it will just teach you a new programming language, which is practical. Computer science in general are things like distributed systems, discrete math, algorithms, etc.

Nowadays the level of abstraction is higher (smartphones, FaaS, VMs, UI frameworks).

My tip to OP would be to set up a home lab on hardware from eBay. Run some services on it and manage it from the metal up.

Couldn't have done it without fooling around with my home server first. An early ASUS Eee PC in that case.

Not OP, but this sounds like an interesting thing to do. Which hardware would you recommend?

(Also, for the uninitiated, what exactly is a ‘home lab’?)

Home lab for me would be a place to build and run servers. I.e. basic tooling to assemble cases, change CPUs and RAM and a wired network connection. A community hacker space may work too and there will be folks to ask for advice.

You could also use micro PCs[1] or a Rasperry Pi, but those are too different from actual servers IMO. That's why I'd prefer a larger form factor to start with.

1: https://www.servethehome.com/introducing-project-tinyminimic...

If it's the latter than I can tell you with high confidence that you are not missing much there. Most companies need CRUD apps, simple Frontend apps, etc., and there is a high demand for engineers with the relevant skills and therefore also salaries are fine. Building OSes, compilers, embedded devices, networking stacks is super interesting, but there isn't that high of a demand since it's rarely what drives business. Therefore even if the topics are sometimes more complicated, they aren't payed that different.

Do you want to become a specialist or a generalist? You can dive into one or two of these topics and specialize the rest of your life into it. By the way: a specialist in any of these topics would probably not know any of the fancy buzzwords you've listed in your resume. As a matter of fact, I think you'd have more career opportunities instead.

Keep in mind that the older generation had a "few layers" to explore and could easily understand each slice until the bare metal. These days, it is much more broad and in-depth.

So you want to specialize in something, or become a generalist?

[1] https://nostarch.com/realmofracket.htm

To be honest, it sounds like youre disillusioned more with application development and not the particularities of the technologies you work with.

The older generation are in this place because back then none of the technologies you listed existed and so they had to know the more low level things.

A book that might help some of your knowledge gaps (at least directionally) is The Imposter's Handbook[1] by Rob Conery.

If you're specifically aiming to get away from CRUD take a look at the various Domain Driven Design books by Eric Evans[2] and others. This is the opposite direction to your ask though (more high level than low level).

If you want to go deeper in Java, read Effective Java[3] and Java Concurrency in Practice (JCIP)[4]

Lastly, if you want to experience the reasons why some of that early comp-sci stuff happened the way it did, playing with expensive hardware is the least effective way to find understanding of constraints. Instead consider grabbing devices with significant limitations like an Arduino, ESP32, or similar (I'm partial to the Wio Terminal[5] and M5Stack[6] as interesting easy to get started devices, but others would suggest various Arduinos or Raspberry Pi devices). Realistically you're going to learn much more by coming up with a self-challenging project and completing it than just by learning it for the sake of learning.

[1]: https://bigmachine.io/products/the-imposters-handbook/

[2]: https://www.amazon.com/Domain-Driven-Design-Tackling-Complex...

[3]: https://www.amazon.com/Effective-Java-Joshua-Bloch/dp/013468...

[4]: https://www.amazon.com/Java-Concurrency-Practice-CONCURRENCY...

[5]: https://www.seeedstudio.com/Wio-Terminal-p-4509.html

[6]: https://m5stack.com/

I would consider myself in older generation, having graduated around when Google was operating in a garage. I took assembly programming and played with circuit boards and logic gates in college. I don't have specialized knowledge of OS writing or compilers but I have enough from a CompSci degree (or maybe it was building gaming rigs in the days of dealing with IRQs) to know if I see issues in those areas. A lot of the concepts in computing haven't changed in ages, yet the buzzwords have.

I would only suggest diving into those areas deeply if you are looking for a job/career change. I say career because working on writing HW drivers or other things is almost a different career than writing business apps since the job opportunities will be different. I've run into folks that got "stuck" into deeply specific areas of computing and have no buzzwords on their resume. They got laid off and are having a hard time finding work.

If you've already determined that you can't do something because it's just too complicated, there's a strong chance you'll be proven right. In my experience, all of the things we software people deal with down to networking, hardware drivers, task scheduling, you name it, were created and explained by humans. Nature constrains us with time and space performance and quantum mechanics. If another human was smart enough to create this stuff ex nihilo, certainly with some work you can be smart enough to understand it.

> My Computer Science degree and the newer degrees do not focus on these, they have become 4 year long coding boot camps that focus on getting people job ready with AWS, React, Ruby or whatever trendy. This is true for the majority of the not so prestigious universities out there.

No one is going to give you a checklist for how to be creative or pursue your own goals. You have to do it. If you're capable of asking this on HN, I think you're capable of setting a few goals. Just write them down and put the deadline a long ways away.

> I see a pattern and I feel the older generation is way more capable and knowledgeable when it comes to Computer Science in general. I am pretty sure a lot of people are or were in the same boat as me. I don't really have the time(Have wife & kids) to take a compiler course or set out to build my own OS, what would be the easiest way to learn these lower level things?

I'm in my thirties. When I was in school, there were a lot of kids who were just punching the requirements and doing what they were told to get by. Any time they got stuck, they'd have to ask the professor or TA in office hours. Others of us learned that we could solve a lot of problems ourselves if we cut them down to their essences and wrote small programs to see what was actually going on. That heuristic was very helpful.

Once I really did get stuck after doing this, and went to office hours. My professor (an old school Unix graybeard) was so thankful that I had put in the leg work of examining the problem with "toy programs" as he called them that he gave me an exceptionally detailed explanation for what was going on, and how I could show this to myself. He made the comment that a lack of curiosity was a problem among my peers, but that it can be grown and cultivated like other skills.

One other thing the older generations did better than we do is look at the history of our field. If you don't know who Alan Kay, Doug Engelbart, Nikolaus Wirth, Alan Perlis, Ivan Sutherland, Leslie Lamport, Margaret Hamilton or Ted Nelson are, go find some information about them (most have talks or demos on YouTube). Then find out the rest of the missing pantheon of computing. Our industry goes through fads in sort of a fashion culture, and in my experience it leads people to feel burned out our stuck exactly as you described. There are some enduring truths which our field does a poor job of passing onto posterity. Turns out more people had interesting ideas beyond Dijkstra and Turing.

> 1. Reading the Dragon book(I find this would be useful only once you have built a bad compiler) 2. Paying people older than 40-45 to teach me some C/C++ and some tricks of the trade (expensive but I gained a huge amount of knowledge in a short time)

My advice: pick a small, interesting problem and tackle it for some set amount of time each day, maybe after the kids are in bed or your wife is enjoying a bubble bath or whatever. It doesn't have to be long - maybe a half hour to an hour. Just do it consistently.

I've found that GitHub, YouTube and HN itself have loads of interesting project ideas and tutorials. I find those much better than just reading a book. In those cases, the book becomes supplementary material, and is usually most useful at exploring theoretical issues, edge cases, history and other more detailed topics.

If you are going to read, always do the exercises. Also, keep in mind that a lot of the books recommended like the Dragon Book or SICP or whatever else are typically a very thorough treatment, but may not be great for a great read. (There are exceptions to this: I find Tanenbaum's OS books to be highly engaging, interesting and helpful. He is both a practitioner and a theoretician, so I find his perspective valuable.)

Not only is my understanding better with experience (rather than just reading), but I find that I feel accomplished, which is a virtuous cycle. And I also find that I learn enough to be able to discover other interesting things for myself.

In the specific area of programming languages, a good example might be to write a toy Forth interpreter.

https://zgrimshell.github.io/posts/interviews-with-floss-dev... https://bonedaddy.net/pabs3/about/#interviews https://wiki.debian.org/PaulWise/InterestingSoftware

This grab-bag of technologies is kind of all over the place. I see two possible paths forward (these aren't exclusive):

First path: everything you've listed is open source. Pick something, and start digging into the code to figure out what makes it tick. Try to fix a bug from the issue tracker, or review a PR. At some point you'll dig down far enough to encounter a tool or technology (also open source) you're unfamiliar with, and you can learn that and keep digging.

Lather, rinse, repeat.

I've found that most recommendations are to study things bottom up (eg. NAND to Tetris), but for many people it is very difficult to see the point of the lower levels, they are just too many layers of abstraction removed from anything they might find interesting. Top down learning tends to be self-directed, less formal, and ad-hoc, but the motivation for learning is much clearer (ie. you're digging down from the interesting thing made possible by the layer you're digging into), which tends to help a lot. Sure, you're always finding out that the abstractions you know are arbitrary, illusory, or even false in important ways, and in any case always hiding nasty messy truths underneath, but TBH that's how most real learning in our field happens anyway.

Second path, those technologies all have use-cases where they are (supposedly) useful. Start with a CRUD app, and try to figure out when one of those technologies makes sense to incorporate, and explore the edges of those use-cases, their failure modes, when it stops making sense to use them. Most CRUD apps in the wild are, well, pretty cruddy, which means that there is a ready market for skills to get them working again after they've broken down for some reason: too many concurrent users, too high volume of transactions, too large volume of data, too many monitoring alerts, backups take too long, transactions fail, specific queries always time out, inconsistent state, search autocomplete has started freezing the page, newly entered data takes 10m to become visible to users or to another app, hosting costs have started increasing non-linearly, a password reminder takes an hour to arrive in the user's inbox, and on and on.

When it comes right down to it, Twitter is basically a CRUD app. So are ebay, Wikipedia, Amazon, AirBnB, Yelp, Facebook, Reddit, LinkedIn, and so on. And yet not, as a Rails clone of these sites can quickly reveal once you start getting a few thousand users (real, or simulated). Sometimes solving a bottleneck can be done by swapping out a component or scaling horizontally or vertically, other scenarios require more extensive changes. Learning how to recognize and mitigate these inevitable growing pains can lead to some interesting places, and more interesting solutions and work.

Sure, CRUD apps as initially deployed, are boring. But they don't have to (and arguably can't) stay that way, especially as they evolve to encompass more users and data, not to mention functionality (eg. geodata, full-text search, stream processing, notifications, workflow, auditing, PII, security, etc.).

I hope this helps. Good luck on your journey!

What I don't understand from your post is what your end goal is? Why do you want to learn low-level stuff like building compilers or operating systems? Is it for work? Desire to improve yourself? What can help you better depends on the answer to this question.

I can offer a couple of pointers I hope they can help:

1. I was frustrated by my lack of understating of low-level concepts and how it prevented me from understanding the more complex pattern. The thing that clicked from me was building a small CPU and related hardware (timer, VGA frame-buffer, etc...), I don't remember which site/guide I follow, but today ben eater videos or nand2tetris seems good choices.

I also read a lot about game console programming (GameBoy color in particular), a lot of that low-level concepts were really a lot simpler to grasp with the visual demo to demonstrate. I'm also constantly surprised how much concepts like interrupt, how a mmu works, etc... are useful even when writing high-level code like I'm doing now.

2. Stop considering Databases, compilers, and operating systems as a single unit to learn. They are made of a lot of smaller and simpler components that together form a system. And while when are together seem very complex, often when divided into smaller part became a lot easier to learn even if the total "complexity" remains the same. Take compiler, for example, there is code generation, optimization, and syntax parsing. Instead of trying to build or understand a compiler start by learning the basics of parsing, for example by building a simple chatbot (one of the first things I wrote was an IRC bot), evolve it to understand more complex queries (for example SQL).

Related to this: One thing I could not understand was how big relational databases, like MySQL, worked, a lot of the literature on the topic seem to have a perspective that didn't help me (like the math behind it). The ah-ha moment to me was with sqlite and seeing that at the base was a simple key-value store, and everything else was built on top of this. One takeaway, for me, was that when you feel that you don't understand something, maybe you are asking the wrong question! in my case I keep searching "how relational databases work" instead of what I was really interested in "how relational databases are implemented".

3. I would not start with a thing like "the dragon book", from what I see is an extremely complete book about "everything you need to know about compiler" (hasn't read it, so correct me if I'm wrong). Book like that can seem intimidating and you risk of reading a lot of material without really learning and I feel more like an "information dump". Search for smaller book or course that are an "introduction" to compiler instead of the "everything you need to know", they may not teach you everything, but you will have a better understating of "core" concepts that allow you to tackle more complex topics a lot easier.

Side note: in the specific case of compiler search for a book/course that focuses on building an actual small language! The book my professor gave to me only had a disjointed example for each topic. Something like "crafting interpreter" with a precise goal is MUCH easier to understand while teaching the same concept.

Hope this can help, or at least a starting point for a discussion.

edit: formatting

- library

- SW combined with the Pi

- find something to mine

- ML prototypes

For consuming non-beginner material, it’s okay to read a blog post and not fully understand everything. Things will click together over time.