> There are other technologies for this, such as DTrace. The kernel's choice isn't eBPF or nothing, it's eBPF or something else like it.
To add on this point: I successfully used SystemTap a few years ago to debug an issue i was having.
Before going further: keep in mind that my point of view (at the time) was the one of somebody working as a devops engineer, debugging some annoyances with containers (managed by Kubernetes) going OOM. I'm no kernel developer and I have a basic-good understanding of the C language based on first-years university course and geekyness/nerdyness. So in this context I'm a glorified hobbyist.
Learning SystemTap is easier in my opinion. I followed a tutorial by RedHat to get the hang of the manual parts but after that I remember being fairly easy:
1. Try to reproduce the issue you're having (fairly easy for me)
2. Skim the source code of the linux about the part that you think might be relevant (for me it was the oom killer)
3. Add probes in there, see if they fire when you reproduce the issue
4. Look back at the source code of the kernel and see what chain of data structures and fields you can follow to reach the piece of information you need
5. Improve your probes
6. If successful, you're done
7. Goto 4
I think it took like one or two days between following the tutorial and getting a working probe.
To add on this point: I successfully used SystemTap a few years ago to debug an issue i was having.
Before going further: keep in mind that my point of view (at the time) was the one of somebody working as a devops engineer, debugging some annoyances with containers (managed by Kubernetes) going OOM. I'm no kernel developer and I have a basic-good understanding of the C language based on first-years university course and geekyness/nerdyness. So in this context I'm a glorified hobbyist.
Learning SystemTap is easier in my opinion. I followed a tutorial by RedHat to get the hang of the manual parts but after that I remember being fairly easy:
1. Try to reproduce the issue you're having (fairly easy for me)
2. Skim the source code of the linux about the part that you think might be relevant (for me it was the oom killer)
3. Add probes in there, see if they fire when you reproduce the issue
4. Look back at the source code of the kernel and see what chain of data structures and fields you can follow to reach the piece of information you need
5. Improve your probes
6. If successful, you're done
7. Goto 4
I think it took like one or two days between following the tutorial and getting a working probe.
It was a pleasant couple of days.