> It is possible to create threads by using the OS syscalls bypassing completely the requirement of pthead. (Un)fortunately, I couldn’t find any popular libraries that implement the functionality by using the syscall interface instead of relying on pthread.
I have tried and failed to do this for a C++ program because the amount of C++ runtime static init/shutdown stuff you would need to deal with isn't practical to implement yourself.
CORE only works on kernels that support BTF. This post introduces one workaround which is to generate BTF data for kernels without it. That's still only half the problem though. You also need to write your eBPF program so every kernel verifier passes it, even though every kernel's eBPF verifier has different bugs, capabilities, and complexity limits. I maintain a large eBPF program that supports 4.14 through 6.14. We implemented our own version of CORE before CORE really existed. In reality, it's a lot more work than "compile once run everywhere."
Yeah same, we maintain some eBPF probes spanning 4.11 to latest kernel, and holy hell, it's really bad. The worst offender being some old RedHat kernels with half-baked backports of the eBPF features containing a bunch of weird bugs or features that aren't perfectly in line with what's used in mainline...
Here's a fun bug we recently had: we had to ban substractions in our program (replacing them with an __asm__ macro) because of a bug in linux kernel 5.7.0 to 5.10.10, which had the (indirect) impact of not properly tracking the valid min/max values in the verifier[0]. The worst part is, it didn't cause the verifier to reject our program outright - instead, it used that information to optimize out some branches it thought were never reachable, making for some really wonky to debug situation where the program was running an impossible control-flow[1], resulting in it returning garbage to user-space.
All this to say, CORE is really only half the problem. Supporting every kernel in existance is still a huge effort. Still worth it compared to the alternative of writing a linux kernel driver though!
Kernels without BTF data are ancient at this point. BTF was added in 4.18, that was in 2018. 2018! If you're running a kernel older than that, you don't need BPF, you need a whole new operating system.
Yes, each kernel version might have different features between then and now. You have to pick a minimum supported version and write against that.
Many, many distributions didn't embed the BTF information until fairly recently. OpenSUSE did it in 15.4, released in 2023. At $WORK, we have many customers running on distros that didn't have embedded BTF - such as RHEL7 (yes, they pay for extended maintenance).
I really wish customers would update to a newer distro, but I also understand why they don't. So it's up to me to adapt.
> You have to pick a minimum supported version and write against that.
What we end up doing is progressively enabling features based on what's available in the kernel. Every eBPF we write is compiled multiple times with a couple of different flags to enable/disable certain features. It works decently well, and allows using the most capable datastructure/helpers based on the kernel version.
We've got customers who complained when we bumped some critical dependencies and our software suddenly didn't work on Windows 2008 R2 servers any more... in 2025.
As someone who studied physics and electronics for many years, I still appreciate an article like this for reminding me how profoundly weird science is. Working day to day with the equations and practical applications of electricity gives you a false sense of confidence that we actually have any fucking clue what’s going on.
GitHub pissed me off recently by giving us a one month reminder of our renewal, but having required two months notice to reduce our number of seats. As a result, we are ending up paying for twice as many seats as we currently need.
If you exceed the inheritance tax exemption then you are taxed on the $10k so LTCG would be double taxing. You could argue that the inheritance tax should have a much lower exemption but double taxation is harder to justify.
Bullshit. "Double taxation" is such a weak argument to me.
When I buy gas at the pump it's taxed multiple times. State sales tax. City sales tax. Federal gas taxes. State gas taxes. Quadruple tax on me there.
My wage income has several taxes. FICA taxes. Payroll taxes. Federal income taxes. Potentially state income taxes. Potentially city income taxes.
When I pay for a hotel there's often a bevy of different taxes on that. When I pay my phone bill there's a bunch of different taxes on that. Even getting a drink at a bar there's a sales tax and a liquor tax.
And all of that is on money I've already paid all those several income taxes on, so it's really all just stacking there.
Oh but boo hoo ultra wealthy get their massive inheritance "double taxed". Get bent crying over your "double taxed", I'm quadruple taxed and more all the damn time. Weak argument.
But like, why? It's a worthless point. If there were five taxes of one percent each versus one tax of 20% how is the one tax somehow better? Which one would you choose, getting taxed five times or once?
It doesn't really matter that there's a FICA and an income tax and a payroll tax in the end, what really matters is the overall tax rate and if that's fair given some moral decision of fairness of sharing costs of society.
If a city chooses to levy sales taxes and hotel taxes to capture more revenue from visitors because their town is a tourist destination and want to collect from tourists more than locals that's fine. I won't weep a tear for those getting "double taxed" on their vacations.
Local and state taxes generally actually help you. Your local infrastructure is maintained, and your kids go to schools funded by your taxes.
Federal taxes, however, are largely transfer payments from the productive to the unproductive, as well as funding wars in areas that have absolutely nothing to do with you.
It's pretty easy to make the distinction between "good" and "bad" taxes.
>> If there were five taxes of one percent each versus one tax of 20% how is the one tax somehow better?
Would you rather pay five different tax authorities a 1% tax each or a single tax authority 20%? Not asking about the morality of those five tax authorities, in this example they're all the same.
The argument is against this idea "double taxation" is something inherently wrong. In the end it's not a matter of how many different taxes actually apply to a given transaction, what matters is the tax amounts and where those taxes are going to.
I'm generally helped by my federal taxes going to help feed students in poor areas. Both from a feel good standpoint and that those students have a better chance at becoming more productive, which means they can buy stuff from me.
I also benefit from those foreign wars because they keep the US empire of cheap shipping open, so I can buy stuff for cheap from countries who've specialized in manufacturing
Payroll tax is a problem because it’s not accounted for in your overall tax rate, it’s accounted for via lower wages. So you have no idea what your actual income would be if not for all the taxes your employer is paying.
> So you have no idea what your actual income would be
Payroll taxes aren't some secret things which are impossible for an employee to calculate on their own. You're right, it's not usually directly shown to an employee on their paystub, but it's pretty trivial to do the math and see what your employer paid (or was at least originally liable before any weird tax handouts, but typically rare) for your salary.
I do think payroll taxes should be required to include on paystubs if even just as an informational aspect to people. People should have a real understanding of how much labor is taxed compared to capital. People see "capital gains is 15%? Gosh that seems high..." without realizing how much of their W2 income was taxed. Most people I talk to can't even ballpark what their effective income tax rate was, I often hear "oh I had to pay like $800 in taxes this year, that sucked!" No, you paid a lot more than that, you just didn't pay attention.
You're right, its pretty common for the wealthy to try and trick poor people into feeling sorry for the wealthy actually paying taxes on things.
As my other post pointed out, tons of things have multiple layers of taxes. But the things people suddenly have some big moral issue with "double taxation" are things like estate taxes and corporate earnings and dividends and capital gains.
Few people bat an eye at us double taxing the low income nicotine addict. Everyone seems to want me to shed tears for the billionaire having to pay "double taxation" on their third vacation mansion they're inheriting.
Argue the rates are too low or too high for a given transaction. I can get behind that. But just crying because there's two different taxes being applied to a given transaction? Really?
> People get plenty mad about sales taxes double taxing taxed income, too.
Yeah, and it's a dumb position to take. Like what, we can only apply one tax from one source to a given dollar bill's serial number after it leaves the fed, and after that point that dollar can never be taxed again? Otherwise, gasp, double taxation!
The company shouldn't pay payroll taxes, they charged sales taxes to get that income to pay the salaries. I shouldn't pay any income taxes, the company paid payroll taxes on the money that had sales taxes on it. I shouldn't pay sales taxes, I paid income taxes on the money that had already had payroll taxes that had already had sales taxes that had...
Suddenly nobody pays any taxes anywhere because somewhere up the chain someone else already paid taxes on that dollar!
Listen. If it’s never supposed to happen then you definitely want a customer to tell you if it happens. Best way to make sure that happens is to insult their mum. It’s just good engineering discipline.
In my younger and more foolish years I worked for a company called Redgate and, whilst there, I had to do some licensing jiggery pokery that stopped people using our tools if the licensing code had been tampered with or removed.
I didn't want to make it too easy for people trying to crack our tools so, if such a problem were detected, the error message spat out to the console bore no relation to what had gone wrong.
By "bore no relation" I mean the error message would be one of a number of lines of dialogue from a scene in a House episode where House is trying to figure out who Wilson's girlfriend is... which led to this:
Yes, Greg Young (the event sourcing guy) ran into an error that said "Because I wanna ask you about your girlfriend. I must know who she is, or you would've told me her name," because he'd moved Smartassembly minus the licensing DLLs. Of course, he didn't realise what he'd done "wrong" so he tweeted it to his considerable following and it went viral.
Because I'm exactly the sort of walking cliche you'd expect I was blissfully unaware of the unfolding drama, being away on sabbatical snowboarding at the time. I hadn't taken my laptop with me, I'd disabled data roaming on my phone to avoid a whopper of a bill after a couple of months away, and had only a gen 1 iPad to access email and internet. So of course I came back in the evening to discover that it had all kicked off, with this ludicrous chain of emails where people were trying to figure out what was going on, until somebody had the sense to go and look in our source control system (which at the time was still svn for most things), and figure out that it was my fault.
People saw the funny side and it wasn't that big a deal but the reason I bring this up is that the last line of dialogue in that scene is, "Your mama," delivered by Wilson to House, and it was also the error message for the final failure scenario I was looking for. I'm glad that one didn't get printed.
Back in the nineties, 3D Studio used to be pirated by every kid in their dorm and more. The software had license checks all over the place apparently, but crackers tried to remove all of them.
However, if they missed one, it would move a random vertex by some random amount which is obviously very visible when rendering a 3D scene.
When people complained about this on forums (usenet mostly), they outed themselves for using pirated software.
You should be 100% sure that such code never ever gets executed by legitimate user. As seen above, it doesn't have to be a bug in licensing code, it could be something external.
Yup. Long ago, join an online Command and Conquer game. Boom, everything blows up. Piracy watchdog that reacted to my repairing Windows and not realizing I needed to reinstall the game after that.
System.exit(0); in Java is probably the most evil thing invented, when the license check fails (preferably launch a thread and do it without stacktrace, after a random time).
To this day I still think Google’s motto only referred to “Do no System.exit(0)”.
Sadly I mostly agree with this. File it with "the first time people get the password right tell them they failed, making them enter it a second time" and other janky but highly effective stuff.
The physics game sim step had to be hardcoded but you could vary the graphics rendering independently. That did mean you needed to use a different RNG for sim objects and graphics-only particle effects.
I worked on the deterministic instant-replay system for a racing sim on PS3. The most interesting thing was that on PS3 we had main processor cores (PPU) and helper cores (SPU) which had their own instruction set. Our physics sim could distribute work across both, but the thread assignment had to be deterministic because PPU and SPU got marginally different floating point results for the same instructuion and inputs. That was a fun one to figure out!
Incredible insight, thank you! Could you elaborate on any more of your experiences with PPU/SPU programming? Also, what have you gone on to do since then?
I implemented a debug replay system for an American Football game when I worked at Sony. The AI hierarchy and all animation and physics state was recorded and game could be rewound to figure out why an NPC made a poor decision. Only needed to track state for the duration of a single play. This was on PS2 which I think had 32MB of memory but the dev systems had an extra 96MB which we could use for this sort of thing.
I have tried and failed to do this for a C++ program because the amount of C++ runtime static init/shutdown stuff you would need to deal with isn't practical to implement yourself.
reply