> You are not "introducing" a new macro for this, you are modifying the
existing one such that all users of it now have the select_nospec() call
in it.
> Is that intentional? This is going to hit a _lot_ of existing entries
that probably do not need it at all.
> Why not just create list_for_each_entry_nospec()?
Let's ignore whether this patch is needed or works for now - I don't feel competent to comment on that. But this is such a bad suggestion. Instead of fixing the default to be safe, and possibly having a _unsafe_but_fast() variant for the places where it makes sense they want to keep the broken version and require user to explicitly opt in into safety.
Same as the infamous PHP mysql_real_escape_string (don't make the mistake of using mysql_escape_string!) [2][3] or whole host of C stdlib footguns like strcpy/strncpy [4].
The default, easy, obvious option should be safe. The unsafe but faster option should be hard to use by accident and obviously marked as such from the name.
This is just how you work with large code bases. Updating 15,000 call sites is far from trivial. These changes happen in multiple phases, and in each phase, you still want to have a working kernel.
The unsafe version can be deprecated and eventually removed, but that is down the road.
When there are hundreds of branches, not just feature branches but real long-lived development branches, and pull requests are managed the way they are in the Linux kernel (chain of trust all the way up to Linus).. "Do the mass replace" becomes quite non-trivial I imagine.
This strategy handles that fine as long as none of those branches lasts from before step 2 until the optional step 6. You pull from a long-lived branch, get a compile error because it calls select_nospec, do the replacement at that callsite, and commit. A headache but a manageable one. If you want to backport new features to old stable kernel versions, you can include a patch supplying select_nospec_unsafe as a synonym for select_nospec.
I not arguing is effort-free, but is effort-minimal. The idea allows, at most, one or two major steps, the first and the biggest is as simple as is possible and can be done in one go, and have zero possibility of breaking.
The only major issues is that the change is let half-way. But that is only possible with MAJOR undisicpline.
---
The major point, to me, is that this EXCESSIVE fear of breakage is not good.
Yes, making upgrades is pain, but you CAN make the pain tolerable with some planning.
Refactoring is like exercise for code: Everyone dislike the idea of excessive, but is GOOD.
And lets be honest, among all the things that could requiere a refactoring, this case is on the most simples of the simples scenario..
I’m not sure how it’s done with the Linux Kernel (and git definately makes it hard), but at Google (using Perfoce model) there are tools for mass renaming that try to guarantee that there are no regressions. Of course in Linux the amount of C macros make it much harder to work on the syntax trees, but comparision after the preprocessor step should be possible.
So the ’iterator’ variable may be removed, but as this is c, might also be reused multiple times, or come from some struct member, union, or whatever. So i presume something like this would have to be fixed case by case.
You can do these kinds of changes with clang-tidy, if necessary. Clang-tidy is extensible and you can write your own rules for transforming source code.
Mind you, there are not a ton of people around who have the time to learn how to write custom clang-tidy rules. It might be done for 15,000 changes that potentially fix kernel vulnerabilities, though.
In practice it is not that hard. The refactoring tool may only need to tackle one translation unit at a time. You create a pattern which matches some usage of a macro, and create some logic to write out some changes to the AST. It can involve a surprisingly short amount of code—because clang-tidy has good tools for writing patterns and modifying ASTs!
You can brows the clang-tidy source code yourself. There are existing checks specific to the Linux kernel in there already. Well, there’s one check. But if you use clang-tidy, you’ll discover that automatic refactoring of extremely large code bases is within reach.
The only question is whether you would want to spend the staff hours working on a clang-tidy check. Large code bases are exactly where the tradeoff makes sense.
At the start forking is like no deal at all. But how do you get new clang tidy features from the main branch later into your fork; I think this is the only instance which I would really call technical debt.
“Git pull” works pretty well to bring new clang-tidy features downstream. If you write a custom check, it goes in its own set of files, so there will be no merge conflicts. The only thing that would break is when the internal clang-tidy API changes, but clang-tidy checks tend to be fairly short.
Really… I have worked at two companies that had extended clang-tidy to add custom checks for their code base. The whole point of clang-tidy is to automate code fixes in other parts of your code base. When done well, use of clang-tide pays down technical debt faster rather than adding to it.
Forking to add new checks only causes problems with pulling from upstream when APIs you are using are changed upstream, which happens rarely. Forking to _modify_ existing checks would cause you a lot of pain, but the solution there is to copy the existing check instead.
IIRC Google works the same way. The automated refactoring tools (Rosie?) may make it go faster, but you generally don't fix up 15,000 call sites in a single change... you break it up into smaller batches spread across the tree.
It's been a long time since I was there, but I thought there were plenty of 15,000 call site refactorings done in a single final CL. Not that the Linux kernel should do the same!
When I was there (several years ago) it was rare to do that across many projects. You don't want to have to roll back everyone due to a problem that affects one project. Also, there's a risk that changes that are happening in the meantime might mean the patch doesn't apply.
It may take a lot of time to get all the approvals from all the code owners. One of the big advantages of a tool like Rosie is precisely that it splits up the update into smaller CLs and has automation that nags all necessary reviewers and reruns the transformation once head moves.
What does git or Perforce have to do with refactoring? Why would whatever version control you're using matter?
> tools for mass renaming that try to guarantee that there are no regressions
beyond sed or whatever a fancy IDE already does? What kind of voodoo magic are we talking about here? Did Google solve the halting problem and I just haven't noticed yet...
Disclaimer: I haven't worked at Google, but you can easily read their papers like https://dl.acm.org/doi/pdf/10.1145/2854146 and understand what's going on. Refactoring tooling is integrated with the revision control system because in a giant codebase with thousands of contributors, you need to split a refactoring patch into smaller chunks and ideally use semantic rather than text-based diff tooling, or the changes will never get reviewed or merged before they conflict with more changes.
As for the actual renaming, yes, definitely beyond what sed does but probably around what the fanciest IDE does. Imagine a big global symbol dependency graph produced by the entire build toolchain all at once and cached, I think?
Also, this isn't the halting problem unless your codebase allows for fully dynamic invocation :)
> Aren't all function pointers dynamic invocation?
Given that compilers can be smart-enough to detect "non-dynamic" function-pointer invocation (e.g. when an execution-trace proves that a function-pointer parameter always points to the same function address), it's not safe to say that "all" function pointer invocations are dynamic.
Another case to consider is when one implements (Smalltalk-style) OOP with message-passing: in many cases it's possible to build that without needing to use function-pointers at all.
I'm not 100% sure, but i remember that i could have have both a two and three function argument be called with the same function pointer, its probably UB however i think that as long as any dereferenced functions adhered to the standard argument behavior of the platform specific calling convention it worked, and i dont think gcc, clang or msvc complained, but my memory might be wrong.
The rename has to be atomic, which is one of the things your revision control system gives you, unless it's like CVS or something worse†. So, that's a necessary part of the solution space.
You want there to be two states: State A where the thing was called old_name and State B where the thing is called new_name. State AB where some code thinks it is called old_name but other code thinks it is called new_name is broken, and must not exist.
† This might seem outrageous to modern programmers, but CVS thinks in terms of files, so from its point of view it's fine if out of sixty files you tried to commit, 48 of them succeeded and 12 failed. Good luck fixing the resulting mess.
I remember having to go backwards from SVN to CVS for a project a long time ago, and I was so shaken by the fact that CVS couldn’t do atomic commits with multiple files!
Yeah, my favorite model of how git operates is "rewind all the way back to RCS, add atomic commits and take it from there".
SVN, baz/bzr and friends are what you get if you add networking before atomic commits. Git is what you get if you start with the proper data model and only then add networking.
I believe the commenter is referring to system where you take a giant “git sed” and automate a rather complicated process: 1. break up one huge diff into a set of many (hundreds or thousands) patches, where each individual patch only touches files in a particular subsystem. 2. send all those patches to the appropriate system owners and run the appropriate tests. 3. manage the actual merging of successful patches, as well as communicating to the original author any individual patches that might need more attention e.g. based on patch review feedback.
as you might imagine, it’s probably a pretty involved process to touch thousands of lines of code in a complicated system.
There are multiple branches being actively worked on. It would be very easy for someone to murge in an unrefacatored branch and now half the code base is using the old call.
Linux definitely has testing. Some helpful person added our repo base to the kernel-test-robot list, and now I regularly get e-mails when something I'm working on in some random WIP branch broke the build on some other architecture (often in COMPILE_TEST mode), or some other random driver. Didn't even have to do anything myself. It's great for avoiding "this broke the build on $obscure_config" issues.
While calling C a typed language is technically correct, I struggle to think of any typed language that's more weakly-typed than C. Arguably, Python has a stronger type system than C.
The C preprocessor is a siren song, lures people in with the promise of "extra performance/features/structures for zero runtime cost!" but murders any readability or rationality within your codebase.
Cool concept, but once you've dealt with any legacy codebase that has used it extensively, it feels like an anti-pattern/footgun. Ultimately you just want the language to do those things directly and for the compiler to be smart enough to optimize it later.
I do like the preprocessor and my major pet peeve with C is that the preprocessor has been stagnant for ages. Like, why the hell can't i do something like
#define BASE hey
#define HEADERS foo bar baz
#append HEADERS bad bal bah
#push OSSUFFIX
#ifdef WIN32
#define OSSUFFIX win32
#else
#define OSSUFFIX unknown
#endif
#foreach H HEADERS
#eval include "$(BASE)/$(OSSUFFIX)/$(HEADERS)"
#endfor
#pop OSSUFFIX
People go to great lengths making all sorts of weird structures from x-macros, repeated statements, defines that only exist to be used by other defines, etc all to work around existing preprocessor limitations - and many of them would simply become unnecessary if the preprocessor could do things like variable editing, loops and being able to eval its own commands.
Even though some stuff can be done via language features, it is often necessary and more flexible to work with the source code itself.
One alternative is to just write a C code generator, which lets you mix C and some sensible language, eg. python. Then use that to generate the code which is then sent to gcc.
Not really? They're part of compilation and obey all the type rules & syntax of the language. They're not textual replacements that run in a distinctly different phase like macros are.
Its not too clever for its own good,
its programmers that are too clever for C, and desperately need basic features, and its the only way to get them unless you switch language.
Its also surreally poorly designed, encouraging worse habits than C itself. Avoiding definition collisions and lack of namespaces alone make it horrific for any moderate sized project and up. Combine that with the near complete lack of static analysis tools, and its a recipe for disaster.
Yeah, I do agree. Interestingly, you could argue that it's fairly strongly typed; at least according to some definitions (and btw "type-strength" is not a rigidly defined term, so the definition itself is somewhat debatable).
I don't claim to be an expert in knowing what is/isn't in the various standards; I just look at the build errors and static analysis alarms. That said, I think the argument is that in the vast majority of cases it's not actually possible to change the type of a variable; off the top of my head, I can only think of pointer co-erosion. Otherwise, if you define an unsigned int, it stays an unsigned int.
Now strong/weak typing isn't necessarily the same thing as type safety. C has always seemed astonishingly bad on that front. It's like they try to trick novice developers into thinking they have a robust type system sometimes.
If you define functions implicitly they will link to any symbol, EVEN A VARIABLE... this is bananas. I think I sort of understand why compilers work this way, but it really feels like a bug in the language. Under some compilers you might not even get a warning for implicit function, either! TI had a compiler that hid them by default.
Enums are garbage in C. Again, you can misuse them, and may not even get a warning. You can pass an enum for color to a function that takes an enum for kittens and the compiler will be happy as a clam. The way they're often used can cause constant implicit conversions every time there's an assign/compare. May not be a problem for most positive values, but it's annoying if you're trying to develop standard compliant code. MISRA defines an "essential type system" and normal enums usage violates it.
I'm probably forgetting a TON of deficiencies. Please add them or correct me where I'm wrong.
That’s a good question, it’s not fair that you’re being downvoted.
It’s the C preprocessor that causes a mess. Tooling for C and C++, like automatic refactoring, lags behind tooling for languages like Java, C#, and Go. Refactoring tools have to deal with macros, conditionals inside #if/#else blocks, and header search paths.
In this case, the refactoring involves removing a variable from the enclosing scope of a macro invocation. The most likely way to automatically refactor it would be to write a custom check in clang-tidy.
Whether something is (statically) typed or not is more of a continuum than a binary.
Eg C tracks whether a variable is an int or a char. Haskell also tracks whether a function causes side effects or not. More sophisticated systems also track whether a function has to return eventually or could run forever.
Atomically flipping a version is still difficult, as anything that needs to rely on the other version has to be fixed forward. If you instead make 5000 (or 500) small changes that affect 10-15 uses, you can solve the unique cases or roll-back and forward.
> Same as the infamous PHP mysql_real_escape_string (don't make the mistake of using mysql_escape_string!) [2][3]
TBF that is really a straight bridge to the mysql C API, which is why it’s also in mysqli.
MySQL actually has a third one called mysql_real_escape_string_quote, because if the sql mode NO_BACKSLASH_ESCAPES is set the escaping function needs to know what context it’s used in, and thus what quote to double up, and mysql_real_escape_string will fail.
And of course, the real answer to wether you're supposed to use mysql_escape_string, mysql_real_escape_string or mysql_real_escape_string_quote is "why the fuck are you pasting queries together using string processing".
That's the deal with backwards compatibility. Even third parties may depend on that name, it's not safe to go and change it.
On userspace that's the time when you deprecate an entire module and switch everything into a new name. The kernel does that once in a long while, but it's a bit harder for them.
The kernel doesn't care about third parties though, that's why the kernel API/ABI is not stable. This is one big advantage to doing it that way: sure, you need to have a consistent migration plan in flight to make sure you don't break mainline in the process, but you can immediately remove the old way without having to care about third parties, once you're done.
> Same as the infamous PHP mysql_real_escape_string (don't make the mistake of using mysql_escape_string!) [2][3] or whole host of C stdlib footguns like strcpy/strncpy
In all of those, the enhanced function takes more paramaters. Switching the existing name to require a new parameter will break existing code, and people will not want to update. Making the new parameter optional is possible, but IMHO is messier than a new function that has required parameters and then deprecating the old function (and eventually removing it).
You are making the rather optimistic assumption that here is a single definition of the macro. There are likely dozens, most of which share a name and randomly overwrite each other, others share the name but are subtly different. Macros are a truly terrible way to metaprogram. In my experience, if you spot one in a code base and can remove it, you have removed a bug you never knew you had, you spot one and change it, you have broken more things than you will ever know.
C99 has one nice feature compared to C89 that is little talked about: you can initialize local aggregates using expressions that can't be calculated at load-time.
void fun(int x)
{
struct foo = { x }; // not allowed in C89
int bar[3] = { 0, x }; // ditto
}
Chances are the kernel does this because, I think, it's also a GNU89 extension.
Linux already uses a lot of C99 features that were GNU extensions before C99, like designated initializers. So in practice, Linux is already C99, and hasn't been buildable on strict C89 compilers in a long time.
If we are walking iterator->list->next ... how do we get the pointer of the next enclosing foo struct? Are they doing pointer arithmetic to get the beginning of the struct from the list field offset and casting it to foo?
I'm one of those people who think OS kernels should stay as portable and simple as possible (i.e. C89 or some other easily-bootstrappable language, to avoid Ken Thompson attacks), so this isn't great news to see "the ladder being pulled up another rung". Then again, Linux has already become immensely complex.
Linux has always relied heavily on GCC extensions, though -- it makes no attempt to actually be C89-compliant portable code. What it actually has is a minimum supported gcc version, which in turn governs whether particular features can be used. In this case the minimum gcc version has for other reasons finally got big enough that C99 and C11 support is definitely present -- the ladder was already this high.
(You can also build with clang, but only because clang deliberately aims to support most gcc extensions.)
The advantages of using a newer language greatly outweigh the disadvantages of theoretical "reflections on trusting trust" attacks. By mandating C89, you're condemning thousands of kernel developers to use a language that's over 30 years old because of a theoretical attack that has never happened and seems practically implausible. Does anyone really think there's a backdoor in both GCC and Clang (remember, Linux can be compiled on either)?
I am ignorant on nearly all compiler related issues, but there was an article on here not too long ago that was arguing that nearly all os development required old C because choices of the committee would break use cases required under the guise of undefined behavior.
is the benefits of "modern C" worth compile times 2-3x times longer?
Times like these I wish I were allowed to say exactly how much money big companies save by using the newest versions of GCC and Clang. The economic value of modern compiler optimizations is staggering.
Facebook has hinted that they employ smart C++ people because some core optimizations can save on the order of several hundred thousand dollars per year (possibly in the millions). Most of that is off because they don't have to buy as much power to cool the server rooms, some of it is buying less servers as what they have can do more. Facebook doesn't actually say what they save, but they have said they measure it, and we know how much it costs to employ an engineer full time working only on optimization, and we know who some of those engineers are.
You have to be very large to notice it, but the likes of Facebook, amazon, and Google have massive warehouses almost entirely filled with computers. It doesn't take much to see how their power bill can add up.
Years ago Alexandrescu said that Facebook estimated a .1% speedup to HHVM would save them $100k/year. It's presumably only gone up since then, but he's stopped giving an actual number in talks.
I would argue that he is bragging, and that the amount of code optimization is minimal. Large companies love to brag about how many servers they have, and optimizing compilation is just a buzz word. Go ahead, buy another servers. $100k is nothing. Its less than 1 engineer year vs 24,000 engineer years.
The Distribution collectors, they brag about speed, and optimization is very important to them. They want to be able to test changes very quickly.
The parent works at Facebook/Meta leading their Rust team, focusing on compiler and ecosystem improvements. So that in and of itself is a sort of hint into what that info could be like, even if it's not actual details of the order of magnitude or anything.
The Google Search optimization team measures changes in terms of millions of CPUs. So if upgrading your compiler saves even 0.1%, well, that's a lot of servers you don't have to buy next year.
"Jun 30, 1987 — The Cray is part of a $20m installation used by Apple's Advanced Technology group and consists of four CPUs operating at 9.5nS per cycle,"
Now we have distcc, that can both massively parallelize compilation, and optimization.
Googles team probably shows some slight improvement, just to justify their work, but in the long run, its just as sloppy as industry wide coding is.
Microsoft is the absolute worst, along with Apple.
One avenue is to think about it in terms of datacenter compute. If modern compiler optimizations improve performance by 10%, aggregate the perf improvements across the entire world's compute and you get a huge saving.
> but there was an article on here not too long ago that was arguing that nearly all os development required old C because choices of the committee would break use cases required under the guise of undefined behavior.
I would guess that you're referring to either "How ISO C became unusable for operating systems development" ([0]) or "How One Word Broke C" ([1]).
Most of the really annoying UB is technically in C89 though, it’s just that compilers haven’t really treated it with such contempt for the first two decades or so. I can’t even recall any new kinds of (non-library) UB in C99 or C11 (though there have to be some).
So “old C” in such a case would need to mean “an old C implementation” (or possibly a new one, but simple or configured to behave like an old one), something like GCC 2.8 maybe, and nobody’s using that on desktop. So the language standard version should be mostly immaterial, and it’s not like the C89-to-C17 difference is anything like the yawning C++98-to-C++20 chasm. (This is a carefully phrased statement: C99 had complex numbers, which are annoying, and variable-length arrays, which are a significant change, but C11 demoted both to optional features.)
I think you're mixing up two different things. Newer standard versus newer compiler. All versions of the C standard have lots of undefined behavior. Newer compilers contain optimizers that are just better at leveraging it.
If the spec requires that the compiler look for a certain error and produce a diagnostic, then the compiler has to spend time doing that. The more complicated the spec is, the more work the compiler has to do to find the errors.
For example, Rust is frequently said to have long compile times. There are a number of interesting reasons why that is often true, but if we ignore the pathological cases then what we find is that borrow checking takes a significant fraction of that compile time. This is a big trade–off between features and complexity that the Rust language made very deliberately: the advantages of the borrowing rules are what makes Rust such a great language. The cpu time spent checking that those rules have been followed are a small price to pay, but not a negligible one.
You cannot disable the borrow checker in Rust with a compiler flag. You might as well try to turn off the type checker, or ask it to ignore syntax errors.
Declaring a variable in the middle of a function takes basically no effort to support. Especially compared to all the ridiculous things the kernel gets up to.
Ken Thompson's hack is purely theoretical, especially in the modern computing landscape where the diversification of software supply chains would make such hack much 1. less effective and 2. easier to detect. If we're talking about kernel security, there are other lower hanging exploits, many of which are enabled by outdated language design decisions and unsafe memory models.
As compiler technologies evolve, it becomes not only a necessary evil but rather a better course of action to trust compilers rather than humans tiptoeing around security risks masquerading as language idiosyncrasies.
Yes, he did, but that just shows that it's possible to inject self-replicating code in a compiler. It doesn't actually tell us how resistant such code is to compiler source changes, audits, binary analysis, etc. which are all required for a successful exploit.
There's a world of difference between that and Thompson's attack. You don't exploit the kernel code: you exploit the compiler code, such that every program it compiles (a) is compromised, and (b) is incapable of detecting the exploit in programs compiled by the same compiler.
That's a completely separate issue. The trusting trust bug requires you to convince people to use your compiler binaries. The thing you're talking about just requires a patch to pass code review.
It's more work - you have to figure out a sneaky bug and write a legit looking patch - but anyone can do it. You don't have to be in a position of power already (e.g. being the Debian GCC packager) so overall it is much easier.
I think they only got past a single reviewer and counted that as success, none of their harm code made it into the kernel proper. However the University also committed thousands of automated "fixes" for linter warnings that weren't part of the study and the kernel maintainers ended up reverting all of them due to the Universities overblown claims.
It's obvious that the hack would work on a technical level, that's fundamental to the whole idea. The question is how complex the logistics of doing it in the real world would be; how you insert the tampered code, how you avoid anyone noticing, etc.
For example, I run Gentoo Linux, and haven't reinstalled my OS since 2004 or so. That means that, modulo a few binary packages, I have a direct source lineage to the state of Linux in 2004. If you want to pull off that attack against my system (and you didn't already back in 2004), you'd have to tamper with source archives. That would both imply changes that are easy to analyze (more than binary patches), and it would involve changing the archive hashes in the Portage tree. That tree is in Git, which means that it would create an immutable public record of what happened (Git is the original blockchain, remember), modulo forced pushes which people would, again, notice all over the place.
In practice, if you want to persistently backdoor a new system (supply chain attack), it's usually easier to do that in hardware or firmware than trying to do a RoTT attack on the distro and its compiler. In fact, it's users of binary distributions (or proprietary OSes) that should be more worried, as it is much easier to do a binary-based RoTT attack that self-updates to handle new versions consistently when all your users run the exact same binaries. Source code users should be more worried about compromise upstream than local persistence. And those attacks are a review / auditing issue, unrelated to RoTT.
In the end, if you are worried about being personally targeted, it's easy enough to make that impractical by re-bootstrapping your computing from an unpredictable source (e.g. walk into a random shop and buy a PC, walk into a net cafe and download your favorite distro and check the hashes there). And if you are worried about large-scale attacks, RoTT style ones aren't practical without someone somewhere noticing; you should be worried about traditional compromise instead.
Does https://en.m.wikipedia.org/wiki/XcodeGhost count? It's not targeting compiler developers, so it can't worm forever, but it is a malicious compiler that weaponizes its output.
In this case practical attacks are very likely, due to insecurities added with C11 and the stoneage review model of Linux development.
C11 added insecure Unicode identifiers, and you won't find them in reviewing them manually via email. You need a special linter to detect such Unicode attacks. Or a proper development environment.
Compiler development usually evolves into more bugs, not less. Just now they caught up with the hundreds of bugs they added with gcc-9. Not talking about const, restrict, and strict aliasing, and the still not existing -Oboring for the kernel. Would you dare to use -O3 and -flto and -fstrict-aliasing in the kernel?
Linux has a checkpatch.pl. If it doesn't already test for Unicode identifiers, it would be trivial to add that. This is a non-issue. You don't ban a new version of a language just because it allows some undesirable things; it's trivial to ban those things specifically, either with scripting or specific compiler option overrides.
Heck, Linux already uses GCC plug-ins to implement some fancier security stuff; it is silly to think they can't handle forbidding Unicode identifiers.
Linux is not written in C89. It is written in gnu C89, which is a mixture of C89, C99, and a panoply of often poorly-defined gcc-specific features. The number of compilers that can successfully compile Linux is one; not even clang is able to fully do so yet, I believe.
Actually, as a compiler writer, I'd go a little bit further and point out that Linux itself isn't even written to the gnu C89 very well; it's often written to a "C is portable assembly" view of the language, which results in nasty grams and invective being hurled at compiler writers if they compile the C specification correctly and not according to the "proper" assembly the code author thought they were getting.
One of the benefits of more modern language revisions is that they actually tighten the wording on a lot of the more ambiguous parts of the specification--C11 in particular adds a much more comprehensive memory model that's very shrug in the older revisions of C.
My understanding is that Linux has often faced issues where ”compiling the specification correctly” means “Ha-ha gotcha, we can actually throw half your code out of the window because you misread the deep aliasing rules on page 8432 of the spec”
Their hesitancy with newer standards is understandable, when viewed against that backdrop
One of these days, I will get around to writing my post as to why that take on undefined behavior is completely wrong.
More to the point, though, the only changes to undefined behavior in the C specification in newer versions (compared to C89) are either clarifying things that were already undefined behavior (e.g., INT_MIN % -1) or actually making some undefined behavior well-defined (e.g., allowing type punning via unions).
That falls under things that were already undefined or impossible to use properly before.
Older standards said "it is
implementation-defined whether the old object is deallocated". This didn't really work well:
- if realloc(..., 0) returns NULL if it freed the object, then you have confusion with error cases. strtol already has this kind of interface and it's unusable
- if realloc(NULL, 0) returns NULL and does nothing it does something different than malloc(0). Some chose to make it do something different, some chose consistency with malloc.
- if you choose consistency with malloc then realloc(NULL, 0) likely will end end up inconsistent with realloc(ptr, 0) where ptr is not NULL. On BSDs the two are consistent but also different from any other platform, so portable code could not rely on realloc(..., 0) doing something known: either you had possible double-free bugs on some platforms, or you had a memory leak.
I'm very unhappy that they chose to make it UB rather than mandating a behavior, but in portable C89 calling `realloc(ptr, 0)` is always a bug in your code, and it was an insane thing to declare implementation-defined originally.
IMO the C standards are conservative enough that even a standard level "only" a decade old is still fine.
With that said, we still have options even when moving to a newer standard. Many new language features can be machine-translated to C89 if needed (similar to how we have protoize / unprotoize, though not always as seamless). If we need to keep a bridge to C89 the kernel authors could hold to a subset of new language features that are amenable to machine translation.
You're confusing portability with age. Just because something is old does not make it more portable. It arguably makes it less portable as new platforms will not support old language versions.
> I'm one of those people who think OS kernels should stay as portable and simple as possible (i.e. C89 or some other easily-bootstrappable language, to avoid Ken Thompson attacks)
Why does the complexity of the OS, or even its ability to be compiled by multiple compilers, matter for "trusting trust" attacks?
As I've always seen it, the problems and solutions all exist at the compiler level. The whole premise relies on starting with a binary compiler that you are expected to trust. The source is also assumed to be safe and un-tampered for purposes of this discussion because that's an entirely different issue.
The solution, of course, is to build the compiler itself with different compilers. If you build the same compiler with two or more different compilers, then use that to compile itself, you should be able to with the right options (see the work that has been done on reproducible builds) get binaries that are equal or close enough to easily compare any differences.
At that point if they are the same then you know either it's good or both of the upstream compilers were also compromised.
If for whatever reason that's not practical at the top level compiler the same concepts apply going further back in history until you get to some early compiler a bored grad student wrote in the '80s in pure ASM.
As a result from a practical sense I don't really see "trusting trust" attacks to be that big of a concern. It's always possible to work your way back down the tree of software until you get to a point where you can actually trust a compiler and then build your way back forward from there.
If a compiler starts depending on its own tricks and gets to a point where it can only be successfully compiled by itself, then there are reasons to be suspicious. Even then you'd just have to have the last version to be able to be built by other packages as one more stop along the way to trust.
Does trusted bootstrapping really become that much more difficult? If you're already doing a trusted bootstrap to an old version of GCC, you can use that to build a modern GCC.
> so this isn't great news to see "the ladder being pulled up another rung"
Man I so vibe with that. But the new Cs do have a ton of new features and, more to the point, I trust Linus to make this kind of decision more than just about anyone.
Ken Thompson's "cc hack" - Presented in the journal, Communication of the ACM, Vol. 27, No. 8, August 1984, in a paper entitled "Reflections on Trusting Trust", Ken Thompson, co-author of UNIX, recounted a story of how he created a version of the C compiler that, when presented with the source code for the "login" program, would automatically compile in a backdoor to allow him entry to the system.
> First we compile the modified source with the normal C compiler to produce a bugged binary. We install this binary as the official C. We can now remove the bugs from the source of the compiler and the new binary will reinsert the bugs whenever it is compiled. Of course, the login command will remain bugged with no trace in source anywhere.
I believe the person you are replying to is using that as an allusion to the issues discussed in the famous "Reflections on Trusting Trust" paper[1] by Ken Thompson.
That extra scope is an issue for what they're doing, however. They have a macro called "list_for_each_entry()" which syntactically behaves as if it were a "for(...)" (because it is a "for(...)"). To make the extra scope work with that macro, either the user of the macro would have to end the loop with an unbalanced amount of closing braces (to match an extra opening brace within the macro), or it would require a second macro to be used at the end of the scope. Keep in mind that this macro is used in literally thousands of files within the kernel.
Sure. This may not be a feasible approach for this case. I'm just pointing out alternative approaches to mitigate this kind of scoping problem in C89. Maybe wrap every use of list_for_each_entry in braces?
Then you get dinged by static analysis tools like SonarQube because you introduced a useless scope or increased the "cognitive complexity" to the code and have to explain it to a team leader who might not understand why you did that.
One thing I've wondered for some time is why don't the Linux developers modify the compiler to suit their needs better.
At that project scale, wouldn't it start making sense to start solving problems like "If it were possible to write a list-traversal macro that could declare its own iterator [...]" by adding the functionality you want to GCC?
Current situation-- do nothing but yell at the compiler devs. Benefit: sometimes they listen. Cost: you cannot always (or maybe even often) get the compiler to behave as you think it should because you don't control it.
Deathtrap situation-- maintain an operating system and a fork of a compiler. Benefit: you can get more control over the compiler. Cost: you still cannot always get the compiler to behave as you think due to time constraints. Death cost: your first cost is multiplied by the fact that you're now maintaining a goddamned compiler.
I rankly speculate Linux is an extant project at its scale because it has refused to fight on two fronts like this. (And yelling across a border isn't the same as crossing it.)
GCC already has that functionality, it's called C++. Like that macro is just a crappy version of std::for_each.
It doesn't seem useful to make a "C+" instead of switching to a language that just has the feature set the kernel needs. Like Rust, which is gaining some support within the Linux kernel already.
They've dug themselves into a hole when it comes to C++.
If you started writing a major project in C now you'd rightfully get sacked, but open source projects like this are often dominated by the opinions of those who only work on that project. That doesn't mean they're automatically wrong but just that they're often massively detached from any feedback apart from disaster.
C++ was discussed as having too many undesirable characteristics.
But that's exactly why I think a customized language for the kernel is an interesting idea. You could get pretty much exactly what you want for the kernel. Add features you need, remove any undesirable behavior or features.
For most programs that would be too much complexity, but the kernel has very particular needs. And I think a similar in spirit approach has worked very well with Qt.
> C++ was discussed as having too many undesirable characteristics.
C++ allowed the Serenity OS people to produce an entire OS with a GUI stack able to play Diablo and their own web browser in something like two years. It's depressing to think where we could be today in terms of OS if the Linux and GNU people weren't as insistent on their hate of C++
Note that Serenity OS also has its own stdlib "AK" that excludes a lot of criticized C++ "features". If the Linux devs were to carefully splice out the Good from the Bad and Ugly of C++, then maybe it'd suit their tastes just fine. I think the bigger factor here is that Linux became decidedly anti-C++ back when C++ was actually pretty crumby.
You must have had one of those progressive professors who did not consider assemblers "a waste of a valuable scientific computing instrument [...] to do clerical work": http://worrydream.com/dbx/
Please write a JavaScript program that will not deploy to your production system if you accidentally pass a string to any function that expects an integer.
The desirable characteristics of C++ overweigh, by far, its "undesirable" ones (whatever those may be, and which by the way you can for the most part safely ignore, if you wish).
I don't know anything about C++ really, but I have been interested in learning it several times. The biggest deterrent for me has been the huge number of languages that you have to know to learn c++ it seems. Every new version comes with so many different features and seemingly (by looking for example repos on git) different paradigms that it essentially feels like many different languages.
This is probably one of the biggest reasons I may turn to rust over C++ - simply less features creep due to less time being around.
Can you comment on how wrong I am about my feelings this way?
Yes, I understand, I would probably feel the same if I wanted to learn C++ all at once and for its own sake. Luckily, I have never had to do such a thing. Indeed, I may never learn C++ in its entirety, and I may even miss some "important" pieces, but I am OK with that. I am very comfortable using C++, and I still continue picking up pieces of wisdom here and there - as I go. My advice, learn by example, start with a small project and go from there.
Learning/teaching C++ is definitely a sore spot about it. Many still teach C++98 which is... Bad. It'd be like still teaching Java 1.3. yeah the syntax is the same basic shape, but pretty much everything else is different.
C++11 and newer are all largely one "category" in terms of recommendations. The CppCoreGuidelines is a good place to cover all that, but it's not a from-scratch introduction by any means
Not wrong, but its a feature, rather than a problem.
C++(17+) template based metaprogramming far more powerful and generic than what you can do in eg rust. Converting eg a rust or go or python, or julia, etc library into c++ is pretty straightforward, just use appropriate overloads, and a few template tricks, and you can mostly copy the code directly. But copying between these, or from c++ is much harder.
The solution is to learn one approach suitable for the problem you have right now, then widen over time. The many languages of c++ as you describe is less of a problem, the problem is that some approaches are deeply and fundamentally flawed, but still in common use.
Let's say you wanted to do that but also stay in the C family. It's going to be a lot easier to take C++ and just ban all the stuff you don't like (such as via lint rules) than to make a new language. In the former case you continue to benefit from all the ecosystem compiler & tooling improvements (ide support, static analysis, etc...). In the latter case you're on your own. Which is no small burden here.
I'm not necessarily advocating that Linux should switch to C++, just that there's probably not a good reason to invest in a new C/C++ hybrid language at this point in time. Not when C++11 is honestly pretty good, but also Rust, Zig, or even D's betterC all already exist.
C++ has implicit allocation and Linus doesn't like that, for good reason. While it's amusing to imagine trying to "ban all the stuff you don't like" when that means core language features, it is not a realistic plan.
In Rust, allocation lives in a library, alloc, and so the Rust for Linux project did all the work to offer alloc (it's full of useful stuff and it isn't like the Linux kernel can't allocate memory) but without implicit allocation.
If you use Rust to write say a Linux command line program, you can write
greeting += " and welcome traveller";
... and of course implicitly this is an allocation, 'cos it's not like this greeting variable just magically already has enough space to append a string. But in Rust for Linux, you can't do that, the implicitly allocating += operator is not provided on this type in their alloc library. If you want to say "Allocate more space for the greeting" you can do that of course, just as you can today in C but you must do so explicitly and so when you try to add 16GB of extra string space because you're an idiot, the API you had to explicitly call gives you an error and that's your problem.
However the most critical reason Linux doesn't have C++ is that C++ proponents didn't do the work. Now, that will probably be because "reform the entire language to suit Linus" wasn't a viable plan, but the fact is that Linus can't accept patches that nobody writes, so even if you're sure C++ would be viable without drastic changes, you didn't write the patchset that does it. Likewise if people don't send Linus patches to do C11 it probably won't happen.
Ah, but see, it doesn't. Linus was wrong about many things in his rant. This being one of them.
Now the standard library does indeed have things that do implicit allocations, such as std::string. But these, like with Rust, are distinct from the language & replaceable. Would it be effort to make a kernel-safe std:: replacement? Yes. Would it be a lot of work? Not really. And it's the kind of thing Linux has been doing for decades with C anyway. It's not like they use a standard libc implementation (much less a rich libc implementation like glibc), for example.
And it's something game devs have been doing with C++ for decades without any issues, too.
So for this one you don't even have to ban anything. Just don't pass a standard library implementation to the compiler. It doesn't come with one, after all. You have to add it. So you could just... Not do that.
As for nobody did the work... That's true. But Linus also pretty much nuked the entire concept of using C++, regardless of the what or how. Time seems to have changed his mindset on some of those things, hence his reaction to Rust, which largely makes all this moot. But we shouldn't confuse short term politics with technical issues, either.
> Linus was wrong about many things in his rant. This being one of them.
Its not what is being discussed here. C++ has an implementation defined memory model. That just cannot work with Linux out of the box. You need an OS and compiler designed from the ground up to handle this.
C++ has with a close, approximation, the same memory model as C, both concurrent and otherwise. In fact C11 just adopted the C++11 concurrent memory model wholesale.
Arduino, ARM mbed, Symbian, macOS, Windows, BeOS, IBM i and z/OS accepted the work of C++ proponents.
There is nothing when can do to change political views.
And lets not pretend that Rust in the kernel won't suffer from creative uses of macros, and there is still a big laundry list of issue to fix before it actually makes it.
Or that for the time being, Rust compilers need to link to C++ code to actually work, at least until Cranelift is a match in code quality against LLVM or GCC.
So in the end Linus gets C++ into the kernel, even if indirectly.
Its notable that the main improvement over gnu98 C to C11 essentially templates, if a worse version of it with better compile time performance.
If they switched to c++17, and banned everything from throw and RTTI to the standard library, they would still get for each, auto, and templates and inheritance which does not have to be emulated in C.
> C++ was discussed as having too many undesirable characteristics.
I tend to agree, as a C++ developer. There are many core issues in the language that haven't been resolved and that are unacceptable for kernel code.
My personal pet peeve: C++ is unable to reallocate a new[] region. This makes basically all structures (vector, hashmap, trees...) unusable for large data handling.
Of those structures only vector benefits from realloc anyway. And if you really want it, it's not terribly hard to write. You can static assert that the template type is trivially movable which would make it safe to realloc.
But this is assuming you have a malloc implementation that does something other than implement realloc as just malloc+memcpy+free. Which not many do, not unless the allocation is so large as to be in its own dedicated mmap or similar.
That aside, sure would be great if you elaborated on these unsuitable, unresolved for kernel language issues? Exceptions and rtti are the only two I'm aware of and both have had off switches for decades.
> By the way maps often use a large bucket, and rehash in-place can be preferable.
std::unordered_map (what I'm guessing you meant by a hashmap) uses a linked list for the nodes. There's no movement in the first place to worry about being realloc'd.
> Do you know a modern operating system that does not have a mremap equivalent ?
You have to be very large before most mallocs will put you on a dedicated mmap that can even be mremap'd at all.
If you're working with stonking huge data inline in a std:: vector... Yeah just make a container for that usage, not really an issue. There's tons of examples out there, typically to add SSO but doing realloc would be the same basic thing.
Google (C++17 subset in Zircon), Apple (IO Kit drivers with Embedded C++) and Microsoft (since Vista), ARM (mbed) IBM (C++ as PL/S replacement) think otherwise.
That's basically the trial period. Nothing depends on drivers so they are pretty safe to try new things with and the hardware they are used for is limited so you can know you the hardware the driver is used for is supported by the Rust compiler. If we go a few years and its all good and the rust compiler supports everything linux targets, I can imagine more core parts becoming rust.
> Like that macro is just a crappy version of std::for_each.
It really, truly is not comparable. list_for_each_entry does a well known operation that's just moving some pointers around. Safe to do while holding a spinlock, or in interrupt context, etc.
std::for_each is a generic iterator implementation. What code runs in order to iterate? Will it transparently take locks, risking a deadlock? Will it need to schedule()? You don't need to ask those questions about the standard linked list macro, and that has a lot of value when every instruction the compiler generates might matter.
list_for_each_entry just does blind string substitution. std::for_each is the basic same thing except with type safety & variable scope safety instead. It's still just substitutions.
All your other stuff about for_each is just... Wrong? It's all well defined what it does. The substitutions that for_each makes and exactly what it calls are all defined.
You could give it a container that does something stupid, but you could also give list_for_each_entry the wrong field or accidentally use it when it's no longer valid (literally the bug being fixed)
Point taken on how for_each works for linked lists. I'm (obviously) not a C++ expert and it shows in that context.
I suppose the thrust of my opinion is that a lot of code depends on the fact that it iterates over a linked list, which is safe in many contexts. The idea that callers could substitute container types which break its assumptions seems unwelcome to me. Nobody provides the wrong fields to list_for_each_entry because they get very nasty compiler errors if they try. But anybody can plug a new container type into a for each statement and get new behavior that silently breaks the assumptions of the original code.
I suppose that could be protected by a better type system though. There's a lot to improve on C even for kernel dev, and "C is greatest" isn't a hill I'm willing to die on :)
> But anybody can plug a new container type into a for each statement and get new behavior that silently breaks the assumptions of the original code.
How would it be "silent"? You're changing the container at the point where you're making any assumptions about it. This would just fall under normal code review purview, just like code reviews are how Linux enforces that anyone even used this linked list macro in the first place. C certainly doesn't care if you use list_for_each_entry for your container iteration, after all.
But it'd be a lot easier to replace these linked lists with a vector that has the same observable contract if C++ was used, just would be way faster to iterate on.
{
iter= xs.begin();
while(iter++ != xs.end()){L}
}
Not exactly hard to see what happens. In particular as iter is typically just a raw pointer.
Regardless of if you use one or the other you must answer the question,
Should I take locks, etc.
With the macro you also have the concern that if any of the headers you include happens to change order of inclusions, the macro called may change without being noticeable. That is a harder problem.
Maybe that's helpful today, but by remaining (relatively) standards compliant the kernel codebase has a more long-term resiliency. The more tightly wedded to a single compiler implementation the greater the long term risk for the project.
If some shenanigans were to happen upstream in GCC it would not be the end of Linux.
> The Linux kernel has always traditionally been compiled with GNU toolchains such as GCC and binutils. Ongoing work has allowed for Clang and LLVM utilities to be used as viable substitutes. Distributions such as Android, ChromeOS, and OpenMandriva use Clang built kernels. LLVM is a collection of toolchain components implemented in terms of C++ objects. Clang is a front-end to LLVM that supports C and the GNU C extensions required by the kernel, and is pronounced “klang,” not “see-lang.”
That would make the code (and the compiler) non-C89 compliant. At which point, why not move to a different standard rather than bork both the kernel and the compiler if it can meet the needs better?
Alright, making it more non-compliant, then. Still creates problems for everyone and also necessitates a larger jump in the GCC version to support (unless you're going to back port the new extensions to every version that the kernel currently works with). Currently you can compile the Linux kernel with GCC 5.1. If you changed the compiler to support this one feature (or any other) you'd either need to back port it several versions (potentially back to 5) or abandon all of them. Which is probably a no-go with a project that's overall as conservative as the Linux kernel project.
I'm not a compiler expert here but when clang was new and one of the BSD distros was making a lot of hay about moving to it, I remember one of the arguments in favor being that the GCC codebase is a mess and not easily extended.
because currently gcc maintenance is almost "free" (in comparison to maintain gcc which is as big a project as linux) and they just have to work with the powers that be over at gcc. That's much better than forking it and having yet one more HUGE project to maintain independently. The advantages of sticking to the current policy FAR outweigh the disadvantages.
In may ways you cant avoid more modern C because more modern standards make thing that were ambiguous more clear, and then compilers follow that even if you select an older standard.
I think C99 introduces some bad things that should be keep out of any code base. Variable length arrays being the biggest one. Generics is another one. Being able declare variables anywhere doesn't make the code better but it makes styles diverge and that's a problem.
If Linux wants to adopt C99 they should define what parts of c99 should be adopted in their style guide.
> Being able declare variables anywhere doesn't make the code better…
It does, though. The ability to declare variables "anywhere" and allows a reduction in the portion of the code where a variable exists uninitialized, or initialized with a dummy value which should never be used, or holding an obsolete value which is not intended to be used. It also allows more variables to be declared `const` (which requires them to be initialized in the declaration, which must occur after the initial value is computed) which helps to detect accidental assignment and also makes the code easier to review.
It's worth noting that you can more-or-less declare variables "anywhere" in C89 too—you just have to wrap the variable's scope in a compound statement. From that point of view, C99 doesn't change where variables can be declared. It just lets you remove some syntactic clutter.
I for one have read a lot of code with C89 declare-first style that was subjectively a lot less clear. This applies in particular to assign-once variables that depend on other calculations.
stuff like this: [0]. I mean, what's the point of declaring variables like cp or err at the beginning when they're assigned first only halfway down? What is the point of having path around for the whole function, when it is a super temporary assign-once variable with a scope of 3 lines? I'm having a really hard time coming up with a justification for this, other than "this is the way we've always done it and we like it" (you can find actual rants in defense of this style from about 10 years ago).
For-loop declarations are another instance of this that make the code more fluent IMO. And they help reducing the scope of variables to a strictly smaller block. I would find it hard to argue that this does not remove some bugs or improve readability.
The only thing that declare-first has going for it is that the variables that are in use can be seen at once, a little bit like in a struct declaration. Looking at how optimizers butcher those variables depending on liveness makes this feature seem less valuable, though.
Couldn’t they solve that macro issue using C89 by just having the macro create a block around the loop, and then declare the variable at the top of the block?
The amount of fear I see from people around C is so surprising to me. Not saying you fit into this, but so far my anecdata suggests that it's largely people who don't know (or know very little) C. Especially among CS grads whose main exposure to C was in the stack smashing exercise in a security class, all they know about C is the part that was intentionally made vulnerable so it would be easy to exploit. Unless you're being wildly negligent and reckless with your programming, C is really not that scary.
What percentage of non-C applications did you review? I suspect C has enough footguns to be an issue, but its popularity, especially in low-level software (kernels, codecs, firmware) ensures that it'll show up in security issues regardless of how bad the languages itself is.
I reviewed mostly C, then reviewed mostly Golang. But I also reviewed codebases in Erlang, Perl, Java, C++, Rust, Python, Javascript, etc.
Mind you I was mostly reviewing cryptographic-related applications, but most C applications contained bugs that had nothing to do with the logic (lots of memory corruption bugs) while most Golang applications contained logic bugs. Or at least I would find logic bugs because Golang was both rid of most memory corruption bugs, and also an extremely readable language (so easier for me to understand the code and find logic bugs). Although Golang still had nil dereference bugs (happens a lot when people used protobuf), because they don't have sum types. Today I think a great language would be a mix between a readable language like Go (with good defaults, toolings, stdlib) and a safe languages like Rust.
There's a difference in GC languages though. strongly typed languages like Golang will always be more secure than dynamically typed languages like Python.
Interesting, I had the same job in mid to late 00s, although I wasn't a consultant so my sample was the company's codebases (of which there were a lot because we built a lot of embedded systems on top of vxworks that did a lot of network communications, sometimes in very niche protocols), not necessarily the codebases of company's that are worried enough that they hire a consultant. That was right around the time when compilers and security tools were becoming available that could flag nearly every possible problem. At that point false positives became a big challenge.
What years were you a consultant reviewing C applications?
I'm guessing you were using tools like coverity? I actually never used such tools. I mostly did manual reviews and sometimes implemented fuzzers with AFL. But most of the code I looked at was crypto code. Did that at Matasano/NCC Group from 2015-2019
it's been 15 years so I don't remember the names of the tools, but coverity rings a bell. There was one that we used to make fun of a lot because it was written in Java, but it was by far the best at finding stuff. It would even show you the AST to help point out problems. I'm suddenly feeling really nostalgic about GUIs written in Swing and SWT :-D
Definitely. Though Linux code has probably been tested more thoroughly, and run through more static analysis, than any other C code base. That does help me sleep a little better at night.
And yet, the most significant part of the C code issues we find is memory corruption. Which is either significantly harder to cause or impossible-by-design in many alternatives. Unless you can realistically say "people working daily, for years, on huge C projects write those bugs, but they're reckless and I'm better than them" - yes, C should be scary to you these days.
We can't even agree on safe string functions for C, half a century later. You shouldn't have security bugs baked into the standard library and you shouldn't have to do a mountain of research to know which functions are safe and in which cases.
However, for most things non-string, non-pointer, and non-array, I agree with you.
It's easy to say "You shouldn't have security bugs baked into the standard library" but it's a lot harder to say, "we're breaking decades worth of working code by removing some functions that have been part of the standard lib and were widely used."
We don't even have a string library in C. Strings are Unicode, not just zero-terminated buffers. You cannot find strings, nor compare them. In the kernel you have filesystems and login systems using unidentifiable names. Because the kernel has no identifier support.
And for insecure standards, the committees rather want to eliminate the safe functions, than fixing the spec bugs or add u8 support.
I've been programming in C for 20 years, and I think it's scary. Not because I don't know it, but because I know exactly how many cases of seemingly normal code can hide UB. I know from experience that even the best programmers following best practices will make mistakes (or run into someone else's). C is an extraordinary amplifier of bug severity. I know how much diligence, effort, and tooling it takes to merely not screw things up in C.
I've seen time after time people saying "nah, C is fine, you just avoid this and that, use these tools, etc." and this turning out to be insufficient. I've heard many times "maybe you're just a bad programmer and can't handle C, but I'm a good programmer and have no problems" and their code not surviving 5 minutes of fuzzing. I've seen people conclude that multi-threading beyond simplest constructs is just infeasible to get right, and think that's an inherent property of threading, and not fragility of C.
Yeah, I find it astonishing to find C programmers that see no problem, even though I think it would be reasonable to say that they see no reason to change. It's mind boggling.
Have they not worked on large projects with other developers? Have they not seen the myriad of ways things can silently go wrong for seemingly no technical benefit? (although I know there's often less obvious reasons, for eg. UB, performance, platform specificity, etc.)
All that, said I do think the following might be reasonable:
> "nah, C is fine, you just avoid this and that, use these tools, etc."
I guess you mean they write off the risks entirely? You should never be this "handwavy", and should always take the risk seriously, especially with a language like C. However, I think it's fair to say, that C is a good choice, many of the risks can be mitigated, and it's not THAT big a deal. In which case, the above doesn't seem that absurd.
Following basic common sense and making an effort to identify and eliminate some of the sketchier situations, backed by some really good integration testing, can really help. I feel reasonably safe under those circumstances (I mean not really, but other languages can be "unsafe" too). A huge chunk of the really evil things I've seen have been the result of taking absurd risks, and/or disregarding the rules entirely. If you were paying attention and "trying" to write good C code, they would never happen; these aren't just individual developer things, but project wide. Eg. I had a compiler that didn't even warn for implicit functions... jerk move by TI, but that should be flagged and dealt with. Instead, the team just thought "great no compilation errors".
I will say there's a lot of developers that are just uninterested in any of this and will deliver some really, really sketchy C code. In their mind they're smart programmers and their code will just be right, and they don't seem to understand any of these issues. Just plow ahead and patch around the bugs, then move on to the next gig.
Skilled C programmers make these mistakes day in day out, we write too much code to trust a language that just let's it happen. Especially one that encourages the writing of code multiple times rather than reuse.
I don't know about that. I agree about it being a popular idea with people that don't understand C. Plenty do and still harbor those opinions. I've been mortified by what I've seen in C, and it seems really preventable.
Lots of developers out there are widely negligent. C provides them with enough rope to hang themselves. To be honest, I'm a little surprised to find veteran C developers that AREN'T scared. I guess they just see every disaster as the fault of "negligence and recklessness". If you're not scared of your code (mistake), you should certainly be scared of other peoples.
You shouldn’t - it isn’t perfect, but it is well understood. So far we don’t have anything else that works better and all the edge cases and weirdness is understood by enough people that someone could build something like Linux on.
It is unwise to build a bridge on something untested with unknown failure modes, and it is equally unwise to do a rewrite or create new core infrastructure in a language without knowing it as well as a civil engineer knows concrete.
That makes some intuitive sense, right? The fact that it got "old" in the first place indicates that it's not being touched a lot, and if it isn't being touched a lot that means that bugs haven't been found, meaning that the bugs that are in there are especially sneaky edge cases, or there simply aren't any large bugs to begin with.
This is kind of one of those issues that can really cut both ways. I don't think it's the best attitude to say, "it's been working for years, so it's fine"; there could be subtle bugs, and areas rarely exercised. Still it often holds true. We all remember times we've meddled with something and messed it up. It seems that some of the low level code has been really heavily used in many different ways, and seems to just work. Especially if it's not safety/security critical (and maybe even if it is), it could be a poor use of resources trying to fix something that isn't broken.
I had this argument with some of my management. There was a push to upgrade all of our libraries that were deemed "old" with no other criteria than "it's outdated." I can appreciate shiny new toys, but if you're not hitting bugs and things are stable, I'd rather put my effort into adding features to our codebase and not chasing down library bugs.
I haven't really touched non-GC'd languages in quite awhile, but I feel like modern C isn't that unsafe, at least from the bits I've played with it; it can even have a garbage collector if you want it [1](which I usually do).
It's worth giving it another try if you haven't in awhile, if for no other reason to understand what's going on behind the scenes of your abstractions in Java/C#/JavaScript/etc.
I actually find it reassuring. C is the most mature and proven language out there. With proper standards in place (best example probably NASA), pitfalls can be avoided. Another example that immediately comes to my mind is Redis, which is such a great and stable piece of software. Java is similar mature, but not the right tool for infrastructure.
I always strongly disliked the kernel's approach to macros (especially how they try to masquerade as everything but macros with their lowercase names). More than once I wasted time debugging code because I didn't realize that a macro was involved (special mention for the cases where depending on the compilation flags a symbol can sometimes expand to a macro and sometimes something else).
I suppose in the case of list handling there's not really a good alternative though. If C had lambdas we could use inline functions instead but that's not a thing. Still think it ought to have been uppercase though.
>Raising the minimum GCC version to 8.x would likely be more of a jump than the user community would be willing to accept at this point.
If you are using a 0 day old kernel. Why would you be using GCC 5.x still. 5
1 is almost 7 years old now. 8.1 is almost 4. Can't people just use apt / yum / rpm / whatever to upgrade to the latest gcc? Is that really too much to ask of the community?
I get the impression that a big part of it is just a general choice to be very conservative about forcing newer minimum versions. So to raise the version bar you have to make the positive case for why it's worthwhile -- merely "gcc 5 is ancient" doesn't suffice. The only reason it moved up from 4.9 is actual data-loss-provoking codegen bugs in 4.9 (see discussion in this lkml thread: https://lore.kernel.org/lkml/CAHk-=wjqGRXUp6KOdx-eHYEotGvY=a... )...
I do think they could move up a bit further, but it's useful to be able to just build kernels with the distro compiler and repology thinks that for instance Debian stretch (still an LTS supported version) is only gcc 6.3.
The distro kernel will stay on the LTS branch, certainly -- but the upstream kernel folk like people to be able to compile upstream kernels and not have to stick with the distro ones. (I've done it myself on occasion and i'm not a kernel dev; if I'd had to also get hold of a new gcc it would have been an irritating extra step.)
The Linux community is way way way larger than just the sum of distro user bases.
Even if all distros have long ago moved to GCC 8+, you still have other chips and systems running, and who knows what buried somewhere depends on GCC 5.
You'll have to upgrade the compiler eventually though, right? I don't know what that process would look like, but generally the longer you wait the more pain you'll face later.
Compiler tech and hardware will improve drastically in the next 100 years. Linux can either evolve with it, or be put in a museum while something else takes it's place. Likely losing a lot of the development that's been done in the past 30 years.
It would be like running OS/8 (for the PDP-8, popular in 1965) on a modern machine. No time sharing for processes, no network stack, only a TTY for I/O... what would you even do with it if you could compile it?
We’re clearly talking about very different things.
If someone made a product which uses it, and people keep buying it, most companies aren’t going to bother with any of those things if they don’t need it.
That is of course completely different from what Linux is doing (as in outside of that context). It may be at version 510, but that doesn’t mean someone won’t be running version 1.0 somewhere. And that’s fine.
For an unstable branch maybe, but if people need to actually use this for real work they will want it compiling on something as well understood and baked as possible - while not being completely obsolete.
People doing real work are using a kernel compiled with a more up to date GCC. From what I can tell online even RHEL is up to date enough by compiling its kernel with GCC 8.x.
Officially yes, although we can argue about how much you can "backport" and still count. But even so - if new code requires a newer C version and thus newer compiler, it's harder to backport.
You'd be surprised how much downstream grief a bump like this can cause in an ecosystem as broad as the Linux kernel. Dealing with the results can be unexpectedly overwhelming.
It makes sense to play it safe and do this in smaller incremental jumps.
It's not just a question of what the new version fixes, there is also the question of what it breaks. Sometimes new versions trigger latent bugs no one knew about that luckily happened to work fine on the old tools. Finding and fixing those can be difficult and time-consuming.
This argument would have merit if everyone still used 5.x to compile Linux, but that's simply not the truth. Most people are using a Linux compiled by a somewhat up to date version of GCC.
Given C's ubiquity and (likely) longevity it'll have a Y2k89 problem: standards released in 2089 and beyond will have to change their naming schema and/or avoid being standardized in particular years. ;)
The C standards aren't published at a regular interval of 10 years. They are published as needed or as agreed upon. So it is possible there may be a C standard published in 2089, but highly unlikely.
It's similar to but not quite the same thing as the Y2K problem.
I wonder what the implications of this are for Bootstrappable Builds. I guess they will just need a longer GCC versions bridge before getting to the step of building Linux.
The reference to moving into C99 is kind of interesting, given the security work paid by Google to remove all uses of VLAs, I thought they were already using it anyway.
This has happened to me a number of times! I think time of day has a heavier influence than we might think. Submitting is only part of the algorithm. Both vote count and velocity of votes are really important too.
I hope this isn't too much of a derail. I'm not proficient in C, and am translating a C module to Rust. My biggest complaint is there's so much DRY! Functions, variables etc are all prefixed with the module names, presumably due to lack of namespaces. The `.c` vs `.h` dichotomy is also DRY. The subset language that uses `#` is also a bit of a mess.
You're using DRY oddly, but I presume you mean "there is so much duplication" (which is the opposite of DRY)?
For the .c/.h part, it's because of use (or intended use). Header files are meant to be specifications (though a lot of implementation ends up in them, annoyingly). It actually is an example of DRY thanks to C's need to forward declare things. You can't call a function the compiler doesn't know about, for instance, so you declare (not define) it in the header if it's defined in an external module. If you didn't have the headers, you'd need to copy/paste or handwrite the declarations everywhere.
You can call a function the compiler doesn't know about, but god help you if there happens to be an identical symbol in the table; it will link to literally anything, including an integer. This should pretty much always be turned off by setting implicit functions to be errors.
Also, this explains all the prefixing that OP was complaining about lol, or at least that's one reason. I saw a particularly bad bug, due to poor naming decisions. Less is not always more, even if it annoys you.
Why do you think there needs to be a declaration beyond the definition? The definition is enough information for the compiler, and having both is a DRY violation.
Header files are a historical legacy from when C was developed. Because machines were small compilers/languages were designed to do single pass compilation on modules which were then linked.
Friends of mine who are actual CS majors say with C it's not fixable without changes to the language. With Java or C# the compiler can do a pass and extract the object definitions. But because of ambiguous syntax and majorly because of the preprocessor it's not possible with C as it stands.
You need the declarations because the compiler requires it, it's not something that I just thought up. If you don't, then you get two different declarations for bar in the following. The one you intended (char bar(int)) and the one you didn't (int bar()):
int foo(int n) {
char c = bar(n);
...
}
char bar(int n) { ... }
The definition is not enough information for the compiler in a case like this, which is also essentially the case for any externally defined function. The compiler doesn't know anything about those unless its declaration is provided, which is what's in the header files.
Did I say there were two definitions? I said there are two declarations. Try using a function with a return type other than int before it has been defined or declared in C and see what happens when you compile it.
The function is declared/defined/described/written once, and used once. There is no ambiguity.
I don't understand your point about using a function that hasn't been declared. Of course it won't work!
I understand your point about using header files as interfaces to third-party code, but there are ways of doing that that don't involve duplicating all functions, structs etc.
The function is used once, and because it occurs before the actual definition (and there was no forward declaration) an implicit declaration occurs (the erroneous int bar()). When the function definition is later reached by the compiler, its signature doesn't match the implicit declaration's signature, causing the problem. If the function signature matches the implicit declaration, then there is no problem:
int main(int argc, char* argv[]) {
printf("%d\n", foo(10));
return 0;
}
int foo(int n) {
return n - 1;
}
Will work just fine, giving you only a warning but will print out "9" as expected.
> an implicit declaration occurs (the erroneous int bar()
I still don't understand; why does the implicit declaration assume int bar() is the signature when bar is being passed an int and returning a value to a char?
You don't need type inference to handle this correctly, you just need to wait until you've parsed the rest of the file and seen the signatures of all functions.
I haven't touched C in any serious capacity in quite awhile; how often does C get new revisions, because isn't C11 still more than a decade old?
I understand why the linux kernel folks aren't moving to the bleeding edge (Linux is old, you have to do these ports incrementally), but I'm curious why the pace of language is so slow. I guess it's because C has more or less stabilized and thus further revisions are less necessary?
The 1995 one was an "amendment", not a full revision (and is mostly additions to libc, which Linux doesn't get to use; the only language change is the addition of digraphs).
C17 was a "bugfix" revision; compilers will have applied these fixes to their C11 implementations, so in effect the only difference between C11 and C17 is the value of __STDC_VERSION__. So for most intents and purposes C11 is still the most recent revision.
Fair enough! As I said, I suppose C doesn't need to change that much; people use C because they know what they're getting, and for that to be the case, it needs to be stable.
If you've ever done CI/CD for a large project in a fluid development ecosystem (e.g. nodeJS), you can understand why it might be refreshing to develop your operating system in a language where standards are measured by the decade.
> but I'm curious why the pace of language is so slow. I guess it's because C has more or less stabilized and thus further revisions are less necessary?
That is a big part of it. C is an old, stable, complete language.
There is also the existence of gnu extensions, which bridge the gap between language revisions. For example, anonymous unions were added in C11 but they have been a gnu extension since forever.
Given that C99 is much closer in vintage to their current C89 than it is to today's date and also just straight up really old, it does seem inappropriate to distinguish that as "modern C".
That said they do seem to be discussing both C99 and C11 so it does seem like going with something that would qualify as modern (especially in light of Linux's need to be conservative) is actually on the table.
You also have to look at the contents of the revisions. C99 was a massive update for C, and C11 was tiny in comparison (and C17 was basically nothing at all).
Writing C89, without variable declaration in a for loop, and without // comments, etc, feels ancient. C99 on the other hand is what defines how C looks today.
This is also why the attitude is: if they jump to c99, they may as well jump to c11 because they are basically the same.
> the objective was not to rewrite the kernel’s 25 million lines of code in Rust, but rather to augment new developments with the more memory-safe language than the standard C normally used in Linux development.
> You are not "introducing" a new macro for this, you are modifying the existing one such that all users of it now have the select_nospec() call in it.
> Is that intentional? This is going to hit a _lot_ of existing entries that probably do not need it at all.
> Why not just create list_for_each_entry_nospec()?
Let's ignore whether this patch is needed or works for now - I don't feel competent to comment on that. But this is such a bad suggestion. Instead of fixing the default to be safe, and possibly having a _unsafe_but_fast() variant for the places where it makes sense they want to keep the broken version and require user to explicitly opt in into safety.
Same as the infamous PHP mysql_real_escape_string (don't make the mistake of using mysql_escape_string!) [2][3] or whole host of C stdlib footguns like strcpy/strncpy [4].
The default, easy, obvious option should be safe. The unsafe but faster option should be hard to use by accident and obviously marked as such from the name.
[1] https://lwn.net/ml/linux-kernel/Yg6iCS0XZB6EtMP7@kroah.com/ [2] https://www.php.net/manual/en/function.mysql-escape-string.p... [3] https://www.php.net/manual/en/function.mysql-real-escape-str... [4] https://en.cppreference.com/w/cpp/header/cstring