As someone who doesn't know a lot about distro work, what exactly did you have to do for Fedora? Fix packages that weren't building anymore with this change?
Packages fail to build after a while all the time. The maintainer fixes them (preferably using an upstream patch), and we move on.
The issue with this set of compiler changes was that a lot of software builds successfully, but the new build suddenly has expected features missing. This happened with the previous attempt:
That happened in 2016. In 2019, we undertook another attempt, this time with config.log/config.h diffing, but for various reasons stopped after fixing a few dozen packages, so it never went to the Fedora/upstream proposal stage.
After Xcode and Clang the changed the defaults, it was somewhat easier to convince people that we should get Fedora & upstreams ready for a future GCC change. It also helped that Gentoo has been pushing fixes for Clang compatibility upstream for a while (https://bugs.gentoo.org/408963, with a broader scope in https://bugs.gentoo.org/870412).
Yep, errors result in autoconf checks failing leading to features being disabled, symbols being removed and more all while the build still succeeds successfully.
This is a long-needed, but gratefully accepted improvement to GCC. I've seen in the wild just how many people see a 'warning' and consider it ignorable, even though many warnings in GCC are certain indication of broken code in pretty much anything new written in the last 35 years.
It's important to bear in mind the main change here is to stop wild C programmers from trying to run or release code that's almost certainly broken, because the diagnostic they got wasn't explicit enough. People can still force the old behaviour if they need to.
> It is no longer possible to call a function that has not been declared.
Heh. Surprisingly, I've seen some people claiming that it's impossible for a language without forward declarations to have a one-pass compiler that could handle calls of not-yet-defined functions, yet pre-ANSI C managed to do exactly that, in a pretty obvious (now that you know that it's in fact possible) way.
In the C compiler I wrote a couple years ago (ImportC), since it was re-using the D semantic routines, forward referencing functions "just works".
In fact, it works so well you don't even notice it working. If the C committee wants to improve C, they should make this an official feature.
P.S. Because of the lack of forward referencing, C code tends to be organized as leaves first, and the entry point at the end. This is simply backwards, the entry point should be at the beginning.
Are there any compilers our language that parse source files in reverse order? Then you could have a one-pass compiler without forward declarations (though you would need “backward” declarations). :)
Someone shared with me their idea for a parallelized parser, where threads would parse different segments of the source file and then stitch their incomplete ASTs together.
But prototype-less functions in C don't actually work all that well. In particular, the inferred return type is going to be an int, so if you're trying to return a 64-bit pointer via a prototype-less function, you're going to have a very bad time.
Historically they worked well enough (on 16-bit and 32-bit machines) because sizeof(pointer) == sizeof(int) == sizeof(general register) on the architectures where C flourished in the pre-ANSI C era.
But with the migration to 64-bit machines, typically int stayed put at 32 bits.
I guess nobody wanted to introduce a new integral type between short and int; they had enough trouble dealing with code which assumed sizeof(long) == 4. I recall stumbling across a comment where the word "beint32_t" appeared where "belong" would have made sense in context..
I love naïve search and replace errors. In the November 1996 version of the Defense Incident-Based Report System codes definitions in DoD Manual 7730.47, the code 092-C2 refers to "shallful" dereliction of duty.
Why not just make short 32-bits? Yeah, you lose the type for the 16-bit wide integers but x64 doesn't natively support it all too well anyway, unlike the 32- and 64-bit wide integers. And that is what the C integer types are about, right, about being efficiently represented by the underlying hardware, not their exact bitwidth? Right?
> And that is what the C integer types are about, right, about being efficiently represented by the underlying hardware, not their exact bitwidth?
If you have one `short` argument to your function maybe. But if you have a `short[]` array, you probably do care about the memory layout of that array. You might need it to be compatible with some particular data format that you're trying to read/write. Same with a field of a struct, if that struct is used for parsing. A lot of C code does parsing like this.
Well, that's a shame because bitwidth of standard integer types is quite uncertain. CHAR_BIT can be (and is, on some platforms) 16 or 32, long was never guaranteed to be 64 bits (it's quite often was 32 bits on platforms with 16 bit ints) etc, not to mention that if that is what the standard integer types were for then they'd probably have names like int8/uint8/int16/int32/etc.
It's almost as if they were not, in fact, intended for precise control of bitwidths in portable manner...
It doesn’t matter what someone in the past thought they were for. That’s what they are for in practice, names are irrelevant here, yes, they are quite bad. But the ones in stdint.h are just typedefs for those, so that’s what we are left with.
You can always use "unsigned char[N]" for that, you know, which is more realiable. You can even union it with an integer type for more convenient access, although please use static_assert to verify that the overlap is exact.
All in all, "it's very easy and straightforward to control the size and padding of a struct's fields in portable manner" is yet another C's imaginary advantage: it's not that straightforward or simple. The padding especially has always been a thorny issue.
The efficient hardware types are handled by int_fast*_t. The legacy types can't be redefined outside their established ranges because that would break things that depend on them fitting into a known amount of memory.
Well, there is an argument to be had that, for C, a plain unadorned "int" should probably be the native word size for that architecture. On 64-bit systems, "int" would therefore also be 64 bits.
I think there are two reasons int has not gone from 32 to 64 bits on 64-bit systems.
Part of it is backward compatibility: code written to assume 32-bit int could break. (Arguably such code is badly written, but breaking it would still be inconvenient.)
Another part is that C has a limited number of predefined integer types; char, short, int, long, long long (plus unsigned variants and signed char). If char is 8 bits and int is 64 bits, then you can't have both a 16-bit and a 32-bit integer type. Extended integer types (introduced in C99) could address this, but I don't know of any compilers that provide them.
Since the extended integer types are just aliases to the other types, they wouldn't solve the problem. Also in C++ these aliases create a problem with overload sets when you mix the two worlds and try to produce portable code. For example long on some platforms is 32 bit and 64 bit on others, also platforms use inconsistent fundamental types for 32 and 64 bit aliases. All in all if you want to produce portable code you neither use those extended integer types nor long. You assume char, short, int, long long are 8, 16, 32, 64 bit respectively, which holds on all relevant platforms.
Extended integer types are decidedly not just aliases to other types - the C standard has separate "standard integer types" which are the regular char/short/int/long/long long, and "extended integer types" which are any additional implementation-specific types. stdint.h-defined types can be either of those categories (and on regular clang/gcc they're all standard integer types and not extended ones). So you could have a system with char/short/int/long/long long being 8/16/64/64/64-bit respectively and still be able to provide an int32_t that's none of those; just, noone does.
> Extended integer types (introduced in C99) could address this, but I don't know of any compilers that provide them.
What environment are you working in? Because I don't know a single half-recent compiler that does not provide stdint (uint8_t, ..., int64_t), but I mostly work with GCC/LLVM toolchains.
Some embedded compilers will provide stdint. And if the compiler doesn't, I've found that one of the first headers written for a project ends up being an equivalent to stdint.
It's pretty common to develop part of an embedded C program under Linux or similar host environment. Better debuggers, better profiling tools, etc. And uint8_t and friends are particularly important when you're working cross-platform.
Extended integer types aren't necessarily related to stdint.h - in the vast majority if not every one of those "half-decent compilers" the stdint.h types are just typedefs over plain old char/short/int/long/long long, which are not extended integer types. Extended integer types is a mechanism to allow having types other than those.
> in the vast majority if not every one of those "half-decent compilers" the stdint.h types are just typedefs over plain old char/short/int/long/long long, which are not extended integer types.
Sure, but isn't that just an implementation detail? Because I really don't care if my int64_t is internally typedef'd to "long long int" or "__m64", as long as there is a standardized interface to ask for it.
Point being, _kst_'s comment of "I don't know of any compilers that provide them" is correct - there are few if any compilers that actually have actual extended integer types, and thus introducing such in compilers might be non-trivial, and plenty of code may exist under the assumption that they don't exist and thus could break (things like integer promotion rules, _Generic, varargs; and also intmax_t is of mention as it must be at least as wide as any supported standard or extended integer type (which is also why clang/gcc __int128 doesn't qualify as an integer type as per the standard))
If 32-bit ints didn't make your structures consume far, far too much memory in the early '90s, when consumer-grade computers came with 4-8MiB of RAM and 256MiB disks, then I don't see how 64-bit ints could have done so in the mid-'00s when they came with 1GiB of RAM and 256GiB disks.
OTH, the explicitly sized integers (int32_t, int16_t etc) had been added already in C99 (e.g. 25 years ago), and should be used when a specific memory layout of structs is desired.
If you're going to use those, then there's no reason to have ints be 64 bit.
Personally, I find using int32_t in general to be an uglification of the code. I never use `long` in C code anymore, as it's sometimes 32 bits and sometimes 64 bits. I use `int` and `long long`.
Do I care about 16 bit code anymore? No. Very few programs would port between 16/32 these days anyway, no matter what the Standard says or how hard you try to write code portably.
Is it the numbers you do not like? If you use “long long”, it must not be the length.
I don’t love long long. As an amateur compiler writer, it hurts me. “long long” makes “long” both a modifier ( like unsigned is ) and a type. Yuck.
I wish it was i8, i16, i32, and i64 ( with u versions of each ). f32 and f64 for floats. Those are easy to understand and fairly easy on the eyes.
If those numbers are too noisy, the CIL ( .NET ) types could work. For example, i4 and i8 instead of i32 and i64. I do no love the look of i1 either though. I guess you could special case sbyte and byte as aliases.
"long" is always a modifier, just potentially applied twice, and potentially to nothing. A more written out version of "long long" is "long long int", and similarly "long" is really "long int".
Google’s C++ style guide similarly recommends using ‘int’ in general and never using ‘int32_t’, though it recommends using ‘int64_t’ for bigger numbers instead of ‘long’:
When the first C++ Standard (C89) was being created, about half the compilers implemented "sign preserving" semantics, which is what you're advocating, and the other half implemented "value preserving" semantics.
A great battle ensued, and many champions were slain.
The value preserving folks carried the field, and the sign preserving folks changed their compilers.
I was always a little surprised C never had an integer type the size of a native word. Probably the closest thing would be intptr_t since pointers presumably use a single word to represent addressable memory.
AFAIK until the switch to 64-bit architectures, int actually was the natural word size. Keeping int at 32-bits was probably done to simplify porting code to 64-bits (since all structs with ints in them would change their memory layout - but that's what the fixed-width integer types are for anyway, e.g. int32_t).
In hindsight it would probably have been better to bite the bullet and make int 64 bits wide.
> AFAIK until the switch to 64-bit architectures, int actually was the natural word size
32-bit int is still arguably the native word size on x64. 32-bit is the fastest integer type there. 64-bit at the very least often consumes an extra prefix byte in the instruction. And that prefix is literally called an "extension" prefix... very much the opposite of native!
IMHO it's only weird because the promotion is to 32-bit integers on 64-bit platforms. If all math would happen on the natural word size (e.g. 64-bits), and specific integer widths would only matter for memory loads/stores, it would be fine.
> pointers presumably use a single word to represent addressable memory.
Only in flat address spaces, which excludes platforms like old 16-bit x86 or modern CHERI. There, "pointer difference within a single object" need not be the same size as "pointer reference".
I would argue that the native word size is still 32 bits on x86-64, though. With many instructions, using 64-bit registers needs a segment prefix override. Some RISC architectures do not even have 32-bit zero-extending integer instructions, so for them, 64-bit as the native word size makes sense. On the other hand, I'm not sure if <stdint.h> and uint16_t were invented at the time, and “short short int“ is not valid syntax (even today), so there wasn't an obvious way to denote a 16-bit integer type.
If the programmer didn’t specify the size of an int, it should mean “dealer’s choice.” Let the compiler pick a default, better yet make it a compiler option.
Yes, they don't; it would be possible for a compiler to gather info about such call sites into a list and then, when it's finished compiling a compilation unit, to check this list against the now-complete symbol table and fail if some called functions have mismatched definitions or are still undefined... that's apparently was too much work back when C was designed so we have "hope for the best" design instead.
This doesn't work in C because the function is not necessarily defined in the same translation unit.
You'd need to do this at link time instead, which would require completely overhauling the format of object files, dynamic libraries, static libraries so they carry information about the types of functions instead of just the symbol names. It's not an easy fix.
> This doesn't work in C because the function is not necessarily defined in the same translation unit.
It does work in C ― that's what the include files are for (among other things), after all. So it's possible to be able to forward-declare functions outside the translation unit only (extern-declare?), those inside the translation units don't need to even if the compiler works in a single pass. And those external declaration could still be introduced at the very end of the translation unit and still would count. I dunno, seems like a pretty reasonable idea.
That's what DWARF is for, and similar formats on other platforms. Also Pascal-lineage languages commonly use this kind of embedded type information to provide module interfaces, it works quite well.
There is no information about the function's prototype in the object file, so the linker can't determine that. The compiler can't either since it's just compiling the compilation unit and has no knowledge of the outside world except what we tell it by prototypes (injected via header files or otherwise), but we aren't doing that here.
> There is no information about the function's prototype in the object file, so the linker can't determine that.
That that information isn’t there is an implementation choice.
For example, they could have hacked it in like C++ did by mangling names (https://en.wikipedia.org/wiki/Name_mangling). That probably would have required supporting longer identifiers (IIRC archives limited them to 14 characters), but that’s doable.
No, you emit all the code in one go, then after you've done that you have some residual pieces of data left one the side: the symbol table, and the list of all the calls of (hopefully)-forward-declared functions. At this point you could run the check on that list against the symbol table, no additional codegen or re-reading the source text needed.
Granted, you can call that a second pass although that's not that different from emitting a function's epilogue IMHO.
There is no need for the second phase. You can record the implicit declaration when you encounter it, and if there is a subsequent declaration, you can check that it's consistent or error out immediately.
This is what C compilers already do, in fact, to produce warnings when an implicit declaration doesn't match a later explicit declaration. But this is a best-effort warning only; it doesn't work if there is no declaration because the function is defined in a different translation unit, as I pointed out above.
Usually, the second pass in a compiler does not re-parse source files. Rather, it operates on another data structure, like an AST, intermediate representation, or the list mentioned in the original comment. At least, that’s my understanding of multi-pass compilation.
Well, this "list mentioned in the original comment" is not an AST or an immediate representation of the program in any reasonable sense just as a symbol table is not. Otherwise, setting the exact values of the text/data/bss size fields in the executable file header's at the end of the compilation would count as the second pass as well which IMO it should not.
The difference is that you need to make a complete second iteration (or pass) over the entire list to correctly check all of the callsites after all function type information is collected. The same is not true for symbol table usage in a single-pass C compiler.
Why surprisingly? In pre-ANSI C is it only possible because there was no type safety. In other languages it could work by deriving the type from the call, but this does not work in C.
>The reason for that is that C does not offer a generic function pointer type, and standard C disallows casts between function pointer types and object pointer types.
I've wondered why this hasn't been relaxed a bit when void* is used. Say you have these functions (modified from the link)
And then you have some FP of type int(*compare)(const void *a1, const void *b1) somewhere...
If you have correct pointer const-ness, why isn't the first method the PREFERRED way of doing this? It's shorter, more clear, safer (calling compare directly and not through a FP still gets you proper type checking (you wouldn't call either of these example functions directly, but for others you might)), and IDE suggestions can better explain what the function is. I've thought several times about suggesting to compiler writers/the C committee to bless the first method. Is there some obscure hardware that the first would be incorrectly compiled or something (i.e. the calling convention for void* and foo* is different)?
A pointer to void shall have the same representation and alignment requirements as a pointer to a character type. [48]
<blah blah pointers to qualified vs unqualified, structs, and unions also have the same representation between themselves>.
Pointers to other types need not have the same representation or alignment requirements.
[48]: The same representation and alignment requirements are meant to imply interchangeability as arguments to functions, return values from functions, and members of unions.
The thing I've heard for systems where e.g. an int* and char* aren't equal is where the "int*" is the "primary" pointer type, and a char* has to add back the would-be-leading-bits or something. Though as per the above quote, char* vs void* would be safe? Doesn't help anything other than char* though.
I am not sure I understand what you say (despite having read your message several times), but pointers don't have all the same size. Not always, at least.
(void* is supposedly big enough to contain any type of pointer, even if I think I remember I read that it's not necessarily true for function pointers).
And if the size of "void" and "char" aren't the same you cannot push two void* on the stack and pop two char*.
But, like I said: maybe I didn't understood what you said.
I was wondering if pointer size is still a problem. Hopefully this explanation is clearer:
int compare_char (const char *a, const char *b) {
return strcmp(a, b);
}
int compare_void (const void *a1, const void *b1) {
const char *a = a1;
const char *b = b1;
return strcmp (a, b);
}
int (*call_void)(const void *a1, const void *b1) = &compare_char; //not valid in current C standard
Why can't the standard be changed so that it is valid to set call_void to &compare_char, so we don't need to write the longer compare_void? Are there architectures are still in active use where not all pointers are the same size in plain C (not worrying about C++) that would disallow this?
AFAIK, x86, ARM, MIPS, SPARC, Alpha, and SuperH would all work fine calling compare_char though call_void. There could be some other issues, like near/far on 16-bit x86, but that would be orthogonal to implicit function pointer void* casting, and could still occur if call_void was set to compare_void. Do one of the more obscure embedded CPUs still in use have a varying pointer size?
Tbf, all of those changes should not be surprising to any C programmer for at least the last two decades or so. I'm actually surprised that GCC was so lenient for so long.
It wouldn't have been surprising to make this the default in the 90's even. The things here are pretty much never done intentionally post-standardisation. And it's easy to force these back to being just warnings to make old builds work if needed.
The only annoying thing really is the second strict aliasing example, requiring the temporary variable. It just seems really inelegant. But it has been a pitfall ever since, since using an explicit cast removes the warning (now error) but still breaks strict aliasing.
I think the most pragmatic solution is to just compile everything with no strict aliasing. You still get errors for accidental incompatible assignments but won't get bitten by the optimizer.
And still not even mandatory if you use -fpermissive or -std=c89.
I don't envy how much tech debt they have to deal with, while still trying to make it possible for new code to have a more modern experience. For all the excitement around new languages, we're still going to have a lot of C code for a long time, and difficult work like this should be appreciated.
I can't help but wonder how many bugs could have been detected and fixed a quarter of a century earlier if those lints had been adopted more diligently by the C ecosystem.
My perception is that more and better warnings, and common use of -Wall, came into vogue around the time of GCC 4, in the mid 2000s. It would have been pretty abrupt to immediately enforce C99 in 1999-2000.
Yet in 2022 this change from warn by default to error by default was consequential enough for Fedora:
>> So what is the current status? How many packages are going to be affected by this change? How are we going to track the progress?
> I see an unaudited rebuild failure rate of about 10% for rebuilds of source packages that produce arch-full binary packages. This number does not include packages which fail to build in rawhide without the compiler change. It includes packages which configure checks for something that we really do not support (like setproctitle or strlcpy). After the first pass completes, I'll have to do a second pass with the expected-to-be-undeclared functions gathered from the first pass filtered out. That should give us better numbers.
My impression is that a lot of the fallout was related to checks in configure scripts and equivalent. That kind of code was traditionally written a lot more sloppily and was much less likely to have the "standard" set of warnings or the make-warnings-into-errors flag enabled. So, yes, quite a lot of packages failed to build, but the offending code was typically not code that was run when somebody was actually using the program and not code where bugs were consequential. And of course Fedora's a big collection of software with a pretty long tail -- 10% of packages likely works out to a lot less than 10% of user-executing-a-program hours.
I would guess that the main cause of failures is autoconf, which has a habit of relying on extremely dubious C code to check for feature availability. On top of that, the compiler runs are done in a way to guarantee that no one will see any warnings, even errors can tend to be silently ignored (!), and a particular version is often baked into a source package, so upgrading autoconf to a more sane version tends to take eons.
The other day I tried to compile a really old program, the Power Crust and its predecessor. It just worked, for some reason I was expecting it not to. I didn't even have the filetype it was supposed to accept, but it was fine, it was just a list of points. One per line.
> I like how delightfully stable GCC and C is when things from C99 is mandatory only now, a quarter of a century later.
This is how stable it is - they were still supporting C constructs from prior to C89 standardisation (which is when, I believe, void pointers were introduced).
"maybe consider using void * in more places (particularly for old programs that predate the introduction of void * into the C language). "
Variable Length Arrays were mandatory in C99, optional in C11, and are mandatory again in C23.
I'm curious how these dynamics play out in less popular compilers. If a compiler implemented VLAs in C99, they almost certainly still have that feature for backwards compatibility even after they support C11.
Is there any compiler which appeared on the scene between C11 and C23, and during that window, chose not to support VLAs and thus C99? It's not like C11 itself was very widely adopted, precisely because of the long implementation and industry rollout windows.
For these reasons, we propose to make variably-modified types
mandatory in C23. VLAs with automatic storage duration remain an optional language
feature due to their higher implementation overhead and security concerns on some
implementations (i.e. when allocated on the stack and not using stack probing).
The syntax and also the semantics. For instance, you can take the sizeof() a variably-modified type, or the offsetof() one of its fields, and the compiler has to do all the layout calculations implied by the type declaration at runtime. These features are partially what motivated the mandatory support. The only part that is still optional is using such types by value as stack variables (i.e., variables with automatic storage duration).
Snarky or just know a lot? It changes quite a bit how the compiler works. It has to know to make sure malloc gets the array element size argument multiplied in runtime by n. To a mere user it broke my mental shorthand of how a C compiler works.
so you can have "char *y = malloc(sizeof(x)); memcpy(y, x, sizeof(x));" and it must work since C89 at least. The main problem with VLAs is that they make exact the stack frame size unknown until runtime which complicates function prologues/epilogues but that's the problem in the codegen part of the backend, the semantics machinery is mostly in the place already.
P.S. And yes, uecker is a member of the ISO C WG14 and GCC contributor, according to his profile.
I think there is a real difference: in the static case, the compiler can just recurse into the type definition at any point, compute fully-baked sizes and offsets, and cache them for the rest of the compilation. But in the dynamic case, you end up with the novel dataflow of types that depend on runtime values, and more machinery is necessary to track those dependencies.
Of course, this runtime tracking has always been necessary for C99 VLA support, but I can easily see how it would be surprising for someone not deeply familiar with VLAs, especially given how the naive mental model of "T array[n]; is just syntax sugar for T *array = alloca(n * sizeof(T));" is sufficient for almost all of their uses in existing code. (In any case, it's obviously not "creating an object on the heap" that's the unusual part here!)
I'm not tuned in to the nuances here but I note that whoever felt they understood it well enough to update Wikipedia summarized this as "The C23 standard makes VLA types mandatory again. Only creation of VLA objects with automatic storage duration is optional."
VLAs were optional in C11 and compilers that supported them in C99 also supported them in C11. The only important compiler not supporting VLAs is MSVC, but this compiler also did not support other features from C99. People using MSVC were stuck using an obsolete version of C for a long time. Recently this changed and MSVC started to support C11 and C17 - skipping C99.
Maybe because they got bundled into distros soon and as a critical dependency. For a long time as the internet ramped, portability meant you could pretty quickly edit a few header files or Makefile to make it compile for your particular hardware/OS. But then as the perl build scripts showed and autoconf tried to spread, it became expected to download and compile with no changes. In the context of a distro, maybe that put pressure to provide quick work around to keep old code compiling with minimal changes to the actual source?
Implicit int becoming errors is going to break certain IOCCC entries, including some more recent entries that might have done it due to size constraints, for example:
It's really sad how the C standards committee is pushing this anti-c89 thing because it breaks one of the most important use cases for C which is writing programs that work under both GCC and JavaScript. See https://justine.lol/sectorlisp2/#evaluation and https://justine.lol/sectorlisp2/#polyglots I've yet to see any explanation of what we stand to gain, by disallowing implicit int. The whole thing just comes across as religious in nature.
I agree with making most these warnings to be errors by default; actually, they are the same warnings that I had previously thought that should be errors by default (the only exception is incompatible-pointer-types, although that makes sense too to be an error, so it is OK anyways). I think this is good
