What I don't understand is why (at least in places I worked at) CI/build systems are configured to print absolutely everything they do, who reads that?
Hopefully error msg is at the end so you can go to the end of the 100kB worth of logs to find what failed. If it's no at the end though it's kinda funny how you just mindlessly scroll through a garbage and sort of instinctively identify what 'looks like normal garbage' vs 'looks like a bit more suspicious garbage', just by it's shape
If something goes wrong it is very hard to debug unless the relevent information is in the logs. Unfourtuantly, the build system doesn't know what information will end up being relevent, so it just shows everything.
Needing to sift through a giant log is annoying, but you can get pretty fast at it for "normal" errors. Needing to debug an issue that is not adequately logged is a nightmare.
The build system for the product that I work on produces a giant log with all the commands that are run and a separate filtered version that only shows the errors (if any). So for a build breaking change the issue is quickly found in the filtered log. But to answer questions about why the build is taking so long we have the full log.
Because something will fail later on in the process based on an [unintentional?] change early in the process.
Like the App Store helpfully upgraded Xcode over the weekend, and the new version won’t compile the 67th dependency in your project. So you learn to have Xcode dump its version number at the start so it’s in the log. And you tell the junior DevOps engineer about how to prevent automatic upgrades on the Mac mini he never touches. And also how it’d be good to have downloaded a few versions of Xcode and make selecting the one you need a step in the build process… but I digress
What often happens is you get a log like (contrived example):
Warning: http timeout to package server 1
Warning: could not find x in package server 2
..(several lines later)..
Warning: could not find dependency x in cache
....
Compile Error: unknown reference to "x"
Because the actual problem by itself isn't fatal -- it doesn't yet know if the package is in server 2, and it might already have a suitable package in cache -- it's not a fatal error. Later, the part of the compiler they needs the dependency doesn't have context to the other failures that happened; all it knows is something is missing.
This is a universal problem in almost every application: very rarely can a single log line provide enough information to fix a problem or identify a bug.
In my contrived example, the partial fix is that the package bit should fail. However the final log message needs all the context of previous errors as well or it isn't useful either. It's turtles all the way down.
It's not that this isn't a solvable problem, it's that people building compilers and other tools make assumptions about context, people putting together the build don't spend the time to learn the nuance of every tool they use (rightfully so - who had time for that?), and everyone falls back on the crutch of giant log files. In other words, through the whole stack (language, compiler, tools, build system, and individual application build itself), no one has optimized or built anything based on the idea the CI server or build system should have a single, useful error message.
The problem isn't the warnings and non-fatal errors. In fact, the largest problem is that the noise obscures them.
The problem is the pages and pages of "did that, everything is fine". Since Linux distros stopped optimizing disk usage by omiting the update time of files a couple of decades ago, I haven't seen build tools lose any single step. The failure is never because something that should be done was ignored. Instead, the failure is often because of some non-fatal error that you have no hope of finding because it's mixed with 10k like of "build tool got here" spam.
Actually I totally agree with that. I was being generous in my example, and often what I showed as "Warning" is just a informational-level message. This leads to much more insidious problems like:
....(thousand of lines)...
Info: Package X version 1.2.3 successfully added
....(thousand of lines)...
Compile error: Unknown type Z
Actual problem: Z was added in Package X version 1.3.0, but some other dependency or reference that wasn't updated caused the incorrect version to be loaded.
If your code has any size, you won't get all that information from a builder log. Even if it is there, if you knew where to look you would have set the version limits correctly from the beginning.
Fixing this on your own computer is quite straight forward, you just need to check the dependencies, you don't need the info line. Fixing that kind of problem on the build server is one of the reasons we have all those practices about reproducible compilation. On this specific case, it's a matter of freezing your dependencies.
Literally every single sibling comment is missing the point.
Yes, this data should be recorded.
Yes, it should be viewable.
Should it be viewed as a continuous 34000 line log file?
Of course not. But this is what many (most?) CI tools do. The correct way is to group output at least by step, make log output from various parts of the system collapsible etc., for example, dumping all environment variables and component versions at every step CAN be helpful to diagnose issue, but usually isn't. So it should be collapsed by default.
Edit: I have the exact same gripe with all of the log4j inspired logging libraries, which in my opinion provide very little value over a print. To little structure, no way to hold back output until you have an error, temporary redirects are difficult and have to modify global state and so on. Uninspired design providing minimal utility.
> The correct way is to group output at least by step, make log output from various parts of the system collapsible etc., for example, dumping all environment variables and component versions at every step CAN be helpful to diagnose issue, but usually isn't. So it should be collapsed by default.
Which means test tools should conform to some agreed-upon universal output format for their console (and make sure none of the underlying libraries print anything to stdout) or have an out-of-band way to communicate with the CI system in real-time (XUnit/Junit isn’t good enough as those are only typically written at the end of the process).
Most CI tools support some notion of multiple stages and tasks; it would already be helpful if they were able to separate the output of those instead of barfing it all into one log file.
IIRC, Travis has been doing this since at least 2016 where they collapse most of the setup steps by default - and if you are interested, it only takes one click.
This of course only applies to the UI, the raw log is complete.
Out of curiosity what tools _dont_ do this by default? I've been using TeamCity for a decade and it has supported that feature. I used (shudder) electric commander before that and it also supported it.
“Collapsible” implies using some sort of interface for looking at the logs… that interface can also include grep/search functionality that searches all the lines. It could also include functionality to let you download the entire raw log to grep locally.
Diagnostic details so you can see the innocuous process steps that led to a failure/error :) It's entirely possible that a certain value/step early in the process is what ended up resulting in a breakage way later in the process. At least, in my experience that has been the case for all kinds of software/services.
This is the same in case of an official build: if some attacker was able to modify what happens in a build that makes it to production, I need logs to build an understanding and explain to auditor/governments exactly what happened, what the effect was, and how I’ll mitigate it.
Since all logs should be immediately persisted off box, they are protected from tampering.
Of course there’s no reason the default view of logs in the CI/CD system needs to be so verbose.
I don't think it is. These are both logging devices that you look at after the "incident". Sure there are differences, the CI job is likely easier and cheaper to reproduce to get more data, but at the end of the day you want to be able to figure out what happened by looking at the logs.
It's fine to log all the info somewhere, but there's no reason to throw it all into one big chunk and then expect the developer to dig through the data. Especially when it's perfectly possible for the software to produce more structured output in the first place.
Looking for a dependency issue shouldn't be remotely comparable to analyzing an airplane's crash logs in difficulty.
Difference is one will have multiple trained engineers working through the logs to investigate according to a standardised method, the other is one person skimming to work out what dependency is missing (in a repeatable environment).
There must be a better abstraction than here is all the things, have fun.
It's because you're configuring it using a language you barely know, so when you're troubleshooting you don't know if it's logic or syntax.. and unless you're lucky someone else is actually owning the CI system and you have to worry about their permissions.
So you end up debugging everything, and when it works you're afraid to remove the debugging because the last time you did it took hours, and you ended up leaving it in.
Edit: also, because all the important messages are being sent to a group chat or other appropriate place.
I know what you mean, it should store everything but not print it all out. That's kind of on application developers too though, so many programs I've worked on had terrible logging (too much information, the right information not at the right level).
This comment isn't serious. I read that. You just tail the file to find the error. But it's very useful to go through it all. It's not garbage to me. I find most logs quite readable.
ENV variables at every step? print them, build has a parser/codegen step? print everything it does, sanity check scripts? I always need to know what they did, setup scripts? That's the most important part, every .o file compiled and linked? show me, with the compile command for a good measure so all the -Ilibs's are there? of course, set -x? why not, need to wget/curl something? output goes to the log, is it in parallel? just append as it goes, build failed? start the teardown but make sure you add everything it does to the log
And as I wrote, I hope it all just terminates at the first error so I can tail the logs... but it's not always that pretty. I don't believe you've read those instead of tailing for an error or doing the fast scroll to find something looking somewhat off
I mean, I'm used to it now, I'll grep, less, tail and I'll find what I need, I'm just baffled that it has so much unnecessary data for every single build. At one company I've got good enough at it that I could press-and-hold pageup from the bottom in less and staring at the streaming bytes Matrix-style looking for a place of interest
Ideally debugging any issue should include an improvement of the relevant loging. Then, over time, the system can perform relevant self-checks and cross-references, and spit out a list of potential causes, while retaining detailed logs nested deeper to the right or to the bottom.
No, developers don't read compiler error messages, unless absolutely necessary. The economics doesn't work out: the compiler creates hundreds of error messages in a few milliseconds, whereas developers only have a very limited budget of attention.
What developers do is to check where the errors show up, usually just the first error, and use pattern recognition techniques, aka 'experience', to fix the error in its context. Only if the fast track error recovery process fails, repeatedly, do developers spend minutes to carefully read the compiler error messages and deeply make sense of it.
This matches the article thesis: the 'better' compiler messages are, the more likely is that pattern recognition techniques can use the error message as part of the fast track error fixing process.
Some compilers create extremely long verbose but useless messages, other create use-full messages most times only as long as necessary with nice formatting making it easy to skip over the parts you don't care about.
Like e.g. I always read error messages when programming rust, sure I fix a error at a time so I don't read all error messages at once, but I do go through them step by step.
With Rust specifically, but also with many languages in general, it's important to read the errors from top-to-bottom because the errors become more and more absurd as they go down.
For example, if there's an error in a macro that was meant to emit a trait impl, the first error will tell you about the error in the macro, and then the fifty errors after that will be fifty different and increasingly confusing ways of telling you that the type doesn't impl the trait.
If you don't start at the top and fix one error at a time you'll just end up chasing and being confused by red herrings.
This is a real thing that rustc does, but we do try to put effort into silencing errors that are caused by earlier ones. If this happens often to you, I would kindly ask for bug reports because I would love to avoid this from happening.
Clang (and modern GCC) warning are quite helpful, but they are ignored because by default they don't break a build (and only a small fraction of C/C++ projects use -Werror). If ignored they may cause problems later when an app will be used. Or may not cause any problems at all - that the case for subpar code in general - it is more likely, but not guaranteed to break.
-Werror can be hang up development at times, which is why we don't use it with Ardour. Sometimes, it can take a while to understand why some obscure message from the compiler is actually relevant and actionable, and that's not always the main focus while working on features or fixing a specific bug.
However, we do have a policy of expecting that all compiler errors (except those in in-tree 3rd party libs) should be fixed. We take a very negative view of any commits that create new compiler warnings. Essentially, we prefer to have a development culture that treats warnings as errors rather than have the compiler enforce warnings as errors.
For small teams that may work, but at some point, you need to protect yourself from tragedy of the commons. This is especially true during growth periods in the team or periods with significant churn. Culture can very easily shift and having something other than humans involved in ensuring good habits are held is worthwhile.
Something I've never seen recommended - while more information is better up to a point, error messages that are oddly phrased, not uniform with all the others, etc. are kind of a good thing, because they're easier to recognize and search for by a distinctive fragment.
In effect, good writing can have negative returns.
It doesn't happen that often, but sometimes I get frustrated searching for an error that shares its wording with way too many unrelated issues.
In the old days errors came with error codes. This helped searchability using an index in a printed manual, but also Google. Unfortunately people are not good at pattern matching random many digits numbers, so this is not very useful in the front line. I wonder if world triplets, possibly semantically meaningful, could be used as error codes of sorts in the compiler error space.
Getting people to believe error messages is surprisingly hard. "The server says I used the wrong certificate - what am I supposed to do here, what went wrong?". 9/10 times they used the wrong cert, and the rest of the time they forgot to register it to the right app. But they don't trust the errors...
People with exposure to tech support often complain that users never read error messages. It's almost understandable for low-skill users. But it's surprising and disturbing how many professional developers also don't read error messages.
Error messages can be wrong or misleading sometimes, but how do you not at least think about what they said and if that could actually be what's wrong?
I will say that Rust error messages are absolutely wonderful. They generally accurately diagnose the problem and provide a suggestion for fixing it which is usually correct.
Do you still find that holds for somewhat ideomatic rust?
I have made the same experience running tutorials and simple example code, but as soon as I took on anything more intricate, I got thrown right back into "GCC compiling a C++ template" territory.
Basically, the compiler told me "constraint Foo<<<<trait::?>'<mut&>><other> does not meet bounds Bar<trait::dist>?<other>" and that was it.
The compiler also loves to tell you that your borrows aren't long-lived enough and you need to make them `'static`, when making them `'static` is almost always the opposite of what you actually do to resolve the problem - change the borrow to be a move of the original value or a move of an Rc/Arc of the original value.
I've seen multiple colleagues new to Rust ask me for help with their code that has `&'static mut`, and when I tell them that's never going to work and ask them how they got there, it's always because the compiler told them to do it.
Like [1] says, you eventually get the experience to ignore the compiler error message (or at least the "suggestions"), and instead understand the real problem and apply the real fix yourself.
Similarly, when you have a case of value of type A being assigned to a binding of type B, there's a 50% chance that the compiler gets the "expected type" and "actual type" the other way around from what you want. This isn't really a fixable problem, because only you the human know which one you expected and which one you didn't. But it's yet another case where you eventually gain the experience of ignoring the "expected type" "actual type" parts of the compiler error because they don't matter, and just looking at the types.
> The compiler also loves to tell you that your borrows aren't long-lived enough and you need to make them `'static`, when making them `'static` is almost always the opposite of what you actually do to resolve the problem
Most of the time the compiler complains about an unmet 'static lifetime, what it's actually talking about is about wanting an owned type. Sadly, I haven't gotten around to making those diagnostics more accurate (from the user's points of view).
> I've seen multiple colleagues new to Rust ask me for help with their code that has `&'static mut`, and when I tell them that's never going to work and ask them how they got there, it's always because the compiler told them to do it.
> Like [1] says, you eventually get the experience to ignore the compiler error message (or at least the "suggestions"), and instead understand the real problem and apply the real fix yourself.
I would appreciate bug reports at https://github.com/rust-lang/rust/issues/ when encountering these kind of situations. If there's something worse than missing suggestions is inaccurate or misleading suggestions.
Errors involving multiple levels of trait bounds haven't historically been great, but we've made strides on them and there's still some ongoing work to make them better.
Feel free to file a ticket next time you come across a case like these! For complex or uncommon cases we rely on people's reports to handle them better.
I have to admit I was pretty discouraged and rage-deleted the code that made me give up, so I don't have it at hand any more.
I also am really early in the process in learning about Rust, and dove in very deep by trying to use warp to write a web application.
I'm usually the type to dive in by copy-pasting examples and modifying them to my needs, and I just hit a brick wall of compile errors here that discouraged me from continuing.
When I gather the patience to continue, I will keep in mind what you said.
If you are amazed by Rust messages then you'll experience out of body experience reading Delphi's error messages. Better pin point error messages I've never seen in any other IDE.
I think this is asking the wrong question. The first question should be, “Can developers understand compiler error messages?” I’ve had C++ errors where templates were involved and I didn’t even know if it was in my code because the template types in the error message were so long, I didn’t see any recognizable symbols in it. You think you’re declaring a std::foo<bar>, but the error message is talking about red-black trees and deques because those pieces happen to be used internally in either foo or bar, but you didn’t reference either of them directly. It’s maddening.
One nice thing about the rivalry between g++ and clang is that it has caused both sides to improve their error messages ... Template issues can still produce ugliness but it is a lot better than it used to be.
Please, compiler engineers, ALLOW the developer to output stack traces in a structured format. It doesn't matter if it's XML, JSON, YAML, whatever. Please allow the developer to configure the compiler so that when a compiled program yields an error, the error isn't some series of random tabbed lines, but something structured that can then be easily processed.
As someone who at one time wrote a parser for the error output of gcc, clang AND cl, I can only upvote this tenfold. It was a minor nightmare with a ton of special cases, and the final result was a heuristic rather than a formally sound method. Even determining which lines of the output belonged to the same error message was a challenge. Suffice to say that no fun was had.
Great that you would consider this! The format doesn't matter all that much to me... JSON is the thing of the moment, seems the most obvious choice. XML would be fine too.
I’m guessing most new languages are moving in this direction? rustc for example allows for different formatters, here’s the setting for enabling Json output: https://doc.rust-lang.org/rustc/json.html
Not for standalone CLI utils - the LSP is more about powering IDE clients. You could author a CLI that invokes LSP under the hood, but there's no need to tie the output format to that dictated by the LSP.
C++ compilers massively improved their error messages in the last couple decades. If you were missing a semicolon, it used to give you some syntax error when it tried to parse the next statement. Now, it just tells you that you forgot a semicolon. These kinds of user friendly improvements really make compilers easier to work with.
The way suggestions come out lead to a lot of verbosity for C++ compilers when reading in a console.
Template errors have gotten better but can get very verbose. I rarely read the whole thing. You learn how to skim and find where I can click to get to the line which has the issue.
g++ (and I assume other compilers) may diagnose it as missing either a comma (because of the comma operator) or a missing semicolon (can't find the end of statement) - silently providing one or the other would not be a good idea.
Actually, I remember Jerry Pournelle asking exactly the same question about Pascal in Byte magazine in the 1980s.
Never happened to me TBH. There's very few cases where the semicolon is actually significant, and when it happens my autoformatter produces garbage which makes it very obvious, or it adds the semicolon itself.
That doesn't mean it never happens (most of us inherited a shitty codebase lacking things like code standards and autoformatting at some point), but I always find it strange that people keep talking about this thing I've never experienced in years writing JS professionally
A missing brace or parenthesis still often leads to pretty confusing error messages, I find. I don't know much about parsers, though, maybe that's just a really hard thing to deal with.
There are multiple problems that missing delimiters introduce for the parser when trying to recover. It depends heavily on the language's valid grammar how much you can infer about malformed code to identify likely places where the missing delimiter should go.
One thing you can do (that Rust does) is keep a stack of opened delimiters and their positions, and pop them as you find the closing one. If you encounter a mismatch or any other parse error, you look through the open delimiters to match for indentation level or valid alternatives. The more "flag posts" or redundancy the grammar has, the more likely you are to be able to recover from malformed code. If you are in a language with a sparse grammar, recovery is less feasible.
C++ is very hard to parse. For non-parseable C++, it is even harder to recover, that is, figure out where to restart parsing.
The classic example is forgetting to end a typedef struct foo with a semicolon, at the end of an #include file.
The compiler will happily combine that with whatever follows the #include# statement in the file including it, and report an error with a line number in the file doing the #include.
Concepts will help a lot with this. I think a large reason that template errors are/were hard to parse is that SFINAE is fundamentally hard to read for humans (but is relatively simple for the compiler to figure out)
No and it's a tragedy. 95% of the time when I'm asked for help debugging a 'problem' I feel like my title should be professional out-loud error message reader.
Build processes output a lot of useless junk messages all the time. First challenge is to make the relevant line stand out. Our build in a similar situation uses a cowsay ascii art where the cow tells you that your foobar is out of date. Stands out very clearly. Also makes it clear that this is not just some obscure system grumbling but that this is a very intentional message from your colleagues.
The other thing “how do i fix this?” might mean is that the person asking doesn’t know how to update their foobar. Our system state the exact command the developer has to run to get their foobar to the correct version. No guesswork needed. Saves a ton of time.
When helping debug issues folks are having I have to non-ironically ask them if they read the error message. Often they haven't and it says what the issue is and sometimes how to fix it.
Yes, it is embarrassingly common that someone tells me something won't build, and I ask them what the error message was. They didn't know because they didn't read it, and reading the error message made the issue blindingly obvious.
I’d be interesting to study how people read Latex compiler errors, which I find to be unhelpful and an extreme case of bad error messages. I think part of the problem is an extensive use of macros, which expand out into gibberish with little line number information. I personally just bisect my commits rather than read the logs.
Latex is my personal example of bad compiler messages, as well. The other day I added a duplicate bibtex entry and/or added two keys for the same entry. The error messages are a dump of the entire bibliography rather than a simple line number.
In my LaTeX book, I have an appendix dedicated to offering up explanations of what’s really wrong for every error message. It’s kind of fun doing everything wrong n LaTeX to get at all these.
This needs either external pressure on the people writing them or an independent solution. Reality is it's only independent solution.
I'd suggest a hash or a heuristic hash or a 3rd party algorithm run on the error linked to a crowed sourced location the user can chose.
So the 3rd party can say ignore this it's stupid. Something the original error message writer could never tell their boss. Or when the error was written their info was wrong, now it's this. Or they can say use root permissions on the code. Or undo this security patch. And just like Stack, people could explain why and why not. Teach around the error.
Their conclusion "that the difficulty of reading error messages is comparable to reading source code" can't be solved at the error message writers level. Politically or even for the ultra high users who need the hard core message.
Even by allowing the user to rule out the easy solutions it might give, they can turn their brain back to source code mode to tackle it.
A while ago I sat down and fixed all the warnings coming out of a C embedded project. It was a real eye opener and not only did the code get more readable it also helped me clean up a few edge case errors I had missed.
I have been programming professionally for over thirty years and most mentors used to shrug at warnings if the code worked.
It inspired me to go back and clean up my C# and Java stuff and as a learning process I highly recommend it. It seems especially valuable in fast changing environments, generally you will upgrade a tool and get new warnings. I used to think they were just a pain but now I treat them like a way of gaining new insights.
I actually prefer PCC to GCC, because in most cases it just says, "line xxx, syntax error" and lets me figure out the rest. Reading GCC's or CLANG's multi-line error messages is a total waste of time (to me).
There's a rule about not editorializing titles on HN. The title here is the title in the paper, so it's the correct title. This sort of suggestion should be sent to the authors of the paper itself.
Then again, dang has also broken this rule and had a justification for it, so who really knows for sure.
A big part of my mental checklist before committing code for a release is "Are all the warnings taken care of." If the build isn't clean then you are setting yourself up for failure. It may work for now, and maybe 99% of the messages are benign. But that 1% will bite you or your successors in the butt down the line.
Huge compiler errors activate our lizard brains. Overcome that and all of a sudden a big compiler message isn't any different than parsing a JSON blob or a chunk of code.
Hopefully error msg is at the end so you can go to the end of the 100kB worth of logs to find what failed. If it's no at the end though it's kinda funny how you just mindlessly scroll through a garbage and sort of instinctively identify what 'looks like normal garbage' vs 'looks like a bit more suspicious garbage', just by it's shape