I will corroborate one main point of the article: debugging with rr is so much better than a traditional debugger that I will only buy Intel hardware, even with all of the security flaws and corresponding performance hits, even though recent AMD CPUs would halve my half-hour compile time.
It really is that much better. Once you start using it, it's really hard to go back to a situation where you don't have it. It's like driving a car that can only turn left -- sure, if you need to go right you can always 270 degrees to the left (by rerunning over and over again to get closer to what you want to examine, assuming your problem is reproducible enough), but it feels burdensome and ridiculous.
If AMD fixed their performance counters so that rr could work, I would switch immediately.
I am also a fan of Pernosco. It is especially useful when hooked up to a continuous integration system that can hand you a ready-to-go recording of all failures. You can get assigned a bug with a recording to go along with it. Even if it doesn't look important or tractable to look into the ordinary way, having a recording makes it easy enough to take a glance and check it out that you'll actually do it. At shops that do a lot of system-level programming (as opposed to scripting), the productivity boost ought to be worth a serious amount to the bean counters.
What prevents you from having a build server that uses AMD cpus? Budget or physical space limitations? Power consumption concerns? Too complicated to have different build and test boxes? Large binary and debug symbols that would result in no effective gains after network transfer time was accounted for?
In theory, nothing, and that's a good idea that I've considered. My past experience with distributed builds (distcc, icecc) hasn't been that great. They've worked, but they tended to break down or slow down, and I ended up spending more time maintaining my setup than I gained in compile times. Perhaps things have improved. I still have bad memories of trying to diagnose why the network transfers had slowed to a crawl again (on a local gigabit network.)
The other demotivator is noise. I work from home, and the other family members who share my office aren't keen on the sound of a fully spun up build server. I could run ethernet cable under the house to another room, maybe. (Relying on wifi bandwidth hasn't worked out very well. If only debuginfo weren't so enormous...)
It very rarely hit the actual bandwidth limit. It would start out close to it for a while, then drop down. And down. And down. Until it was using like 2% of the full bandwidth, but never completely stalling.
I have use gdb and various debuggers for a long time. They are very useful but there are limitations.
"gdb rr" - tried it with demo helloworld. That works. But when tried it with a more complex program (FBOSS - Faceboss' Switch/Router open source program), gdb rr function core dump immediately.
Also there are certain categories of bugs what are very difficult to use gdb with:
* multi threads / race condition bugs - gdb break point in affect the flow of the program and cause different behaviors.
* Bugs can not be reliably reproduced.
* Server programs or embedded system programs where the code can not be stopped.
* This problem get much worst when combine with performance related issue - I work on one system design requirement that would switch over to backup system when system missing a heart beat for 50 milliseconds. Any gdb breakpoint in that app would automatically trigger a hard system fail over to backup.
BTW, I do use gdb a lot and can script gdb to do conditional breakpoints with script that auto dump variables, info etc.
I'm not sure what you mean by "gdb rr". "rr" is a separate debugger that uses the gdb frontend but a different backend. Importantly, it does not utilize breakpoints during the initial program execution, precisely to address your first, second, and fourth points. The third point is still a problem, since it requires the program being debugged to run under rr from the start.
I would highly recommend looking into dtrace or bpftrace (depending on your platform).
They're not as powerful as a debugger, but you can use them when nothing else will work. They're totally safe to use in production, because they can't modify memory, and they have a very small performance impact when in use.
Never got popular enough to be ported to cpython, let alone python 3.
It's no surprise though: my experience is that most python devs don't know how to use a debugger. In fact, the huge majority of tutorials to learn python don't include one, nor they give explanation about python -m, venv, pip, the python path and other very important basis.
Even the people using pdb don't know very important commands like until, jump, up or tbreak.
Wait what? Do you know about the %debug magic? It is the best thing ever, particularly when prototyping in the notebook which keeps most of the state.
I saw Fernando Pérez use it in a YouTube video. Shame I cannot find it anymore. He was saying he debugs his code only by placing 1/0 in the part of the code he wants to inspect. After the exception is thrown %debug allows you to step into the stack and explore it via a repl. I found that crazy and tried it it. It is life changing. That's how I debug my python code for years now.
It's sad really. I gave a talk a year ago on debugging with the pdb, the basics. Turns out I was the only developer who had ever used the pdb as opposed to bucket listing it.
same with arduino. breaking from pro emedded work to Arduino for some just for fun projs, im flummoxed wo my debuggers and astonished how unfamiliar the community is!
Maybe that's because of the different kind of work I do with but I have wrestled, frustrated at some bugs hard to spot with gdb, but I never felt retrained by pudb despite it being, I guess, more limited.
All the bugs I had in python were easier to reproduce, probably because it provides less rope than C or C++ to hang yourself. Usually, stopping a bit before the exception, exploring the variable states was enough to spot the bug.
Isn't there something about garbage-collected, dynamic-types, GIL-constrained languages that makes their typical bug easier to spot with a more limited debugger?
Depends. Image-based runtime languages such as Smalltalk and various Lisps will, crudely put, pause at the site of the error and give you options to fix it and continue. That already covers 99% of the OP's issues with gdb.
edit: Implied is that image-based runtimes by definition produce reproducible snapshots of the error, with the full power of the Lisp or Smalltalk introspection tools. Edit-continue is the icing.
I think you are missing the implications of working with image-based language runtimes.
The CI or even the client can attaches the live image of the error, the programmer opens up the image when he opens the ticket and has the full venerable introspection tools of CL or Smalltalk. This directly addresses the reproducability issue you raised in said paragraph. There are indeed occasions where you need a proper rr-like trace, but what fraction of bugs fall into that category.
> There are indeed occasions where you need a proper rr-like trace, but what fraction of bugs fall into that category.
Good question. I guess it depends on what you mean by "need".
If you mean "for what fraction of bugs will a developer be unable to figure out the bug, given only the state where the error surfaced, but given unlimited time", I don't know. Most developers I've worked with either have rr recordings available or they do the work to reproduce the bug locally so they can work backwards to the root cause by rerunning the test, so their experiences don't reflect those constraints.
If you mean "for what fraction of bugs is it valuable to have an rr recording", I think the answer is very clearly "almost all". Many rr users, some on this very HN thread, will testify that debugging with rr is almost always better than debugging with gdb alone, whether or not they are able to reproduce the bug on-demand for debugging with gdb.
To restate this: developers almost always want to work backwards in time, by rerunning the program under test if necessary, even if you argue that in some sense they don't have to in some cases. So I think providing a detailed snapshot of the state when the error surfaces, while useful, is definitely going to be seen as inferior to the experience of a full record-and-replay system (all other things being equal).
I agree that debugging in Common Lisp is much more powerful than debugging in GDB. For one thing, when you get a condition and your repl enters the debugger, you immediately learn what condition handler you should add to your code so that it can automatically handle this case gracefully in the future. And you have the full compiler available to you in that repl, so you can redefine functions, change variables, and so on. This lets you recover from the error and leave the program running and in a better state than it was before you started. This is actually a super power that you just don't have when you're debugging a program written in C. (Technically I suppose some IDEs have an edit-and-continue feature, but I've never heard of anyone using it to save a program running in production.)
On the other hand, rr gives you a different set of features. You can rewind the full state of the program to any previous point in time. I've used rr a lot, and this capability is extremely useful. It makes debugging heap corruption, buffer overflows, use-after-free, and many other complicated bugs much, much easier. You just set a watchpoint on the data that was overwritten, then run the program in reverse (using the reverse-continue command) until it gets back to the point where the write happened. It really is like having a super power, but it's a different kind of super power than the Common Lisp repl.
Pernosco is another level above that. Instead of presenting you with the state of the program at a single moment in time and letting you run it backwards and forwards, it lets you query the full state of the program. Where before you would set a watchpoint and run backwards until you got to the previous write, with Pernosco you click on the value and it presents you with a list of _all_ writes to _all_ locations in the chain that leads to that value. For example, you can see that the value was written to the memory location from a register, and that before that it was read from some other memory location into the register, etc. It's really very cool.
There's no reason why a Common Lisp or Smalltalk runtime couldn't have these features in addition to a proper repl.
> You can rewind the full state of the program to any previous point in time
Where does all this saved state go? I'm running a game that runs at 60fps, every frame 10s of megs of data change. It won't take more than a few second to run out of memory to save all the state. Further, if the game doesn't run at 60fps (or 90fps on VR) it's not usable, if saving this state slows the game down to 20fps or 15fps I can't tell if my code is working since the entire goal is to run at speed and check it all feels good.
Is there some technique these kinds of tools use to make it possible to do keep all changed state in this type of app or do they only work for some other type of app?
It's a good question. rr writes all the state to disk. Every syscall of any kind gets saved and replayed back the same way when you replay the recording. This means that if your program reads a network packet in the recording, during the replay it will read exactly the same network packet at exactly the same point in it's execution.
On the other hand, outputs are not really recorded. During the replay your program will generate exactly the same output, so saving it to disk isn't necessary.
There's no such thing as a free lunch, so it'll certainly slow your game down a bit. I recommend saving the trace to the fastest disk you can find. Also, you would want to run your game during development both with and without rr, precisely so that you know that you're not exceeding your time budget.
It's just a pity that it's on the "If you have to ask, you can't afford it" level of expense.
I'd like to try Pernosco, but we don't use it at work. There's no way I could afford it for my open-source projects, and I can't lobby for it without first trying it.
I'm fairly certain that they're working towards letting anyone sign up to use it. Of course I've no idea what the pricing will actually look like once they get there, but I doubt it will be that bad.
The real expense is in integrating with lots of different environments; Roc mentioned that integrations with Github and Travis were already working, but not Gitlab yet. (On the other hand you can do without any integration at all. Take a look at https://github.com/Pernosco/pernosco-submit/; it will upload the source from your computer to your debugging session if it's not cloned from a recognized git/hg host.)
I could pay a bit for personal use, and hopefully it'd still be useful without integrations etc; I don't think I could ask anyone else I'm working with (open-source wise) to do that.
Thats what I like about lisp environments, and sadly after trying out Clojure, it can do no such thing (as of the time I tried it, about a year and a half ago), continuing at point 9f exception after allowing the user to change state interactively.
Clojure is heavily dependent on the host (JVM, JS) for such things, vs. a Scheme like Racket which is ground-up Lisp, or even a naive s-expr interpreter, which allow greater flexibility in this area.
The image contains the state that the point where the error was raised, but AFAIK it doesn't contain the history leading up to that. It's more like a coredump than an rr recording, right?
Doesn't need an image based runtime either. Ruby's "pry" is inspired by Smalltalk, in that it's providing functionality to manipulate the running environment, edit and reload code etc., and continue. During development I almost always have a make target to drop me in a pry session with all the app code loaded and initialized so I can play with the code as it is actually used.
My experience is this is only if it's your own code and you have a full mental model of the state of the program. I worked on a lisp project and the lead would see a crash, type some lisp, continue, app would run. The rest of the team not so much. Maybe we were all just bad lisp programmers.
I will concede that rr is likely extremely useful, but as a counterpoint to this and the entire article, getting bugs earlier and earlier in the development cycle beats better debugging any day. I will definitely try to add rr to my toolbelt if Ryzen support ever lands, but I still prefer catching problems by writing fast test cases instead. It's like debugging that happens automatically.
It's nice to say that catching bugs earlier is better. But what if that time is long gone?
Lately, I've used Pernosco to debug my changes to a multi-process-relevant decade-old part of two-decade-old codebase, which was single-process when written, only later became multi-process-relevant, and recently became even multier-process. (And there are test cases, but there's a huge mass of them and they depend on decade-old comments not actually being true the interesting situations.)
Pernosco is very useful for debugging multi-process interaction in cases that would be very hard to debug even with rr.
OK, I'll just be honest: I had no idea that's how people used rr. I saw demos some years ago and assumed one used it similarly to gdb but with rewind.
I think this is an awesome use case for rewind debugging, that I had never thought of, and look forward to hopefully seeing it run on AMD Ryzen processors in the future, since I've bought into that ecosystem too much to go back already.
"I saw demos some years ago and assumed one used it similarly to gdb but with rewind."
You can do reverse debugging in gdb.
Here's how:
Run: gdb --args /usr/bin/foo --bar baz
Type "start", then "record", then "continue"
Now can try a number of reverse-debugging commands, for example:
* reverse-stepi
* reverse-next
* etc
gdb's built-in recording doesn't scale. You can't record across system calls, thread context switches, etc, and it's about a 1000x slowdown during record and replay. (rr is more like 2x.) It is, unfortunately, an attractive nuisance, because a lot of people have tried it out and concluded that record-and-replay debugging is useless.
Indeed, finding bugs at compilation is better, which is why I prefer languages with strong typing.
I reread my comment to make sure, but I am being painfully explicit that good debug tools are still useful. But I prefer good tests over even good debug tools. I very rarely even printf debug as a result of automated testing.
This isn’t really just lip service; I have a project right now where I literally can’t attach a debugger or run locally. Of course, that’s a flaw and unreasonable. But, with good test hygiene, you can still be pretty productive in the face of this. Not only that, pushing code feels safe, even without the ability to actually run the code locally.
Do I think debugging is bad? No. I think it’s worse than testing, which when done right can save you from a hell of a lot of manual debugging, even in the face of complex software with networked dependencies.
I get ire on HN every single time I mention the virtues of testing instead of debugging. I suspect people literally don’t believe you can exchange quantities of one for the other, but where I’m standing it’s obviously possible. Same thing comes up with strong typing frequently, where you have people making the claim that typings don’t prevent bugs - but anyone who’s transitioned a medium sized code base to TypeScript can attest to the improvements it can bring.
In my opinion, automated tests are almost literally just debugging done by a script instead of a human.
edit: It's also being pointed out to me that rr is being used to debug failing tests, which is not something I thought of and my ignorance may be contributing to some of the reason my comment was so poorly received here rather than the testing vs debugging comparison. (In many cases, for unit tests, I had never thought of using a debugger to try to find the problem, because the actual problem is often made obvious enough by the failed assertion output.)
I agree on the value of tests, but it somewhat depends on what you're working on as to how hard it is to get a certain level of test coverage. I tend to work on a garbage collector, which is notoriously good at collecting the flaws in all memory manipulation code anywhere, and funneling them down to a couple of crash sites where we are traversing memory. We use assertions very heavily to get mileage out of everyone else's test cases, and we add regression tests studiously, but we still end up in the debugger a lot.
Also, one surprisingly useful feature of rr is that just about everything is debuggable without needing to plumb through debugger invocations through your whole system. It will record an entire process tree, so you can go back after the fact and select which process to replay debug. It's incredibly useful. I've been so happy to ditch my weird wrapper scripts to run cc1plus under a debugger while compiling with a buggy static analysis plugin I'm working on. Now I just run `rr record make` and wait for it to crash. It's magical.
"a project right now where I literally can’t attach a debugger or run locally" ... this is exactly where record-and-replay tools like rr can change the game. You can (potentially) run that code under rr on the remote machine --- without interrupting it, which is often problematic --- then upload the recording somewhere (e.g. Pernosco) for debugging.
It matters how you say something. This doesn't leave much room for discussion. That's why this would attract criticism. Even though your points are valid they are not beyond discussion. Personally I prefer interpreted languages, Because of the faster edit-run cyclus. Compilation Time can be excessive. I prefer debugging over extensive testing all the time because testing takes time all the time and debugging only sometimes. Now testing is still necessary for production level applications pre-release because you don't want to make people angry. But otherwise....
I don't like obligatory typing because I find the extra effort of specifying the types to usually not be worth the effort of typing.
OP here. I am obsessed with improving debugging, but I am also obsessed with Rust. Pernosco is mostly written in Rust. I am a huge fan of strong typing and tests, but I know from experience that they don't eliminate the need for debugging in complex systems.
Yes, automated tests are almost debugging by a script.
If you've ever used GDB in command-line mode then it's evident to you that it's the same thing. You can write tests of any level.
With a bit less funky GDB it's completely viable to solve a hard task of writing unit tests for the old untestable code without refactoring for example.
https://github.com/mozilla/rr/issues/1373 is tracking that, but the short story is that ARM seems to not provide enough performance counters to make the current rr approach work at the moment. So it would take either CPU-side changes or a somewhat different approach.
As someone who's also worked on a debugger, I'm sad to say many programmers aren't working on problems that needed complex debugging. I couldn't understand why friends weren't interested in what I was working on, until I realized they were building CRUD apps all day.
There's little control flow to speak of in there, so if they had a problem, they could just dump locals() to get a good view of everything that happened.
When I went back to doing compilers work, I missed my debugger dearly. I also realized why I needed a debugger and my friends didn't: debuggers like this are particularly useful for work with compiler-like characteristics, of which there is unfortunately not enough.
> I couldn't understand why friends weren't interested in what I was working on, until I realized they were building CRUD apps all day.
As someone who gets paid to clean up the cluster fuck that can be created by people building "crud" apps let me tell you better debugging tools would be a godsend.
In the old days with more monolithic apps you at least had one pile everyone shit in, and it was possible to get a handle on how it "functioned" and what its quirks were. Between micro-services and docker it isn't one big shit pile its a bunch of small ones, everywhere. And because of docker on occasion I now get to dig down to the underlying OS, because someone who had no right touching a production server, ever, built and tweaked a container and it "runs" so it went out. Did you know that you can run diff over ssh to compare and contrast the code on two containers? I can think of two rather heathy pay days in the last year where some Jr. on site engineer is the one who found the problem by going bottom up while I went top down.
The article states that "developers suck" and on a bad day I would say that the author is giving them too much credit. On a good day I realize that the issue is much deeper and that we hire technological magpies who want to chase shiny new objects (the javascript world is probably the worst at this). On a really bad day I realize that our entire industry is "wrong headed" starting from hiring. We test people on "can they write code" but no one ever asks the people walking through the door to read 50 pages of it. To find the one misplaced semicolon just to see if they picked up on any of the logic traps that were littered all over the other 49 pages.
So to you sir (or madam), I raise a glass for at least trying to build tools for me, the guy who fixes the "gnarly bugs" in crud apps. I wish you would go back and finish your work, the 6 of us who really appreciate you would happily buy you a beer.
The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in. It's a hard career and industry for this reason. We all know the systems we work on are full of subpar choices, design flaws. We all know the delta between what it is an what it could be. But almost nobody can reconcile that gap on their own. And multiply that by entire teams. It's orders of magnitude less likely an entire team is that skilled and enthusiastic.
I have three philosophical perspectives that help me better frame the software industry.
Number one: People aren't stupid. Rather, things are hard.
Number two: Writing software is like doing a trillion piece jigsaw puzzle.
Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.
"The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in."
It's not just that, but the typical company is too busy chasing new features and fighting fires with understaffed engineering teams on constant death marches to unrealistic deadlines. Nobody has time to do things right, though many of us want to. Instead the systems accrete endless layers of bandaids and half-assed solutions which everybody hates but no one has the luxury to fix or solve properly the first time.
You need strong engineering management to communicate the business advantages of well-made software. The benefits of good software are exceedingly hard to measure and understand as a businessperson, because design is a lot like risk management: you can't measure the cost of the disasters that don't happen. The only substitute is to communicate thoroughly on the business's own terms and to show examples of payoffs whenever they're tangible enough to make the point.
Number 3: MOst SOftware Engineers are bad at their job, they aren't anything close to Surgeons.
This I think is the reason most software nowadays is garbage, there is a lot of low skilled people creating it. I blame bootcamps and dumbed down CS courses for this. It's not necessarily the developers fault, they can't improve on the things they don't know they are bad at.
People who could do better are pushed to ship a feature tomorrow because hey it's simple and it can't take more than a day or so, right? And then we have the next feature..
I haven't seen anyone pull off a large code base without ever going back and reiterating on the design and refactoring things as needed in an attempt to try and maintain the qualities we (developers) would want in software. Unfortunately there are too many places in the commercial world where the time for that work is never given.
Funny, because aside from some cheap tools and coffee, time is pretty much the only thing a developer needs to do anything.
What I've experienced is that in open source projects, patches get rejected and rejected until everyone agrees they're good enough, and sometimes that takes months or years of iteration. Not good enough? Spend more time on it. That's how things improve.
Wanna write a test suite? Need time.
Wanna run a battery of static analysis tools and inspect their output? Need time.
Wanna run a fuzzer? Need time.
Wanna document things? Need time.
Wanna rework the buggy guts of your application? Need time.
Want to figure out the heisenbug that rears its head every once in a while? Need time.
Realize the API the guy before you wrote is crap and needs to be rewritten, and all components that depend on it need to be updated to match the new API? Need time.
Want to formally verify that the core mechanisms of your critical application are correct? Need time.
Need to figure out and rewrite the part of the program that slows everything down to a miserable crawl? Need time.
On the projects I'm on right now, I could think of months of stuff to do to get them to a state where I'm somewhat happy with them.
Every goddamn thing just takes time and not much else. And IME the commercial world presents the same problem over and over again, far too often: you don't have time, you can't bill your customer for all the work that should be done (if good software were a concern), and you can't block the thing they want just because you first want to fix three other things that'd take a while to do. Crap just accumulates, because people build in a hurry, on a bad foundation.
Even open source projects where deadlines and "we sold this as an item that takes three days of work" aren't a thing, time is always a problem because there's only so many people putting hours in.
"Never enough time" is an artificial construct. Set up a pace that gives you a manageable output, nobody will be able to measure you precisely anyways.
Then suddenly you learn to do the right thing ("right" means the one that is not stressing you out) faster. And you're a happy person.
How many managers have you heard giving importance to refactoring or maintenance? (Some pay lip service, but very few actually make sure that its gets done in my experience).
Or customers, for that matter. Sucks when there's weeks' worth of reworking guts, refactoring, and testing to be done and you need to bill someone for the time.
"Even systems with well-defined architectures are prone to structural erosion. The relentless onslaught of changing requirements that any successful system attracts can gradually undermine its structure. Systems that were once tidy become overgrown as PIECEMEAL GROWTH gradually allows elements of the system to sprawl in an uncontrolled fashion." http://www.laputan.org/mud/
There is that old adage "perfect is the enemy of the good", well I would argue that "working is the enemy of the good" as well. When your "average" engineer spends 2 to 3 years at a company how often do they have to pick up the messes they make? Rarely? Never? It doesn't pay to be a good citizen in that case - it pays to be (as another poster pointed out) fast.
> The time, patience and level of enthusiasm required to learn the craft to the level you can develop a piece of software that is well-designed, performant, easy to reason about and an overall joy to work with is about 10 to 100x what the average engineer is willing to put in.
If software is hard, and average people are not ~willing~ to put in the work to make it less so, doesn't that make them lazy? Willfully and willingly lazy at that.
> Number 3: Software Engineers are like Surgeons, except the twelve years of training is received on the job.
Look around you,how many engineers do you know who's code you would be willing to bet your life on? The problem is that every engineer thinks they are a surgeon (and a good one at that) when many of them wouldn't even make good nurses.
It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.
No, it simply makes good systems brutally improbable.
Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.
> It doesn't make them lazy. You're not lazy for not wanting to hike the Oregon Trail.
If I walk the 1/4 of a mile flat loop at the park down the street from my house every day and call my self a hiker I would be a liar.
Really were talking about Dunning–Kruger, "If you're incompetent, you can't know you're incompetent". I go back to "stupid" as the clear winner, and add in that they are too lazy to get out of that rut (put in the work) and that the title "engineer" becomes a willful or delusional lie.
> Re: Surgeons - precisely the point. A perfect result demands a high level of skill which the average developer has nowhere near. As a result there are lots of casualties.
To your earlier point, we give everyone with a pre med degree a scalpel and a patient with a traumatic brain injury and say "go" were going to have a lot of bodies. The case is clear that we are far too tolerant of bad code and bad engineers, and we don't have roles like "podiatry" or "nurse" to demote people down to where they can contribute and we can limit their harm.
In healthcare, podiatry and nursing aren't demotions. They are entirely separate career tracks from medicine. Most nurses and podiatrists do those jobs because they want to, not because they failed at becoming physicians.
I haven't touched a production windows box in 10 years, and probably haven't run windows on the desktop for 15. Your comment did access enough of my windows PTSD to make me twitch. I have abused the ldap interface for active directory several times in those same years periods but I don't think that really counts.
The situation of the "live" (container) works and we can't figure out how to build a new one comes up more often than you think. Knowledge leaves the building. People run off in the weeds and then quit so they don't have to clean up the mess they just made and no one is willing to touch it. Some times an executive shits a kitten and your policy changes from "vendor everything" to "always use the latest and greatest version cause "security" . Sometimes in house staff are stupid, or incompetent, or lazy, or is smart enough to know that if they touch it they own it forever and choose to play one of the aforementioned roles. Sometimes the staff does know and no one in management listens to that guy or gal. Other times it is a minefield that every one dodges on their way to getting what ever else they need done, and no one addressed the elephant in the room till it causes a panic.
I think there are more versions and variations of the "we don't know how any more" tale that I haven't listed, and I am sure that my list is no where near complete however I'm sure you get the picture. The call aways starts the same, I always listen and nod and say not a problem, and use soothing tones to tell whoever is on the other end of the phone that the world isn't going to end. Some of them freak out when I ask for an NDA, and a development environment - many give in but some are unwilling to let me peek under the kimono. Most of the time I can tell them that "price" might get better (or worse) but it will be more accurate if they do so. It is good to get into a system, to talk to someone who can "solve your setup problems" because most of the time they will point you to a closet marked "dead body(es)" and in many cases your half way there. Sometimes it is simple and large (back to that elegant) other times there are a few corpses or an orderly grave yard, other times it is an Alderan level apocalypse and you shut the door and slowly back away.
At that point I can give a rate, and a guess at a time line for a fix (and yes I estimate like scotty from Star Trek). Most of the time the person on the other end of the phone is desperate enough to say yes. With that magic little word the hunt begins - and many systems are shockingly resilient to being ripped apart and abused. Often times things that look important can be ripped out, swapped out or shut off and things still happily churn along up to a point. From a global view you can quickly get to a city or block sized area and then the fun begins. The solution in more than half the cases is fairly simple, I'm changing code because requirements, directions and documentation artifacts have become more sparse as time goes on. Sometimes I'm quashing good ideas for the sake of consistency but most of the time its some sort of "cute" piece of code that is getting ripped out for the betterment of all involved.
Rare is it that I tell someone to really "start over", to burn the whole thing to the ground and not look back. Those closets with Alderan in them do exist and are frighting and fun when you find them. I do love a green field but I am wary of the company that offers me one because most of them make a whole separate set of mistakes on V2.
No one every reads any of the 475 billion lines of crap they are updating, who really knows if you are making things worse or better, or where any of it comes from. Most of us wouldn't eat a ham sandwich handed to us on the street by a stranger but will happily run code from 100's of other people on our servers with out even looking at it.
Often big CRUD apps with a lot of business logic tend to have higher level, "logic flaws" that generate bugs. In those cases having something like `rr` to trace the origin of certain values would be a godsend.
I think these apps tend to be very wide in breadth, where instrumenting the entire system can be hard. Languages like Python (with `pdb` being a single-line in the middle of a program) are great in these scenarios because it allows for a pretty exploratory strategy when debugging.
Unfortunately there's a lot of people who do "print statement debugging" because the tooling really isn't practical in most places, but it would definitely be useful.
In general, I find a sense of learned helplessness among coworkers. They don't feel like they can get much useful out of the debugger.
In a way they are often right, but in part because it's a self-fulfilling prophecy. If you don't use the debugger much, then excessive amount of indirection in many codebases is academic to you, the way (unfortunately) people think of color blindness, accessibility, hell internationalization for some. It's abstract, not something you have to deal with all the time. It doesn't affect you personally so you just ignore it (in other words, complete lack of empathy).
Once the codebase is full of shared mutable state, lasagna code, lavaflow antipatterns, callbacks, handlers, etc, then it really does seem like it's easier to pepper some print statements around and pray.
It's coherent, but wrong. Like a teenager's justification for how they deal with their messy room. Your life really would be easier and healthier if you cleaned some of this shit up, man.
There's another way out, which is that better debuggers can be much more effective at punching through that complexity. E.g., rr/Pernosco-style reverse dataflow tracking lets you jump from an observed value directly to where it was set, which often bypasses a lot of byzantine control flow. When a callback is invoked, we could show you where the callback was set as well as where it was called. Etc.
Having done enough maintenance programming its safe to say that lots of developers manage to add complexity in to what should be fairly simple systems.
Yes a debugger is necessary for debugging a system that has been written by someone else.
I'm not sure it's specific to "debuggers" or "CRUD", I've noticed a lot of developers are just uninterested in advancing their tooling, they don't use IDEs, they don't learn keyboard shortcuts, they don't try to understand build tools. Just just build simple habits and stick to them without stretching themselves to things that might multiply their productivity.
CRUD apps have their own kind of complexity, and would gain a lot by some kind of data-based debuggers that instead of focusing on control flow said where data changes (from what, into what), where it's coming from, and where it's going to.
I wish CRUD apps were as easy to do as a operating system or something.
Seriously. I do this for living. For relaxing, I made a language in the side, because is MORE relaxing.
---
In the case of debuggers:
1- Yes, most debuggers suck not only suck, are near useless. What exactly can I do with "0Xfff23dfblabla" when I try to poke at a value?
This simple thing fail almost everything (.NET, swift, rust, java, ...)
Ok-ish: python, foxpro, js, probably other scripting langs?
2- Skip over thirdy-party and/or standard libraries and focus on MY code. Too much stupid "step in/out" when get in cross calls to code I'm not interested. This also mean better filtering of modules? Libraries?
3- Be INTEGRATED with my logging or tracing calls! print debugging is so useful, but is disconnected from step debugging!
4- Have search capabilities: Let me search in the debugger for vars, functions, etc to step in.
5- Have all the breakpoints on view! I wish I can enable/disable breakpoints with simple clicks without jumping around code.
6- Yes, I wanna look at that array/map with a MILLON values there, Imagine if computers could search stuff like that. Also, how nice if looking at/scrolling(remember, the debugger have not search capabilities) a array of few hundreds not kill my 32 GB machine..
This is also key. I fear A LOT to debug if it have too much variables or if some of it hold large values, like above 20...
Also: Allow to put simple invariants: Stop when this function is called N times, or when this var become Y, etc
This is some of the minimal stuff. I have not use rr or similar but if I can't see simple values, no opaque, can't filter code out, etc is the same pain but with more sophistication, I fear...
Some other stuff, nice but maybe is ask too much:
- Diff of values. Most debugging is to see what have changed (also nice: only stop IF changed)
- The debugging overlay the editor. Put the values above the vars, color/mark the touched lines, tell me how many times X have changed...
- Pull in a SINGLE view all the code on the stack, ie: Instead of jump around dozen files, put the dozen call in a single view and jump into that instead.
- Find a solution if thing are around threads. What if I have 100 threads?
- Have a command window (alike terminal) as useful as FoxPro 2.6 DOS. Something close today is Jupiter or the console in the JS debug tools. But this requiere integration that is available on FoxPro and maybe not cross easy to other langs
--
A step debugger is just a part. Also:
- Structured logging is good... but where is the log viewer for that? How collapse all the noise from logging that is only a repeat of the same stuff? How filter to the section where ERRs/WARns happened? What can I do with a 1 GB of logs to truly find the issue?
- How put tracing that can I enable/disable at runtime?
- How put tracing at runtime, if I don't have put the code for tracing before? Could I say "Trace before/after the code enter this function call?
Heh. Although it's currently really only for system languages (so doesn't do much for your #1), you're basically describing the feature set of Pernosco.
Another testimonial from me. I basically refuse to do any debugging on systems that rr doesn't run on unless there's a very good reason. I use rr extensively whenever there's a bug in Julia and I would singlehandedly credit it with a measurable increase in system stability due to it catching bugs we'd never have gotten otherwise. roc is one of the few people with both the vision and persistence to really improve the debugging experience.
Yes, the latter (mostly for various weird corner cases). Back in the Gallium.jl days we could actually debug the recorded julia code, but we haven't regained that capability yet.
Maybe I missed something, but what really sucks is making "how to use a debugger" demos, because most developers don't even know what their debugger can do for them, let alone use it properly.
Visual Studio team started doing VS productivity talks, because they noticed that many feature requests were about features that VS already supports for ages.
It can be worse. I work on Python debugger in Visual Studio and VS Code, and it's still unbelievable just how often, when I describe what I do to people who write production code in Python, I have to explain what e.g. conditional breakpoints even are, and then they say things like, "Wait, there is a tool that can do that? That could have saved me a few hours the other day!". And this all has been around for many years now...
There's some progress - these days it's mostly confined to the data science crowd, and web devs have mostly caught up, due to the rapidly increasing popularity of PyCharm and VS Code over the past few years.
In general, it feels like there's some weird gulf when it comes to tooling. On one side, you have the hardcore C gurus (often doing embedded), who do heavy scripting in gdb to solve very complex problems. On the other side, you have IDEs that grew out of 1990s RAD - Eclipse, Visual Studio etc - in ecosystems where the language and the tooling are usually developed and packaged together (e.g. whenever a C# compiler ships a new language feature in stable, there's also a new stable version of VS that supports it in the editor, the debugger etc - and UX for those is explicitly considered when designing features). And in the middle, you have all the high-level "scripting" languages, which have both low-level debuggers and powerful high-level IDEs - but their respective dev cultures largely ignore both for historical reasons.
There is more to it. In the past, I was a all-day VS User (C++ most of the time, but even Python. Today I prefer Wing IDE for this (and sometimes VS Code)).
While conditional breakpoints can be very useful, they often are not (especially because they slow down the code - hitting a variable value after 20000 iterations in a heavy-loaded loop as an example). It follows, for most developers syntax doesnt sink in. At least in the past, documentation was too far away. These things have to be literally at your finger tips. Don't know about the situation nowadays, though.
As much as I love rants like these, the problem with debuggers is discoverability.
I have lost track of the number of times that I kill myself debugging something with gdb only to later explain it to some group and somebody says: "LOL, n00b. Why didn't you just use <magic gdb command>?"
Why? Because I didn't know about the magic gdb command and there was no documentation anywhere that would have pointed me to the fact as to that command being exactly what I was busting my head over.
Debuggers are like build systems. You don't diddle with them every day and when you are something is wrong and you're on a deadline.
If people want programmers to appreciate better debugging, we need some good videos of people debugging real problems in real time with debuggers.
Speaking of GDB and arcane commands - GDB does have decent documentation. It actually ships with a whole book on how to use it. Try `info gdb` in your terminal (if you get a man page, you're missing info pages; on Ubuntu/Debian, you have to install gdb-doc package).
One other way developers suck (myself too often included) is that they don't RTFM. And I don't mean "skim the manpage" RTFM, but actually read the included manual. If you're going to use something day in, day out, it's really worth it to spend the couple of hours required and read the included manual cover-to-cover. It saves a lot of time and frustration in the long run.
(I don't use GDB myself all that much, but the last time I had to, I got tired of flailing around; I read the info page cover to cover, and the time investment repaid itself on that single task.)
OP here. I get it. I led rr development and even I sometimes find gdb features I never knew about!
We explicitly tried hard to make Pernosco discoverable, partly by creating simple but powerful and composable features, and partly using traditional GUI techniques. We definitely haven't fully succeeded, but I think we moved the needle. If you dig through https://pernos.co/about/ you can see what we tried, and if you try the demo https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html you can see how well we succeeded.
> static.pernos.co has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.
> The issue is most likely with the website, and there is nothing you can do to resolve it.
It would be amazing if someone could design a programming language around debugging. There are many languages that are designed around preemptively catching bugs using a type system, and I do use static type systems, but this in no way oblivates the need for a debugger.
The way I want to write code is directly as it is executing. I think the distinction between writing code in an editor and debugging is arbitrary - I want to edit programs as they are running and see the data as it flows through beside the code. I want to be able to browse the object graph as if it were a filesystem like I could in Objective-C (I think Smalltalk invented this?).
This does require a different code architecture to be effective: more pure functions and less tangled state, but those are usually considered to be good things. Improving debugging might have a positive effect on code quality.
Clojure is both a good and bad example of this. Live hot-reloading means that you really are editing programs as they are running. This is especially visceral if you're working on something with a GUI component as you can see the GUI change before your eyes as you work on it. And if you're really adventurous you can open up a REPL connection to a running program in production and edit your production service as it's running (don't worry you can disable this so that random folks can't start modifying your production code)!
On the other hand, running a debugger in Clojure is a bit of a mess. Generally I just don't and end up creating "watchdog" atoms that hold bits of state that I think are relevant to the situation at hand and evaluate those with a live REPL connection. See e.g. https://eli.thegreenplace.net/2017/notes-on-debugging-clojur.... Although the article ends on an optimistic tone, I've found using a debugger with Java or Scala more rewarding than using one with Clojure.
I would like if existing languages allowed you to "level up" your code.
examples that have worked for me: "use strict" in perl, "tainting" in perl, assertions in C (runtime and compile time), -Wall -Werror in shared codebases, and similar.
the idea would be to incrementally make everybody's code a little better in manageable increments.
> The way I want to write code is directly as it is executing. I think the distinction between writing code in an editor and debugging is arbitrary - I want to edit programs as they are running and see the data as it flows through beside the code.
I do something like that every day, especially when I'm dealing with weird and arbitrary APIs (which is most of them). I write as much of a function as I know how to do, set a breakpoint on the next line, and run what I have until it hits that breakpoint.
Then I can write the next few lines of code while looking at actual data.
If it happens to be Python code and I'm using IntelliJ, I can also open a Python console - a REPL running in the context of my breakpoint.
This helps me understand the libraries and APIs I'm working with in a way that just editing static non-running code doesn't.
Counter-intuitively, I wonder if this has contributed to jupyter notebooks' popularity. You write your program in pieces (cells) that you can run independently; modifying, going back, rewriting, and rerunning bits on the fly. Which is hilarious to me, since the usual kind of debugging is so painful in jupyter. You could imagine an alternate history where we had amazing debugging and jupyter never got quite as popular. Hell, maybe even REPLs wouldn't be as popular.
Elm has some of this built in. Not a debugger per se (you cannot add breakpoints and step through the code), but since everything is pure and there is only a single state, you have a single source of truth and a function from that truth to your view. One can replay the different transitions, and see the whole state of the program at those times.
elm/browser was recently updated. If you compile in dev modus it gives you a small button to press in the webapp, that opens a new window where actions and state can be replayed.
Visual Studio lets you do some of this in a clunky way with ‘Edit and continue’, where you can edit code after a breakpoint in the same method. I believe most browser JS debuggers let you edit the JavaScript, though it can be quite buggy.
What is clunky about it? It even allows you to drag back the current program pointer to a previous line in your program, change the code and run the section.
One problem I found was it doesn't work with generic methods though, not sure if MS is working on making that work. To me "Edit and continue" is a great tool.
Somewhat opposite direction is languages features that increase the need for more debugging tools.
As c++ is getting more and more features for compile time code evaluation it raises need for debugging them. One language that gets it right is Haxe. It allows executing arbitrary code at compile time for AST manipulation. Official implementation comes with a debugger that can debug it just like regular code.
Does anyone knows something similar in other languages or build systems? Does any of LISP environments provide such debugging functionality?
The compile-time code evaluation in Lisp usually is just normal Lisp code evaluation, since the compile-time environment is also just Lisp. Thus the usual debugging tools apply.
Julia does this. You can run the code and you get the output right next to it in an expandible dialog. You can write your while program this way, executing a line of code, tweaking it until it is correct. It can be quite nice. My one complaint is that VSCode doesn't have support for this, so it runs in Atom, and I find Atom to be quite slow and buggy.
> In particular, developer culture is that developers don't pay for tools, especially not debuggers
I think there are a couple of reasons for this. One is that for many if not most developer tools, there are free options that are at least competitive with paid tools. Another is that developers can, at least in theory, read source code to better understand a tool they use, or even improve it, and therefore want access to the source code. Which means open source tools have an advantage over proprietary tools in addition to price.
This line galled me too as it showed a serious misunderstanding of the principles of libre software. There's a reason we say Free as in Free Speech, not Free Beer! Why would I ever want to make my workflow dependant on a third party blob whose internals are obscure from me?
> There's a reason we say Free as in Free Speech, not Free Beer!
Agree.
A major problem that needs to be solved is creating a user-funded, donation-based model of free software development. The old argument was that copies of free software can be given a price and sold on tapes, thus the distinctions between price and speech - you could potentially support the development of free software by selling copies. But this practice has largely disappeared in the Internet age, and it seems the only business model of free software today is value-added services.
One could argue that free software, by its definition, allows unrestricted, free redistribution (both speech and beer), others can still distribute it for free, so it's meaningless to sell copies. But I would say that many users may still want to pay to support its development if there's a system in place to encourage it. Currently, the visibility of donation in free software remains low.
For example, there's nothing to prevent us from asking for a donation to download on the official website or an "app store", or integrating a payment system on a code hosting platform, or a micropayment system that encourages you to pay $1 for reading some documentation. You can even integrating a micropayment system to a package manager that pays while you are installing something, or even refuse to provide documentation from the official website or support from a package manager unless you've paid, forcing people to either pay or find a 3rd-party's copy, it's ethically questionable but remains valid under the principle of free software), the possibilities are endless.
The only obstacle I see is the lack of a good micropayment system, but it can be solved in principle.
One thing about debugging is that there isn't much lock-in. If you don't like the debugger or it goes away, switching to another debugging approach is very easy. Your workflow doesn't have a hard dependency on it (other than to the extent you enjoy using it).
It's a bit like Github, which also isn't free software, but so many free software projects use it, in part because people think they could easily switch away from it if they wanted to. (I think in practice switching away from Github would far harder than switching debuggers.)
The price does also play a role. Learning tools and frameworks takes a lot of time, and if it's not a free tool then if you change jobs or even departments you might find yourself unable to get a license. If you want to work on stuff at home, you need to pay. If you want to run a test server you need a license. Some tools are better than others in that regard, but it's basically why I would always choose learning a slightly worse free option than a paid option.
* Most developers start to learn at a time where they don't have the money for tools.
* If the free solution is good enough, there is little motivation to invest in higher grade tools.
* If there are "good enough" free tools, the market is very small for a paid solution so the price has to be high.
Developers tending to read code to understand a tool or even improve it, is a very rare case in my experience, which is limited to mostly computer science students and some researchers though.
There are many reasons why I don't do it myself, e.g., code of many projects is hard to read and takes a lot of time, implementation only losely helps me understand the tool besides edge cases where it doesn't do what I want, I am interested in solving a problem with the tool instead of understanding the tool.
>* Most developers start to learn at a time where they don't have the money for tools.
There is even more to this point. Most programmers still like to work on some personal projects here and there, and they don't like to have a completely different workflow at work and at home. Plus they usually have multiple devices (desktop PC, laptop) or even multiple VMs and in the past licensing wasn't really compatible with these kinds of scenarios, and you'd often have to purchase the software 5 times.
Not to mention it is annoying when you sit in front of a new device and you can't just install your tools and instead have to think about how you're going to transfer or apply your license.
> in the past licensing wasn't really compatible with these kinds of scenarios, and you'd often have to purchase the software 5 times.
That is only because of greed. During the 80s and 90s, Borland for example had a "no-nonsense license agreement" that told you to treat the software as a book: make as many copies as you want, have as many users as you want and even resell it as long as - like a book - only one person at a time is using it.
They did change it at some point in the 2000s, probably around the time when they stopped selling Delphi for $99 as the cheapest option, introduced online DRM and prices skyrocketed to the thousands.
If there were reasonably priced tools like this then I'd definitely pay for them. But this tool is still "contact us for pricing". I contacted UndoDB for pricing a while ago and the minimum was $50k. :/
>my messages about Pernosco, which I think is much more interesting and important, have relatively little exposure
It is not very surprising given that it is a SaaS which doesn't even list pricing on its page.
>I want everyone to know about Pernosco, not just to attract the customers we need for sustainable debugging investment, but so that developers everywhere wake up to the awful state of debugging and rise up to demand an end to it.
This is rather conflicting statement as Pernosco business model seems to be solely targeted at bigcorps paying big money, and not developers promoting Pernosco through using it in own hobby projects. (I don't know if this is actually true, but email-us-to-just-get-a-quote is huge discouragement)
It is also very difficult to reach a usage example or a workflow on their site. There is no demo. I agree that debuggers suck, but I don't see any clear/obvious evidence that Pernosco doesn't suck. I really just have to take their word for it.
I don't want to pay an undisclosed amount of money for the opportunity to spend an unknown amount of time to get started with a new tool that may very well suck.
It's true that we aren't opening the gates for people everywhere to use Pernosco for their hobby projects, because we simply don't have the resources to support that. Hopefully that will change.
I used to sell debuggers a decade ago. We even had reversible debugging. The amount of personal development productivity I had was unmatched; I’ve never gotten back to that point.
It is true, nobody wants to pay for engineering tools. I wish the author luck.
The market is so small, and there appears to be lots of premature optimization of the checkbook.
I remember many years ago trying out purify, and it revolutionized my debugging. The problem seemed to be that the program I wrote could be debugged by everyone - if you just got everyone a per-user license.
It would have been hard enough to justify a license for me (a person at the bottom with no power), it was impossible to justify a license for everybody.
> It would have been hard enough to justify a license for me (a person at the bottom with no power), it was impossible to justify a license for everybody.
"Floating" licenses are better here. In my experience, companies are reluctant to buy licenses for an entire team for something that's used only occasionally. It's easier to make them buy a few floating licenses to be shared by the team.
I am uncertain if floating licenses were available at the time. Even so, when floating licenses became popular, there was also the "license in use" denial problem.
Unfortunately there's another way to do this nowadays. Offer convenient tools with analytics, etc... (and coincidentally spy on everyone)
It's double edged: the co's that can afford to proactively seek better dev tools can also afford internal vanity projects to do them internally. So it becomes a political timing game for the project leads to move on etc. Not fun.
I found an easier path for dev tooling is focus on tools for non-traditional/non-fulltime coders, e.g., analysts, devops, secops, ... , for whom being able to code introduces a superpower, not an increment on status quo within a crowded space. It's a shame devs destroy our own potential.
That is a concern, but Pernosco is challenging to implement. We built on rr, which was already state of the art, and we've added a lot of secret sauce. Of course the smart big tech companies could duplicate it, but it is not simply a matter of putting a team together and grinding out the code.
About a dozen years ago my workplace built something like that in-house. I see you've made progress. You have the reverse dataflow. You can't debug an arbitrary firmware or bootloader or OS however, and you can't properly debug a bunch of interacting systems.
Simics also seems to have most of that. I last checked about half a dozen years ago, so the other features might now also be supported. It's not cheap, but you could model all the computers of a factory or aircraft all at once. You could then step forwards and backwards as desired, coherently examining the RAM and other hardware of all the computers as you please.
I wouldn't be surprised if dozens of similar tools have been built.
Out of those things, the one we do want is the ability to debug a collection of interacting distributed processes. Our goal would be a bit different though: splice together recordings from different nodes of a real system, rather than simulating all the nodes from the ground up. So our implementation approach will have to be different.
Fair, but that's not rr's target market. AFAIK Simics is not competitive with rr or Pernosco in recording overhead or scalability when you're debugging a Linux user-space application that executes, say, a trillion instructions.
Computing today looks different compared to 2009, AWS came out in 2006, the original iPhone came out in 2007. Why, then, would the market for debuggers and other developer tools still look the same as it did 10 years ago?
Coding bootcamps have sprung up and regardless of what you think of them, there are more developers today than before. The more developers out there, the larger the market to sell developer tools to. Auto-mechanics and other tradesmen have a couple thousand dollars worth of tools that they bought themselves. Why then are software developers so uniquely hard to sell to?
JetBrains, makers and sellers of Python-specific IDE PyCharm, seems to be doing okay.
With companies now tracking "developer velocity" as a metric (somehow), a tool that improves velocity seems like it would be an easy sell. If I can't rewind code execution in the debugger, I might as well just be printing variables to the screen. (Which is a time honored debugging method that will never die. But maybe it should.)
There's a B2C vs B2B question in there, but maybe the market is changing in the software developer tools market.
The Visual Studio .net (c# in particular) debugging experience has always been excellent in my opinion. One great plus for a managed language. I guess that the closer you get to the metal the more difficult it is for a debugger to do its magical things and hence the poor experience for c++ devs.
Apart from Intellitrace, AFAICT it's a great implementation of the traditional stop-and-inspect debugger feature set, but that's all. Intellitrace adds something, but not full access to all program states like rr (or Pernosco). So I think your expectations could be higher :-).
> it's a great implementation of the traditional stop-and-inspect debugger feature set, but that's all
It provides more than that - the "Edit and Continue" feature is invaluable, allowing you to pause execution at a breakpoint and change the code, such that the new code runs when execution resumes.
As an aside, Rider also has this feature, and I find it to be much faster and more reliable than VS.
Visual Studio also recently got "Time Travel Debugging"[0], which lets you record execution then replay in the debugger. I haven't actually tried this yet.
Ah yes, TTD is a real game-changer. It compares fairly well with rr, though the recording overhead is often a lot higher. Lacks a lot of the Pernosco feature set though.
I think that not many people have used TTD yet so when people talk about how great the VS debugger is, they're not really talking about TTD.
I think Visual Studio's TTD is still only available with Azure web apps or VMs, which certainly limits the audience. Hopefully it will be made more widely available soon.
I used VS for a long time with C++ and C#. It really isn't fashionable though. I moved to more popular language stacks and continue to be amazed at all the problems, crappy debuggers included. I really miss edit and continue, hovering in breakpoints. I'd love to go back to C# but its just not what FANG-style companies do.
So I dug down into the bowels to see why it's so lame......
Oh my.
The hideous complexity imposed by (multiple) CPU designs, OS interfaces, the vast mismatch between the code as written and the code as compiled. The complexity of the DWARF standard....
Sigh! Sorry gdb dev's... I'm sorry I ever thought bad of you guys...
I dearly wish that I could use rr and Pernosco, but I do graphics work and GPUs don't work with fancy debuggers. The state of GPU debugging is even more dire than CPU debugging.
But if the debugger doesn’t come with source code that I can inspect and modify, that’s a big negative that will have to compete with whatever advantages the debugger has. (It’s not just a hypothetical use case. I’ve grepped through the source code of GDB and LLDB on several occasions to figure out why they weren’t working properly.)
And if the debugger is SaaS, meaning it will almost certainly disappear in a few years, after I’ve gotten used to it… it would have to be some super magic unicorn stuff for me to consider touching it, even if it were free.
Try out rr first. It has at least half the magical unicorn sparkles you're looking for, is open source, and runs locally. It's not hard to modify. (I've done it, though not for any core functionality changes.)
Then you'll be in a far better position to evaluate the additional benefits Pernosco might provide you. I think your reasons for being leery of it are valid, but there are very legitimate scenarios where it makes sense IMO. (Recording CI jobs automatically, collaborative debugging, situations where the critical dataflow is hard to follow...) And they integrate, too - you can upload rr recordings to Pernosco when you need the extra functionality.
I guess it also depends on the pricing model, which I know nothing about.
Tragically, one of my regrets in life is never really learning proper debugging. It is embarrassing that even as a senior engineer in the industry, the vast majority of my debugging is basically a bunch of console logs, and it gives me a serious case of imposter syndrome when I see others using all sorts of fancy debugging tools.
If there is a silver lining, it must be that my lack of debugger skills can only mean I spend a lot of time writing safe, bug free code. Or more likely, no code at all.
Fixing bugs without a debugger is a great skill. If you're good at it, you'll be able to figure out many problems far faster than people who depend on debuggers, and you'll be more likely to get to root causes and change things to be more robust at an architectural level.
Fixing bugs with a debugger is a great skill. If you're good at it, you'll be able to figure out many problems far faster than people who don't use debuggers. You can skip a lot of deep thinking and go straight to the site of a problem. You can explore the actual execution paths in ways you'd never figure out from all the layers of templates and overloads. You can debug other people's code almost as easily as your own. You won't get caught by some subtly wrong assumption you're making when thinking through how things are supposed to work.
Debuggers are dangerous. They're insanely useful, but once you start depending on them, your ability to fix things without a debugger will atrophy. If I had the self-discipline, I'd ban myself from using debuggers for a month out of every year.
There's no mention here of inspecting core files, something that very much allows you to inspect the state of a program when the bug occurred, but after the fact. Is this because inspecting core files also sucks, since it's a snapshot of a point in time, and it's no use if your stack is corrupted? Or is it because this is a lost art?
In my embedded work I avoided using debuggers because setting up remote debugging can be a PITA. I figured out a long time ago that building remote debug setup into my workflow will save oodles of time in the end. Still, I find myself using plain old logging more often than not - the bug may not be in C or C++, it may be on the JavaScript run in the interpreter embedded in that C++ application.
Core dumps are great, especially for the occasional odd bug that occurs only in long running processes. However, the ridiculous systemd default of attempting and failing to capture the kernel core dump into the binary journal completely ruins this. Only small, uninteresting core dumps are ever captured; large ones are silently discarded because the journal is not allowed to grow that big.
Yeah, when I'm crashing and want to keep cores, I'll temporarily modify /proc/sys/kernel/core_pattern to prevent systemd from eating them. I sometimes want them suppressed, sometimes want them written out, and never want them to go to systemd.
If a core file is all you've got, then you do your best (and there are some nifty researchy tools that help with that). But if you could have a full recording instead of the core file, then of course you'd want the recording, because that dramatically increases the probability you'll be able to debug the bug (to near 100%, if the bug is in the code you've recorded).
"The only valid use of a debugger is to analyze deadlocks in post Mortem core dumps. Interactive debugging is the greatest time waste. Write automated tests for your debug hypotheses instead." (Peter Sommerlad)
I don't care much about fancy features. I just want a simple debugger GUI in my IDE that works consistently, and never crashes. Until I have that, I feel like pursuing even more functionality is pointless.
(Context: c++ developer on Linux. The only debugging experience I've ever had that approaches what I describe was in c# on visual studio)
A lot of debugging UX is editor integration. For top end IDEs, these are available out of the box. Set breakpoint by clicking left margin of any kind of code, another click to run trigger debugging. Similar story for VS Code though a bit more configuration is required. It's only an issue when it comes to lightweights like Sublime Text/Emacs/Vim. The philosophy of things being "loosely coupled" means that you have to be prepared to spend an hour browsing man pages just to figure out how to create a breakpoint. Edit and continue isn't just a fancy gimmick used in Lisp; it's a first class feature in Visual Studio (iirc it originated in the original Visual Basic, got removed during the transition to .NET before being added back again to Visual Studio, someone from MSFT should confirm this) Python is just "rediscovering" how to do this. JavaScript (in Chrome at least) have partial support, mostly because they were forced to do livereloading so frontend development isn't too unpleasant. Rust and Go and all the new age languages never seemed to have heard of it despite proclaiming their commitment to good developer debugging process (Elm might have some support though their blog releases seemed more interested in time traveling functional data structures rather than benefits to developer experience, and no, you do not need fancy stateless functions to add Edit and Continue though I suppose they do make it easier).
How about being able to debug multiple threads and processes? That would be actually useful.
> Rust and Go and all the new age languages never seemed to have heard of it despite proclaiming their commitment
Go rebuild / full rebuild times are fast enough. I don't think edit and continue would be that amazing with 1000 concurrent goroutines.
Rust will hopefully have edit and continue one day. Hopefully compiler will get faster as well... That said, I much prefer Rust compiler catching a significant portion of issues at compilation phase, thus not requiring debugging in the first place.
I don't think I've ever seen an IDE for a multithreaded language that couldn't debug multiple threads. Most can also debug multiple processes in a single session, and some can automatically attach to child processes.
I both use Windows and deal with software using shared memory. Neither of which rr can deal with.
While rr seems to be very nice, there's still room for improvement. But I guess this kind of tool might be impossible on current hardware, at least without emulation.
Edit and Continue appeared in VS 4, 5 or 6 (can’t remember which one exactly), right before the first VS for .NET which I think was released in 2002. It supported C/C++ out of the box and required quite complex linker plumbing. Second .NET VS was 7.1, releases in 2003, before MS switched to confusing year-versioning in user-facing contexts and number versioning internally (including installation directories, project files, etc), which is made way worse because these days internal versions and years are off by two or three units (ie, Vs2018 is internally version 16 or 17, can never remember). My commercial vi/vim emulator, ViEmu, supported VS.NET 7.1/2003 on its initial release in 2005, and I still maintain that build, as in I build for that target every new ViEmu release and occasionally use it (it’s a much leaner and more stable debugging environment than more recent VS versions).
Note that with the edit and continue appearing in VB he meant the immediate window in VB1, not the C++ feature. Also this didn't really appear with VB1, it most likely "appeared" with the first BASIC Bill Gates wrote (as far as Microsoft is concerned, BASIC had it before MS) as any number-based BASIC was able to both run and edit code at the same time and this was evolved to QuickBASIC/QBASIC's "immediate" pane at the bottom where you could run functions as soon as defined them in the editor and modify the program's code and state by pausing it and then this was passed on to VB1 though a dedicated window that remained there until VB6.
One bit I’d add is the bad attitude towards debugging in optimizing compilers. It’s not obvious to me that optimizing for perf at the cost of debugging fidelity is a good trade off but here we are. Like, if you could save enough time debugging with better tools then you’d have more time to optimize. And maybe if compiler devs confronted full-speed debugging head on then we’d get to perf that is good enough for everyone and we’d all
be more productive.
Or maybe it all sucks
and all we can do is bitch about it.
For clang users things would get a lot better if someone just fixed a lot of clang bugs.
Omniscient debugging creates new opportunities to overcome the impact of optimizations. E.g. are you trying to display the value of a variable after its live range ended? No problem, look back in time to when it was live. However, in many cases we need extensions to debuginfo to take advantage of that; e.g. DWARF can't express the difference between "variable outside live range, but has not been modified" and "we don't know where it is".
If we had a debug build and an opt build of the same program then there are some mad-science approaches that would be fun to try.
So I actually think the perf/debuggability tradeoff is not fundamental and with the right resources we could mostly make it go away.
JSC tries to go into the tricks you describe for the purpose of optimizing OSR exit. It’s really hard. There is a register pressure overhead from situations where the program thought a thing was live but really at that program point some of the necessary inputs to the value you’re trying to recover (and the value itself) would have been trashed by regalloc if it wasn’t for the need to be able to rematerialize that value. That translates into register pressure and you can feel it even on forgiving platforms like arm64.
I believe that this overhead in conditions where the codegen is otherwise behaving well is around 5%. So not terrible, but that’s my call of the price relative to -O3 or whatever.
Perhaps I wasn't clear. None of things I mentioned would entail changing the generated code in any way.
In the comment about omniscience, here's an example of what I'm proposing:
-- Debuginfo says variable 'x' is in register R1 in the PC range 0-10
-- At 10, R1 is set to something unrelated
-- Debuginfo says in the PC range 10-20, 'x' is equal to its last live value (i.e. its value at PC=10)
Pernosco would evaluate the value of 'x' at PC=15 by actually reconstructing the value in R1 at PC=10, using the omniscient database we already have.
Of course DWARF can't currently express this, because normal debuggers can't make use of it. Even in rr it would be quite expensive to compute.
If you're familiar with DW_OP_entry_value, it's a bit like that. That's a way for DWARF to say "the value of this variable (e.g. a parameter) is equal to the value of this subexpression (usually a register) on entry to the function", which regular debuggers can often obtain using stack unwinding. Pernosco doesn't do the unwinding, instead it (logically) warps back in time to the point where the function was called and evaluates the subexpression in that context.
You can always tune it one way or another with the optimization level (-O0, -Og, -O3, ...) But I agree. That doesn't really help when you need to debug release binaries, which are understandably at maximum optimization levels.
gcc is actually pretty good at preserving debuggability through optimizations. clang? Not so much.
> As far as I know, the only way to fix this situation is by building tools so much better than the free tools that the absurdity of refusing to pay for them is overwhelming, and expectations shift.
This of course will only work for a little while until someone who has more time than money (or is a big company that wants to commoditize the tool) will build a command-line version of it on Linux. Weird UX and the need for a spaghetti of shell scripts to integrate with vim/emacs/vscode/sublime/ed will soon follow with an Eclipse addon that nobody will use a bit later. After a macOS port, assuming these are still a thing, Apple may create a nice looking UI and integrate it with Xcode or someone like Panic may create a good front end and sell it for ~$99.
Most people will keep using Windows, think Visual Studio has the best debugger and everyone will be happy.
Those that learn about the "overwhelmingly better approach" will consider the free one the best one and its spaghetti of shell scripts approach the obvious best approach, because they wont have any real deep experience with the paid tools (either the original one or the macOS shiny wrapper - which will be considered as unnecessary by most anyway) to properly judge.
Yeah, nah. I sympathize with your point of view, but implementing Pernosco requires some serious science. If you try building a database of all memory and register changes in a naive way, it isn't going to work at all for nontrivial programs. Of course it could be cloned, but you would need a very good team and a lot of work.
Also either you start with rr as a base, in which case you need people with deep rr knowledge, and I know their names, or you build or buy your own record and replay framework --- more time and money.
It's no longer likely at all, but some years ago my workplace nearly open sourced a serious tool. That would have been big news, with the sudden availability of source for a tool that does rr-like stuff.
Some other company could do it. Remember that we've gotten Navigator (now Firefox), Blender, OpenSolaris, OpenOffice, and the .net stuff. Governments can surprise us as well; the USA did that with Ghidra.
I'm not expecting it anytime soon, but surprises happen.
That's a possibility, but I don't know of any closed-source tool that does what Pernosco does. Tetrane is the closest I know of, but they aren't as scalable as Pernosco (they don't need to be).
Certainly possible that someone has a secret tool that's as good or better. However, I expect the secret tools in this space are all focused on so-called 'security research', i.e. more like Tetrane and targeted at adversaries of the software under test, rather than its developers, which leads to a somewhat different feature set.
I think some times the conversations about debuggers is one where developers talk past eachother because a developer who uses a high level language debugger in a fancy IDE to debug only high level code, vs. one that uses a native language debugger using a command line tool, will have wildly differnet experiences and expectations of what the tool can do, and especially about how discoverable and simple those features are.
I belong in the first camp (Debugging to me means "running", It's not a tool I dig out to actually de-BUG something, it's just the tool I use to run my code every time).
They also talk past each other because the value of debuggers vastly depends on what you're debugging.
It's relatively easy to get value out of a debugger in a straightforward synchronous userland process.
It's much harder in a complex embedded application full of threads, callbacks, asynchronous execution, and a target where you may need to set up something like remote debugging and potentially tweak the whole system config to make it possible in the first place. This is also more likely to be the kind of system that doesn't run from your fancy IDE..
I do almost all of my work in Clojure and have for nearly 7 years. I don’t actually miss a traditional debugger much. I’ve occasionally tried one out, but an interactive REPL and immutable data means printlns with the occasional conditional to sample an expression are surprisingly pretty good.
If you add generative testing via spec or test.generative, you can simulate lots of the scenarios you’d otherwise be desperate to have a dump of the environment to inspect. If you get there, profilers usually tell the story in adequate detail. Profiling and debugging are related and often share responsibilities.
Often, you can't attach a debugger to the production system, but you can get logs. Also, you are often interested in the whole sequence of events leading up to the problem, not just the current state.
And, to find where to look in the first place, you need to get an idea of what's wrong - a debugger doesn't help with that.
Logging always has a tradeoff between utility and overhead. It would be great to log everything, but it would impact performance too much. So you end up with logging levels, categories, release-only logs, etc.
Pernosco has something for that too. Since it has replay capability, it can replay with more logging than the original execution had. So to some degree, you can have your cake and eat it too. It's not perfect: I think it requires some level of integration with a given logging system, it's not going to do the full I/O to send your logs to where they would ordinarily go (syslogd or whatever), and you can't log values with debug-only code that wasn't compiled into the binary. But it's still a large boost to the fundamental tension around logging.
One of the points of the OP as that a really good debugger does capture and show you "the whole sequence of events leading up to the problem".
It's totally understandable that people prefer good logging to traditional debuggers which really suck. But you have missed the point of the OP that much better debuggers are possible.
Those are orthogonal to each other and you should use both. But in development, a good debugger with data breakpoints, logging and conditional breakpoints is much more focused than an overly verbose log.
Overly verbose in the context of debugging. Logs almost can't be too verbose, because as you said, they should be able to tell you the sequence of events that lead to the error.
Concerning making money with proprietary debuggers, DDT (no, not that one) and Totalview are very expensive parallel debuggers of longstanding. Totalview has some form of reversible debugging and is unusual in not using GDB. They're actually rarely used in my experience, for various reasons -- not just because they're un-affordable for most sites.
I find that the problem is that the better you become at engineering (i.e. designing and writing your code) the worse you become at debugging. This is because in a well designed system you have many (as many as possible) independent components that you can look at one piece at a time and you've written the appropriate unit tests for them. And subsequently you don't spend that much time in the debugger. In my experience this way most bugs (integration bugs aside) are rather trivial. In my own projects (over the past +15 years) most of my debugging involves just running gdb on my unit test until a test assertion fails, setting a breakpoint before the failed assertion and rerunning so that I can step through the computation.
On another note there's actually this thing I'd like to call "debuggability".
I find that if you just thinking of this as a measure of how easy your program is to debug and when making design decisions you think of "which design makes this thing easier to debug" your program will be simplified. To me mean high "debuggability" at design level means cohesive independent components which makes it easy to write unit tests for them which then subsequently makes it easy to debug them as well.
Other thing is sometimes I see people write code like this:
getFoo()->getBar()->doComputation();
or something like
doComputation(getFoo()->GetSausage());
And depending on your debugger this can be super inconvenient to debug through. For example with gdb trying to step into any specific function basically requires you to find that function and set a breakpoint there. Otherwise you'll have to go through a lot of unrelevant stuff (especially if you're for example using std::unique_ptr)
If you used some temporary variables to store the results of those function calls again you'll make your program more "debuggable".
Any decent high-level debugger will provide some way to distinguish between your code and stdlib code when stepping, such that you don't end up inside std::unique_ptr and similar.
Good ones will let you set a breakpoint on a subexpression, so in this case you'd just highlight the call that you want, and it'll stop right before it happens.
My personal pain debugging C++ with GDB recently has been the difficulty of creating objects. I spent a long time trying to figure out how to call operator[](vec2u) on my custom data structure.
In the end, I gave up and built self-introspection tools directly into the program itself. It may not be a general solution, but it worked for me.
There's another class of debugging tools: tracers such as dtrace and bpftrace. There are a lot of bugs that you just can't reproduce in a debugger, or even in a development environment at all. You need to be able to debug in production safely and without huge performance penalties.
I actually looked at learning dtrace recently, wondering if it could help me with a handful of isuses, but so far I've been unable to find information related to its capabilities, or up to date guides / books/ tutorials
I use the Visual Studio debugger on an almost daily basis, and it'll be hard for me to use another tool if/when I leave my industry. Conditional and data breakpoints are gamechangers.
To be frank, I didn't use debuggers much at all throughout my tenure at university, my internship at Amazon, and my first few gigs in the games industry. I still had bugs, but I was able to solve them mostly through printf, gut checks, and dousing rods.
But when I got to AAA console game development, the leads had to show me how to use the immense power of the VS debugger, and now I have difficulty imaging life without it. Different scales of problems require different tooling to manage.
Break-point debuggers take too long in most environments (with a few exceptions). I use linters, compilers and type checkers as my first line of defense. Next, logging and unit tests. Lastly, I like to have in interactive command line REPL with good tab completion if one is available and not to difficult to setup.
In python, for example I use: pycodestyle, pyflakes, mypy and logging, and bpython.
For web apps, reload time is the most important factor. Work on reducing the time to compile, reload and test.
But have you used a really good debugger, e.g. a record-and-replay debugger that lets you jump back in time to where some selected variable was modified?
The whole point of the OP was that just about all debuggers suck, but debuggers can be a lot more powerful.
Second time I saw an ad for this product on this site. Don't get me wrong. I like the idea of innovation in this space. But I think they should just buy advertising space.
Unit tests make debugging suck a bit lest, because of the ability to change the test case (on local machine only!) / and tested code iteratively in combination with the debugger to find the source of the problem, with fast feedback cycle. Debugging using the UI is another tool. Advantage is you can quickly cover new scenarios and change what you do based on new info. Downside is loading time can be a lot longer than running a unit test.
The problem with runtime debuggers is that they are only really useful in a development environment. Outside of that you have to rely on core dumps, logging and various other post analysis tools.
Which is why projects have to built the debugging into the product in one form or another. A topic that is far to broad/long for a posting here, but most projects end up implementing in various forms if they live long enough.
With record-and-replay debugging that is not true. People can and are capturing failures outside the development environment (e.g. in CI) using rr and debugging those recordings in Pernosco, for example.
> In particular, developer culture is that developers don't pay for tools, especially not debuggers.
I blame larger programming teams. It’s a chicken and egg problem: if you want to start using a commercial debugger, you need to buy it for everyone on dev team, so you can share your knowledge and benefit from others’ help; and you are not going to buy a debugger for everyone if you are not using it yourself.
you used to have them: Borland had a good IDE with a good debugger for Pascal, C, ... (even Prolog) in the early 1990s.
IBM's visual age (Smalltalk, Java) was great too: stop in the neigbourhood of the bug, edit the code, and continue without restarting...
Now we're exporting logs from a docker in docker on travis.
O tempora O mores
Sometimes just having a debugger is nice. There's an entire ecosystem based on cheap Arduino boards but the Arduino (teensy, anyway) intentionally hides its debug SWD pieces away to prevent pirating, leaving the developer high and dry with nothing but printf debugging.
It's probably the only part where hardware development (FPGA or ASIC) has a significant edge: "time travel debugging" is the norm, where you have a wave window with your design state cycle by cycle. Can't wait to see more of that in software!
Plan to spend a lot of time first narrowing down the time window of interest so it doesn't overflow its recording buffer, and then waiting for it to run at 1/100 speed.
I used to use it. It was so painful and limited that I think it helped only a single time, and was all in all a net loss in productivity.
WinDbg Preview's Time Travel Debugger can do this: https://docs.microsoft.com/en-us/windows-hardware/drivers/de...
The great advantage of such a debugger is that you can use it to follow data flow: if you see an incorrect value somewhere, you can set a memory breakpoint and then run the program backwards to jump to the point in time where the value was written.
I've only used it with C++ applications so far, but the WinDbg blog claims that it also works when debugging managed code with the SOS debugging extension.
Visual Studio confusingly has the somewhat-similar "Intellitrace", but AFAIK that only records previous states collected for certain events (breakpoints, first-chance-exceptions, ...).
It can't rewind the program an arbitrary point like the time-travel debugger can, so it's not useful to follow data flow.
I was under the impression the Enterprise version of visual studio includes a debugger that functions in a similar manner. Capture the entire execution and allow it to be replayed and shared with others.
> TL;DR Debuggers suck, not using a debugger sucks, and you suck.
While at the outset this post seems like a rant, I believe the opinion expressed is very astute.
In a perfect world, we would all write programs as a composition of mostly small pure functions, with type systems that can enforce provable correctness, and a small part of the code would generate 'effects'.
We don't live in such a world.
We often inherit gargantuan codebases written in an extremely imperative fashion, with a lot of state management baked into functions thousands of lines long with conditional and loop blocks nesting 5 levels deep.
And there's a bug in one of those loops.
A good step debugger that we could reverse step and inspect would save us somewhat in such times of perils. But they don't really exist. Or if they do, don't integrate with your editor.
Disclaimer: I'm a senior dev at rookout.com, which does something similar to Pernosco/rr but tailored to languages used for backend/web like Node.js, Java and Python instead of statically compiled (to machine language) languages like C/C++.
I definitely agree - I've watched people painstakingly stare at code, add print/log statements and waste enormous amounts of time -- and then I asked them why they don't just use a debugger and always got unsatisfactory answers that seem to hide the fact they have a hard time using the tool effectively.
Of course, like you say, using a debugger effectively in many settings is difficult. A multi-threaded program setting a breakpoint and inspecting state difficult, as finding the right thread to inspect can be difficult -- not to mention having other threads alter your debugger's display state by breaking once you're already inspecting the correct thread. I've had this happen with native software (e.g. C++), for which rr is great (but unfortunately I've worked mostly on Windows), but also for backend software written in higher level languages such as Python.
Using a classic debugger effectively in a multithreaded environment requires being aware of the issues you're about to encounter, and knowing how to deal with them with the debugger's tooling -- which ranges between difficult and not possible. For example, with Windbg - it is possible to break on the correct thread, but setting a conditional breakpoint (even a simple one) is nothing less than an incantation to someone who's never used it before. As such, the cost of entry is very high, and I've watched kernel driver developers use prints and asserts to debug.
While at my previous job, I thought that logs are a really crappy way of debugging an issue in production when it arises - if you have all the right information and can fetch it quickly, great. But often, that's not the case - you might end up needing to add logs, or worse - the issue will be in a hotspot where you simply can't log - since you'll have _too much_ data, or you can't log everything for compliance reasons.
I had a similar idea to Rookout's (and even almost accidentally raised money just by talking to the right people, although I was not prepared at the time to start a company), and discovered Rookout later on, going on to use them.
When I left my last job, I joined Rookout because -- like O'Callahan (and Pernosco) -- I'm very much a strong believer that the way debugging is today isn't good enough, and can be a lot better.
Nowadays Rookout's product for interactive debugging backend apps is pretty much complete, works well and fast -- and we're exploring other ways to make debugging easier.
Rookout's a bit different to rr/Pernosco in that it allows you to collect variables from anywhere during runtime (instead of allowing you to replay a recording) -- it's a bit similar to allowing you to add a log that logs locals() anywhere, except without changing your code, redeploying, etc.
I agree but this whole thing just says 'debugging sucks, my commercial tool is great, I won't explain a single detail about it, follow this link' (it is a wrapper around rr). He's preaching to the choir and I still think this post is sleazy.
Also replay debugging doesn't have to be complicated.
This enables replaying sections of a program with hot reloading by requiring that the data structures are serialized. It also catches low level exceptions so that replay and hot reloading won't crash the running program.
There's a great series of posts going into more detail about Pernosco's design and advantages that is linked to in the final paragraph: https://pernos.co/about/
Not an actual debugger, but this tool can record Java applications (unit tests or full-blown applications) while they run, and then you can explore visually what happened through a web interface: http://findtheflow.io.
It's similar to rr because you can go back and forth in time, but unlike rr it doesn't record much of the state (besides return values) and instead focuses primarily on the execution flow by the threads across the different methods, classes and packages.
Author goes on a big rant and says rr solves everything. Big claim with little backing.
There’s tons of languages. The closest I’ve seen to a sane debugging experience is vscode. It works the same across languages. Python, browser js, node js, go, Java etc
Mostly because they all share same underlying debug adapter protocol.
Now if you truly want to make debugging phenomenally better. Get those new apis in the protocol for reverse debugging and the likes. Then make it happen across a bunch of languages. I will pay It. I know most people will.
We just want something uniform that works
Log debugging is easiest. Why ? Because it’s simple AF. Running a 1000 things in parallel. Aggregate the logs and filter what you need.
Going through layers of virtual computation? Physical -> Node pool -> service -> pod -> container. Write to log, aggregate.
There’s been a lot of promise but haven’t seen anything that beats the simplicity of structured logs + good log analyzer tool.
It really is that much better. Once you start using it, it's really hard to go back to a situation where you don't have it. It's like driving a car that can only turn left -- sure, if you need to go right you can always 270 degrees to the left (by rerunning over and over again to get closer to what you want to examine, assuming your problem is reproducible enough), but it feels burdensome and ridiculous.
If AMD fixed their performance counters so that rr could work, I would switch immediately.
I am also a fan of Pernosco. It is especially useful when hooked up to a continuous integration system that can hand you a ready-to-go recording of all failures. You can get assigned a bug with a recording to go along with it. Even if it doesn't look important or tractable to look into the ordinary way, having a recording makes it easy enough to take a glance and check it out that you'll actually do it. At shops that do a lot of system-level programming (as opposed to scripting), the productivity boost ought to be worth a serious amount to the bean counters.