If anyone is curious about what this would look like on a whole-system level, there is a series of patches [1] to QEMU that enables deterministic replay and reverse execution (i.e., step backward in gdb).

One can also use PANDA [2], but it doesn't currently support the gdb interface in replay mode.

[1] http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg0... [2] https://github.com/moyix/panda

What's new compared to GDB's Process Record and Replay [1] and Reverse Debugging [2]? The ability to save the traces? I am not a (regular) GDB user.

[1] https://sourceware.org/gdb/wiki/ProcessRecord

[2] https://sourceware.org/gdb/wiki/ReverseDebug

What other languages/frameworks have similar tools?

I wrote something like this for cgi-bin scripts in 1996. Problem: you don't necessarily see everything happening in the browser, especially someone else's, especially if they're remote (or unknown), and the only evidence might be a a line in the error log. "playback <script>" would create a wrapper script that saved all environment/server variables and POST data to a file (for later playback) and then call the original script. When run from the command line or debugger, a file containing the selected playback info would be "re-hydrated" into the proper settings, and then call the original script, and it would run as it had previously on the server.

There was a library to extend this to arbitrary data (within the framework). The idiom was something like (after some initial setup):

  if ( $PLAYBACK_MODE ) {
    $result1 = get_playback_value('result1');
  } else {
     $result1 = normal_function_call();
     save_playback_value('result1', $result1);
  }

Which is pretty much portable to any language. If your language supports it, maybe automagically wrap normal_function_call so the caller doesn't have to--if you want this for every call. Recording multiple values could be done by appending something to the identifier (e.g. "result1:$i"). Generally, too much data collected in tight loops (or every db request, etc.), but you could programmatically turn it on in certain cases, only save after an assertion has failed, etc.

Playback use can be great for debugging, somewhere in between unit tests and end-to-end testing, kind of like a mock for medium-level complexity. Decouple data, events, etc. from the context in which they occur ("the real world") for later examination in a more controlled environment. Helping make irreproducible results more produceable.

Wow. That's impressive. Only a factor of 1.2 slowdown for full capture of execution history on a program the size of Firefox. This could, if properly used, eliminate the closing of bug reports as "can't reproduce". It may mean sending in gigabytes of bug trace data, though.

rr can compress traces, so while it's big, it's not as big as you think.

https://github.com/mozilla/rr/issues/700

record/playback tools are immensely useful for debugging.

For instance, with rr, you can easily playback the same run through valgrind to isolate complex memory issues that would be very hard to debug by looking just at the final state.

You can easily use rr -g and gdb to debug issues that couldn't be stepped back using gdb's (limited) reverse execution.

All of this in the same development environment and negligible impact. If that's not enough, with qemu+kvm+gdb you can push it even further, though at a higher speed penalty.

rr recently got amd64 support, and it's in debian as well.

In the same vein, for OpenGL there's vogl from Valve: https://github.com/ValveSoftware/vogl

(hint: if some debian maintainer is reading this, a package of vogl would also be nice!)

This is great - something I always wished I have as a tool.

I haven't found it explicitly, but this is Linux-only, right? And I guess porting to other platforms is not too trivial.

Btw., why is this not written in Rust?

I took a fast look at the code - very modern and nice C++11 code. And quite a few Linux dependencies (Linux headers + /proc usage).

This is awesome! Is it a drop-in replacement for gdb? The docs seem to suggest so but just wanted to know if there are any fine prints.

You use it with GDB, it wraps it and makes the execution under gdb deterministic, so you can set breakpoints earlier in your execution by going back to the start and replaying up to your breakpoint. This sounds a bit inefficient, but compared to reversible debugging it works quite well.

You should watch the videos