Python has a built-in API for tracing: https://docs.python.org/3/library/sys.html#sys.settrace

Here's an example of how to use it (obviously a minimal case, you can customise your handler do do whatever you need):

    def mytrace(frame, event, arg):
        if event == "call":
            print("call", frame.f_code.co_name, frame.f_locals)
        elif event == "return":
            print("return", frame.f_code.co_name, arg)
        return mytrace

    import sys
    sys.settrace(mytrace)

For example, given the following code:

    def three(z):
        return 3

    def two(y):
        return three("BBB") - 1

    def one():
        return two("AAA") - 1

    one()

It will output:

    call one {}
    call two {'y': 'AAA'}
    call three {'z': 'BBB'}
    return three 3
    return two 2
    return one 1

I confess I'm surprised the builtin wasn't one of the demoed options. Curious what the reason is.

I just pushed an addendum, and it is being rendered right now.

Kudos on making the update! Makes sense that you might not find it if you didn't know it was there. Python's "batteries included" means most things you may want to do, you should check to see if there is a standard lib option, as well. Sometimes it is barebones enough that you may want more, of course. So, really nice to see all of the options you presented.

Pdb also employs this approach for debug. I had wondered about pdb's implementation, when I saw it, I felt a little surprised.

Wow, I can't believe I didn't know about this until now! I'm the lucky 10k[0] today.

[0] https://xkcd.com/1053/

There are many different tools, but when I want to quickly understand what the code was doing I have been reaching for pysnooper: https://pypi.org/project/PySnooper/

Has a pretty nice and importantly colored output by default, which I find much easier to follow. In many cases snooping one specific function has done the job. It has an easy to use decorator to decorate any function without having to otherwise intercept the main python invocation or set something up at the start. Good for libraries etc.

pysnooper has mostly been enough for me but the "snoop" mentioned in the article seems to be a superset and more. Will have to give it a go next time.

Outside of this if I want to understand performance or more roughly what code paths are executing (and not a per-statement trace) then austin and py-spy are both amazing.. convert to flamescope and drop into flamescope.app.

A more traditional debugger for Python.

https://github.com/inducer/pudb

I'll throw in my own too: https://github.com/kunalb/panopticon

Fantastic article and something I've been looking for better tools for. None of them seem to have my ideal workflow though -- providing an external manifest of what and how to trace that can be used against an existing program without changing the code. Something like Otel's auto instrumentation.

Fyi, the table of contents gets cut off (its box is smaller than its contents) at zoom levels of 90% or larger for me. Here's a screenshot at 100% zoom https://imgur.com/a/0t9cKs5

Great article!

What's the modern practical use case for tracing in python? Environments without an IDE/remote debugger?

I can't recall the last time I needed to enable tracing, in the last couple of decades, besides bash scripts.

Originally the question was "What is Honeycomb.io and Opentracing doing and how can I get this into my existing application?" on IRC. On explaining, the discussion went into various directions (Fred Fish C Debugging Macros and tracing Python).