Trace function execution in large Python projects

David Y.

The Problem

I have a large Python project with many different inter-related functions and module files. I would like to see which functions are being executed in what order without having to add print statements everywhere. What would be the best and simplest way to accomplish this?

The Solution

Two of the best tools for viewing function execution order in a large project are Hunter and VizTracer. Both are easy to use and produce an appealing visual output of the function execution order in a given project.

To demonstrate this, let’s consider the following toy project consisting of three Python files: trace_test.py, lib_one.py, and lib_two.py:

# trace_test.py

from lib_one import e

def a():
    b()

def b():
    c()

def c():
    d()

def d():
    e()

a()

# lib_one.py

from lib_two import g

def e():
    f()

def f():
    g()

# lib_two.py

def g():
    h()

def h():
    print("Hello!")

This project calls several different functions across all three files. If we run tracetest.py, it will print the following:

Hello!

Below, we’ll visualize this code’s execution with both VizTracer and Hunter.

VizTracer

To produce a graphical, interactive visualization of the code’s execution without adding any additional code, we can install and run VizTracer:

pip install viztracer
viztracer trace_test.py

This will produce output something like the following:

hello
Total Entries: 405
Use the following command to open the report:
vizviewer /tmp/result.json

Execute the vizviewer command above to open a browser window showing an interactive tracing graph like the one shown in the project’s README. Click on different parts of the graph to see more details in the bottom panes.

Hunter

If VizTracer provides too much complexity and unneeded functionality, we can use Hunter instead, to show a simple, terminal-based diagram of our code’s execution.

First, install it with PIP:

pip install hunter

Then, add the following lines to the top of trace_test.py:

import hunter # new import

hunter.trace() # start tracing

from lib_one import e

def a():
    b()

# ...

Now run python trace_test.py. You should see output similar to that below:

/tmp/trace_test.py:6     call      => a()
/tmp/trace_test.py:7     line         b()
/tmp/trace_test.py:9     call         => b()
/tmp/trace_test.py:10    line            c()
/tmp/trace_test.py:12    call            => c()
/tmp/trace_test.py:13    line               d()
/tmp/trace_test.py:15    call               => d()
/tmp/trace_test.py:16    line                  lib_one.e()
/tmp/lib_one.py:3     call                  => e()
/tmp/lib_one.py:4     line                     f()
/tmp/lib_one.py:6     call                     => f()
/tmp/lib_one.py:7     line                        lib_two.g()
/tmp/lib_two.py:1     call                        => g()
/tmp/lib_two.py:2     line                           h()
/tmp/lib_two.py:4     call                           => h()
/tmp/lib_two.py:5     line                              print("Hello!")
Hello!
/tmp/lib_two.py:5     return                         <= h: None
/tmp/lib_two.py:2     return                      <= g: None
/tmp/lib_one.py:7     return                   <= f: None
/tmp/lib_one.py:4     return                <= e: None
/tmp/trace_test.py:16    return             <= d: None
/tmp/trace_test.py:13    return          <= c: None
/tmp/trace_test.py:10    return       <= b: None
/tmp/trace_test.py:7     return    <= a: None

This will be followed by additional tracing of Python’s shutdown process and the Hunter library itself. To suppress the tracing of functions from Python’s built-in libraries, we can add stdlib=False as a keyword argument to hunter.trace(). More information is provided in the official documentation.

As an alternative to changing our code to use Hunter, make Python import it automatically by setting the PYTHONHUNTER environment variable before execution:

PYTHONHUNTER="" python trace_test.py

This will have the same effect as adding those two Hunter lines above, even if the environment variable is set to an empty string. We can put any keyword arguments we want to pass to hunter.trace in PYTHONHUNTER. For example:

PYTHONHUNTER="stdlib=False" python trace_test.py

This is equivalent to:

import hunter
hunter.trace(stdlib=False)