Is there a way to attach a debugger to a multi-threaded Python process?

Use Winpdb. It is a platform independent graphical GPL Python debugger with support for remote debugging over a network, multiple threads, namespace modification, embedded debugging, encrypted communication and is up to 20 times faster than pdb.

Features:

• GPL license. Winpdb is Free Software.
• Compatible with CPython 2.3 through 2.6 and Python 3000
• Compatible with wxPython 2.6 through 2.8
• Platform independent, and tested on Ubuntu Gutsy and Windows XP.
• User Interfaces: rpdb2 is console based, while winpdb requires wxPython 2.6 or later.

_{(source: winpdb.org)}

You can attach a debugger to a multi-threaded Python process, but you need to do it at the C level. To make sense of what’s going on, you need the Python interpreter to be compiled with symbols. If you don’t have one, you need to download source from python.org and build it yourself:

./configure --prefix=/usr/local/pydbg
make OPT=-g
sudo make install
sudo ln -s /usr/local/pydbg/bin/python /usr/local/bin/dbgpy

Make sure your workload is running on that version of the interpreter. You can then attach to it with GDB at any time. The Python folks have included a sample “.gdbinit” in their Misc directory, which has some useful macros. However it’s broken for multi-threaded debugging (!). You need to replace lines like this

while $pc < Py_Main || $pc > Py_GetArgcArgv

with the following:

while ($pc < Py_Main || $pc > Py_GetArgcArgv) && ($pc < t_bootstrap || $pc > thread_PyThread_start_new_thread)

Otherwise commands like pystack won’t terminate on threads other than the main thread. With this stuff in place, you can do stuff like

gdb> attach <PID>
gdb> info threads
gdb> thread <N>
gdb> bt
gdb> pystack
gdb> detach

and see what’s going on. Kind of.

You can parse what the objects are with the “pyo” macro. Chris has some examples on his blog.

Good luck.

(Shoutout for Dan’s blog for some key information for me, notably the threading fix!)

My experience debugging multi-threaded programs in PyDev (Eclipse on Windows XP) is, threads created using thread.start_new_thread could not be hooked, but thread created using threading.Thread could be hooked. Hope the information is helpful.

If you mean the pydb, there is no way to do it. There was some effort in that direction:
see the svn commit, but it was abandoned. Supposedly winpdb supports it.

What platform are you attempting this on? Most debuggers allow you to attach to a running process by using the process id.
You can either output the process id via logging or using something like Task Manager.
Once that is achieved it will be possible to inspect individual threads and their call stacks.

EDIT: I don’t have any experience with GNU Debugger (GDB), which is cross platform, however I found this link and it may start you on the right path. It explains how to add debug symbols (handy for reading stack traces) and how to instruct gdb to attach to a running python process.

pdbinject allows you to inject pdb into an already running python process.

The pdbinject executable only works under python2, but can inject into python3 just fine too.

PyCharm IDE allows attaching to a running Python process since version 4.0.

Here is described how to do that.

This can be used as a dead simple “remote” debugger:

import sys
import socket
import pdb

def remote_trace():
    server = socket.socket()
    server.bind(('0.0.0.0', 12345))
    server.listen()
    client, _= server.accept()
    stream = client.makefile('rw')
    sys.stdin = sys.stdout = sys.stderr = stream
    pdb.set_trace()

remote_trace()

# Execute in the shell: `telnet 127.0.0.1 12345`

On Windows it’s easier to use Netcat instead of Telnet (which will also work on linux).

python3 provides gdb extensions. Using
them, gdb can attach to a running program, select a thread and print its
python backtrace.

On Debian (since at least Buster) the extensions are part of the
python3.x-dbg package (ex. python3.10-dbg installs
/usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py) and gdb auto-loads them.

Example with a simple threaded python script:

#!/usr/bin/env python3

import signal
import threading

def a():
    while True:
        pass

def b():
    while True:
        signal.pause()

threading.Thread(target=a).start()
threading.Thread(target=b).start()

Running gdb:

[email protected]:~$ ps -C python3 -L
    PID     LWP TTY          TIME CMD
   1215    1215 pts/0    00:00:00 python3
   1215    1216 pts/0    00:00:19 python3
   1215    1217 pts/0    00:00:00 python3
[email protected]:~$ gdb -p 1215
GNU gdb (Debian 10.1-2+b1) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
(gdb) info auto-load python-scripts
Loaded  Script
Yes     /usr/share/gdb/auto-load/usr/bin/python3.10-gdb.py
(gdb) info threads
  Id   Target Id                                  Frame
* 1    Thread 0x7f2f034b4740 (LWP 1215) "python3" 0x00007f2f036a60fa in __futex_abstimed_wait_common64 ([email protected]=0x7f2ef4000b60, [email protected]=0,
    [email protected]=0, [email protected]=0x0, private=<optimized out>,
    [email protected]=true) at ../sysdeps/nptl/futex-internal.c:74
  2    Thread 0x7f2f02ea7640 (LWP 1216) "python3" 0x000000000051b858 in _PyEval_EvalFrameDefault
    (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:3850
  3    Thread 0x7f2f026a6640 (LWP 1217) "python3" 0x00007f2f036a3932 in __libc_pause ()
    at ../sysdeps/unix/sysv/linux/pause.c:29
(gdb) thread 2
(gdb) py-bt
Traceback (most recent call first):
  File "/root/./threaded.py", line 7, in a
    while True:
  File "/usr/lib/python3.10/threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 966, in _bootstrap
    self._bootstrap_inner()
(gdb)

We can confirm that thread 1216 which used the most cpu time according to
ps is indeed the thread running function a() that is busy-looping.