multiprocessing: the Resource Tracker process is never reaped

[viktorivanov]

BPO	44724
Nosy	@moreati
Files	multi.py: Minimal program leaking resource tracker zombies

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-23.13:29:53.414>
labels = ['3.7', 'library', '3.9', 'performance']
title = 'multiprocessing: the Resource Tracker process is never reaped'
updated_at = <Date 2021-11-24.23:24:39.596>
user = 'https://bugs.python.org/viktorivanov'

bugs.python.org fields:

activity = <Date 2021-11-24.23:24:39.596>
actor = 'Alex.Willmer'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-07-23.13:29:53.414>
creator = 'viktor.ivanov'
dependencies = []
files = ['50173']
hgrepos = []
issue_num = 44724
keywords = []
message_count = 1.0
messages = ['398053']
nosy_count = 2.0
nosy_names = ['Alex.Willmer', 'viktor.ivanov']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue44724'
versions = ['Python 3.7', 'Python 3.9']

Linked PRs

• gh-88887: Cleanup multiprocessing.resource_tracker.ResourceTracker upon deletion #130429
• [3.13] gh-88887: Cleanup multiprocessing.resource_tracker.ResourceTracker upon deletion (GH-130429) #131516
• [3.12] gh-88887: Cleanup multiprocessing.resource_tracker.ResourceTracker … (GH-130429) #131530

[viktorivanov]

The multiprocessing.resource_tracker instance is never reaped, leaving zombie processes.

There is a waitpid() call for the ResourceTracker's pid but it is in a private method _stop() which seems to be only called from some test modules.

Usually environments have some process handling zombies but if python is the "main" process in a container, for example, and runs another python instance that does something leaking a ResourceTracker process, zombies start to accumulate.

This is easily reproducible with a couple of small python programs as long as they are not run from a shell or another parent process that takes care of forgotten children.

It was originally discovered in a docker container that has a python program as its entry point (celery worker in an airflow container) running other python programs (dbt).

The minimal code is available on Github here: https://github.com/viktorvia/python-multi-issue

The attached multi.py is leaking resource tracker processes, but just running it from a full-fledged development environment will not show the issue.

Instead, run it via another python program from a Docker container:

Dockerfile:
---
FROM python:3.9

WORKDIR /usr/src/multi

COPY . ./

CMD ["python", "main.py"]
---

main.py:
---

from subprocess import run
from time import sleep

while True:
    result = run(["python", "multi.py"], capture_output=True)
    print(result.stdout.decode('utf-8'))
    result = run(["ps", "-ef", "--forest"], capture_output=True)
    print(result.stdout.decode('utf-8'), flush=True)
    sleep(1)

---

When the program is run it will accumulate 1 zombie on each run:
---

$ docker run -it multi python main.py
[1, 4, 9]

UID PID PPID C STIME TTY TIME CMD
root 1 0 11 11:33 pts/0 00:00:00 python main.py
root 8 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 17 1 0 11:33 pts/0 00:00:00 ps -ef --forest

[1, 4, 9]

UID PID PPID C STIME TTY TIME CMD
root 1 0 6 11:33 pts/0 00:00:00 python main.py
root 8 1 3 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 28 1 0 11:33 pts/0 00:00:00 ps -ef --forest

[1, 4, 9]

UID PID PPID C STIME TTY TIME CMD
root 1 0 4 11:33 pts/0 00:00:00 python main.py
root 8 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 3 11:33 pts/0 00:00:00 [python] <defunct>
root 30 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 39 1 0 11:33 pts/0 00:00:00 ps -ef --forest

[1, 4, 9]

UID PID PPID C STIME TTY TIME CMD
root 1 0 3 11:33 pts/0 00:00:00 python main.py
root 8 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 30 1 4 11:33 pts/0 00:00:00 [python] <defunct>
root 41 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 50 1 0 11:33 pts/0 00:00:00 ps -ef --forest
---

Running from a shell script, or just another python program that handles SIGCHLD by calling wait() takes care of the zombies.

[babaMar]

The same happens for python3 -c from multiprocessing.semaphore_tracker import main;main(3), when the parent exits, this process doesn't get reaped

[luccabb]

or just another python program that handles SIGCHLD by calling wait() takes care of the zombies.

we've fixed this issue by calling the below on our code:

import os
import signal
from types import FrameType

def reap_children(signum: int, frame: Optional[FrameType]) -> None:
    try:
        # -1 meaning wait for any child process, see https://linux.die.net/man/2/waitpid
        # WNOHANG for waitpid to return immediately instead of waiting, if there is no child process ready to be noticed.
        # see https://www.gnu.org/software/libc/manual/html_node/Process-Comption.html#index-WNOHANG
        while os.waitpid(-1, os.WNOHANG)[0] > 0:
            pass
    except ChildProcessError:
        pass
# SIGCHLD is sent to a process to indicate a child process stopped or terminated
# https://man7.org/linux/man-pages/man7/signal.7.html
signal.signal(signal.SIGCHLD, reap_children)

[vstinner]

Fixed by f53e7de.