Skip to content

pawl/celery_pyamqp_memory_leak

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This repo is an attempt to reproduce a memory leak issue with Celery + pyamqp's heartbeat functionality.

More info about the memory leak:

  1. Start Celery and RabbitMQ by installing docker & docker-compose and running:
    docker-compose up
  2. Wait until you start seeing ConnectionResetError: [Errno 104] Connection reset by peer error messages.
  3. Run sudo docker stats in another terminal window to watch memory usage.
  4. Watch memory usage gradually increase by 300 KB every 10 seconds.

In this example, the broker_heartbeat is set to 1 and this will make RabbitMQ constantly close connections because Celery isn't responding to heartbeats fast enough. This may simulate what can happen over time when celery fails to respond to heartbeats due to high cpu usage or long running tasks.

It may also be possible to increase memory usage with a higher broker_heartbeat by killing RabbitMQ connections with this command:

sudo docker-compose exec rabbitmq /bin/sh -c 'rabbitmqadmin -f tsv -q list connections name | while read conn ; do rabbitmqadmin -q close connection name="${conn}" ; done'

About

An attempt to reproduce the pyamqp + celery memory leak from: https://github.com/celery/celery/issues/5047

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published