[Fixed]-How to ensure task execution order per user using Celery, RabbitMQ and Django?

by

4👍

If you configure the celery workers so that they can only execute one task at a time (see worker_concurrency setting), then you could enforce the concurrency that you need on a per user basis. Using a method like

NUMBER_OF_CELERY_WORKERS = 10

def get_task_queue_for_user(user):
    return "user_queue_{}".format(user.id % NUMBER_OF_CELERY_WORKERS)

to get the task queue based on the user id, every task will be assigned to the same queue for each user. The workers would need to be configured to only consume tasks from a single task queue.

It would play out like this:

  1. User 49 triggers a task

  2. The task is sent to user_queue_9

  3. When the one and only celery worker that is listening to user_queue_9 is ready to consume a new task, the task is executed

This is a hacky answer though, because

  • requiring just a single celery worker for each queue is a brittle system — if the celery worker stops, the whole queue stops

  • the workers are running inefficiently

👤Mark Chackerian

Leave a comment