[Fixed]-Celery vs. ProcessPoolExecutor / ThreadPoolExecutor

by

7👍

✅

You need to use celery if:

  1. You want to scale easily and independently from your webserver
  2. You want a way to monitor your task and retry them if they fail
  3. You want to create more advanced task execution patterns (ex. chain them)

In addition to this is a very mature library with side projects that helps you also on UI presentation side, have a look at Jobtastic.

If you don’t need any of the listed point and you just need to execute this task without caring to much about status and without particular needs of scalability than just keep it simple.

About using ThreadPoolExecutor or ProcessPoolExecutor just keep in mind that the second will be able to receive and return only pickable objects and that the first will spawn child thread attached to your main process (probably your webserver if you are not using it inside another detached process) so the approach of mix them can make sense depending on the details of your implementation.

👤Mauro Rocco

Leave a comment