[Solved]-How to handle Python multiprocessing database concurrency, specifically with django?

by

9👍

Okay, so I determined (with help of a friend) that the issue is that django is using the same database connection for all the processes. Normally, when you have concurrent db requests, they are either in the same thread (in which case GIL kicks in) or they are on separate threads, in which case django makes different database connections. But with multiprocessing, python makes deepcopies of everything, so it’s passing the same database connection to the subprocesses, and then they step on each other until it breaks.

Solution is to trigger a new db connection from within each subprocess (which is relatively quick).

from django import db
...
def sub_process():
    db.close_connection()
    #the rest of the sub_process' routines

#code that calls sub_process with the pool

Went back and forth from having that line, and not having that line, and definitely fixes everything.

👤wdahab

3👍

Actually i have recently got the same problems, and see this post: Django multiprocessing and database connections
…
and just invoke the connection closing operation in subprocesses:

from django.db import connection 
connection.close()
👤Yx.Ac

Leave a comment