[Fixed]-How to prevent django from loading objects in memory when using `delete()`?

by

6👍

You can use a function like this to iterate over an huge number of objects without using too much memory:

import gc

def queryset_iterator(qs, batchsize = 500, gc_collect = True):
    iterator = qs.values_list('pk', flat=True).order_by('pk').distinct().iterator()
    eof = False
    while not eof:
        primary_key_buffer = []
        try:
            while len(primary_key_buffer) < batchsize:
                primary_key_buffer.append(iterator.next())
        except StopIteration:
            eof = True
        for obj in qs.filter(pk__in=primary_key_buffer).order_by('pk').iterator():
            yield obj
        if gc_collect:
            gc.collect()

Then you can use the function to iterate over the objects to delete:

for obj in queryset_iterator(HugeQueryset.objects.all()):
    obj.delete()

For more information you can check this blog post.

👤Augusto Destrero

2👍

You can import django database connection and use it with sql to delete. I had exact same problem as you do and this helps me a lot. Here’s some snippet(I’m using mysql by the way, but you can run any sql statement):

from django.db import connection
sql_query = "DELETE FROM usage WHERE date < '%s' ORDER BY date" % date
cursor = connection.cursor()
try:
    cursor.execute(sql_query)
finally:
    c.close()

This should execute only the delete operation on that table without affecting any of your model relationships.

👤Shang Wang

Leave a comment