[Fixed]-Caching queryset choices for ModelChoiceField or ModelMultipleChoiceField in a Django form

The reason that ModelChoiceField in particular creates a hit when generating choices – regardless of whether the QuerySet has been populated previously – lies in this line

for obj in self.queryset.all(): 

in django.forms.models.ModelChoiceIterator. As the Django documentation on caching of QuerySets highlights,

callable attributes cause DB lookups every time.

So I’d prefer to just use

for obj in self.queryset:

even though I’m not 100% sure about all implications of this (I do know I do not have big plans with the queryset afterwards, so I think I’m fine without the copy .all() creates). I’m tempted to change this in the source code, but since I’m going to forget about it at the next install (and it’s bad style to begin with) I ended up writing my custom ModelChoiceField:

class MyModelChoiceIterator(forms.models.ModelChoiceIterator):
    """note that only line with # *** in it is actually changed"""
    def __init__(self, field):
        forms.models.ModelChoiceIterator.__init__(self, field)

    def __iter__(self):
        if self.field.empty_label is not None:
            yield (u"", self.field.empty_label)
        if self.field.cache_choices:
            if self.field.choice_cache is None:
                self.field.choice_cache = [
                    self.choice(obj) for obj in self.queryset.all()
                ]
            for choice in self.field.choice_cache:
                yield choice
        else:
            for obj in self.queryset: # ***
                yield self.choice(obj)


class MyModelChoiceField(forms.ModelChoiceField):
    """only purpose of this class is to call another ModelChoiceIterator"""
    def __init__(*args, **kwargs):
        forms.ModelChoiceField.__init__(*args, **kwargs)

    def _get_choices(self):
        if hasattr(self, '_choices'):
            return self._choices

        return MyModelChoiceIterator(self)

    choices = property(_get_choices, forms.ModelChoiceField._set_choices)

This does not solve the general problem of database caching, but since you’re asking about ModelChoiceField in particular and that’s exactly what got me thinking about that caching in the first place, thought this might help.

You can override “all” method in QuerySet
something like

from django.db import models
class AllMethodCachingQueryset(models.query.QuerySet):
    def all(self, get_from_cache=True):
        if get_from_cache:
            return self
        else:
            return self._clone()


class AllMethodCachingManager(models.Manager):
    def get_query_set(self):
        return AllMethodCachingQueryset(self.model, using=self._db)


class YourModel(models.Model):
    foo = models.ForeignKey(AnotherModel)

    cache_all_method = AllMethodCachingManager()

And then change queryset of field before form using (for exmple when you use formsets)

form_class.base_fields['foo'].queryset = YourModel.cache_all_method.all()

Here is a little hack I use with Django 1.10 to cache a queryset in a formset:

qs = my_queryset

# cache the queryset results
cache = [p for p in qs]

# build an iterable class to override the queryset's all() method
class CacheQuerysetAll(object):
    def __iter__(self):
        return iter(cache)
    def _prefetch_related_lookups(self):
        return False
qs.all = CacheQuerysetAll

# update the forms field in the formset 
for form in formset.forms:
    form.fields['my_field'].queryset = qs

I also stumbled over this problem while using an InlineFormset in the Django Admin that itself referenced two other Models. A lot of unnecessary queries are generated because, as Nicolas87 explained, ModelChoiceIterator fetches the queryset everytime from scratch.

The following Mixin can be added to admin.ModelAdmin, admin.TabularInline or admin.StackedInline to reduce the number of queries to just the ones needed to fill the cache. The cache is tied to the Request object, so it invalidates on a new request.

 class ForeignKeyCacheMixin(object):
    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super(ForeignKeyCacheMixin, self).formfield_for_foreignkey(db_field, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        if cache.get(db_field.name):
            formfield.choices = cache[db_field.name]
        else:
            formfield.choices.field.cache_choices = True
            formfield.choices.field.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield

@jnns I noticed that in your code the queryset is evaluated twice (at least in my Admin inline context), which seems to be an overhead of django admin anyway, even without this mixin (plus one time per inline when you don’t have this mixing).

In the case of this mixin, this is due to the fact that formfield.choices has a setter that (to simplify) triggers the re-evaluation of the object’s queryset.all()

I propose an improvement which consists of dealing directly with formfield.cache_choices and formfield.choice_cache

Here it is:

class ForeignKeyCacheMixin(object):

    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super(ForeignKeyCacheMixin, self).formfield_for_foreignkey(db_field, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        formfield.cache_choices = True
        if db_field.name in cache:
            formfield.choice_cache = cache[db_field.name]
        else:
            formfield.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield

Here is another solution for preventing ModelMultipleChoiceField from re-fetching it’s queryset from database. This is helpful when you have multiple instances of the same form and do not want each form to re-fetch the same queryset. In addition the queryset is a parameter of the form initialization, allowing you e.g. to define it in your view.

Note that the code of those classes have changed in the meantime. This solution uses the versions from Django 3.1.

This example uses a many-2-many relation with Django’s Group

models.py

from django.contrib.auth.models import Group
from django.db import models


class Example(models.Model):
    name = models.CharField(max_length=100, default="")
    groups = models.ManyToManyField(Group)
    ...

forms.py

from django.contrib.auth.models import Group
from django import forms


class MyModelChoiceIterator(forms.models.ModelChoiceIterator):
    """Variant of Django's ModelChoiceIterator to prevent it from always re-fetching the
    given queryset from database.
    """

    def __iter__(self):
        if self.field.empty_label is not None:
            yield ("", self.field.empty_label)
        queryset = self.queryset
        for obj in queryset:
            yield self.choice(obj)


class MyModelMultipleChoiceField(forms.ModelMultipleChoiceField):
    """Variant of Django's ModelMultipleChoiceField to prevent it from always
    re-fetching the given queryset from database.
    """

    iterator = MyModelChoiceIterator

    def _get_queryset(self):
        return self._queryset

    def _set_queryset(self, queryset):
        self._queryset = queryset
        self.widget.choices = self.choices

    queryset = property(_get_queryset, _set_queryset)


class ExampleForm(ModelForm):
    name = forms.CharField(required=True, label="Name", max_length=100)
    groups = MyModelMultipleChoiceField(required=False, queryset=Group.objects.none())

    def __init__(self, *args, **kwargs):
        groups_queryset = kwargs.pop("groups_queryset", None)
        super().__init__(*args, **kwargs)
        if groups_queryset:
            self.fields["groups"].queryset = groups_queryset

    class Meta:
        model = Example
        fields = ["name", "groups"]

views.py

from django.contrib.auth.models import Group
from .forms import ExampleForm


def my_view(request):
    ...    
    groups_queryset = Group.objects.order_by("name")
    form_1 = ExampleForm(groups_queryset=groups_queryset)
    form_2 = ExampleForm(groups_queryset=groups_queryset)
    form_3 = ExampleForm(groups_queryset=groups_queryset)
    ```

@lai With Django 2.1.2 I had to change the code in the first if-statement from formfield.choice_cache = cache[db_field.name] to formfield.choices = cache[db_field.name] as in the answer from jnns. In the Django version 2.1.2 if you inherit from admin.TabularInline you can override the method formfield_for_foreignkey(self, db_field, request, **kwargs) directly without the mixin. So the code could look like this:

class MyInline(admin.TabularInline):
    model = MyModel
    formset = MyModelInlineFormset
    extra = 3

    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super().formfield_for_foreignkey(db_field, request, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        formfield.cache_choices = True
        if db_field.name in cache:
            formfield.choices = cache[db_field.name]
        else:
            formfield.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield

In my case I also had to override get_queryset to get the benefit from select_related like this:

class MyInline(admin.TabularInline):
    model = MyModel
    formset = MyModelInlineFormset
    extra = 3

    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        formfield = super().formfield_for_foreignkey(db_field, request, **kwargs)
        cache = getattr(request, 'db_field_cache', {})
        formfield.cache_choices = True
        if db_field.name in cache:
            formfield.choices = cache[db_field.name]
        else:
            formfield.choice_cache = [
                formfield.choices.choice(obj) for obj in formfield.choices.queryset.all()
            ]
            request.db_field_cache = cache
            request.db_field_cache[db_field.name] = formfield.choices
        return formfield

    def get_queryset(self, request):
        return super().get_queryset(request).select_related('my_field')