[Fixed]-Django – rendering many templates using templatetags is very slow

You might want to try the caching template loader, django.template.loaders.cached.Loader – it should certainly reduce the amount of IO needed.

Edit to add You need to be careful of assuming that just because the majority of time is spent in the template rendering phase, that the query count is not to blame. Don’t forget that querysets are lazy, and unless you’re specifically slicing or iterating them in the view, they will only be evaluated when the template is loaded. I would say that reducing your query count through good use of select_related and other techniques should be significant help.

I’m assuming since you’re using the debug toolbar that you’re getting these numbers in development. However, because of this, these are not “real” numbers.

The built-in Django server is good for development, but it has a number of shortcomings that make it much slower that a real webserver would be. First, it’s single threaded, so that means no parallel requests. This also means that IO ops are discrete. Second, it’s tasked with not just serving requests to Django, but also static resources.

Long and short, if you want to truly profile your site for page load times, you’ll need to install a true webserver locally. Basically set it up like you would in your production environment. I’d be willing to wager the request times will be far better, then.

I had the same issue, and maybe for the same reason. I optimized the performance of a custom template tag. The number of db requests fell from 640 to 2, and the resulting db time was under 20ms. My page however, had become slower! 7s -> 10s. Sigh. I tried a cached template loader, without effect.

I had given up, disabled the django debug toolbar, after which the response time fell to 1.2s, unbelievable!! In my case, the huge response time was only caused by the debug toolbar! A related issue can be found here. I’m not diving deeper in this toolbar issue, as I was planning to cache the template tag using template fragment caching anyways. During development I have 10s response time every 15 minutes, which I can live with. Anyways, hope this helps.

This might not apply to this particular problem but in some cases it helped me to use select_related on the queries. Might not be the case but it might lower your query count.

Time spent on queries is relatively little.
But ORM takes much more time to generate these queries and parse the results into the model instances.

So, despite huge number of queries your app is CPU bound because of the slow ORM (in your case it may take a second). So you have to reduce number of queries anyway.

Most probably the queries are made inside of your template tag. So you have to get desired data in a few queries, and set it on the photo instances.

{% for photo in photos|annotate_comment_count %}
   ...
{% endfor %}

def annotate_comment_count(photo_list):
    counts = dict(Comment.objects.filter(photo__in=photo_list).values('photo') \
                                 .annotate(count=models.Count('id')))
    for photo in photo_list:
        photo.comments_count = counts[photo.pk]
    return photo_list

So, inside your templatetag you don’t have to query comments count for a single photo, but you already have this information in comments_count attribute. And you achieved this in one query.