12👍
You can write the rows to a file in the format
“field1”, “field2”, .. and then use LOAD DATA to load them
data = '\n'.join(','.join('"%s"' % field for field in row) for row in data)
f= open('data.txt', 'w')
f.write(data)
f.close()
Then execute this:
LOAD DATA INFILE 'data.txt' INTO TABLE db2.my_table;
12👍
For MySQL specifically, the fastest way to load data is using LOAD DATA INFILE, so if you could convert the data into the format that expects, it’ll probably be the fastest way to get it into the table.
- Models inside tests – Django 1.7 issue
- Fabric: How can I unit test my fabfile?
- Load Balancing Multiple Django Webservers
- Should south migration files be added to source control?
4👍
If you don’t LOAD DATA INFILE as some of the other suggestions mention, two things you can do to speed up your inserts are :
- Use prepared statements – this cuts out the overhead of parsing the SQL for every insert
- Do all of your inserts in a single transaction – this would require using a DB engine that supports transactions (like InnoDB)
- Django/python How to get a list of id's from a list of Objects
- "[Errno 101] Network is unreachable" when trying to send email using Django
- How is pip install using git different than just cloning a repository?
- How to Hash Django user password in Django Rest Framework?
4👍
If you can do a hand-rolled INSERT statement, then that’s the way I’d go. A single INSERT statement with multiple value clauses is much much faster than lots of individual INSERT statements.
2👍
Regardless of the insert method, you will want to use the InnoDB engine for maximum read/write concurrency. MyISAM will lock the entire table for the duration of the insert whereas InnoDB (under most circumstances) will only lock the affected rows, allowing SELECT statements to proceed.
- DRF create method in viewset or in serializer
- How to have a nested inline formset within a form in Django?
1👍
what format do you receive? if it is a file, you can do some sort of bulk load: http://www.classes.cs.uchicago.edu/archive/2005/fall/23500-1/mysql-load.html
- How do I add authentication and endpoint to Django Celery Flower Monitoring?
- Django CORS Access-Control-Allow-Origin missing
1👍
This is unrelated to the actual load of data into the DB, but…
If providing a “The data is loading… The load will be done shortly” type of message to the user is an option, then you can run the INSERTs or LOAD DATA asynchronously in a different thread.
Just something else to consider.
1👍
I donot know the exact details, but u can use json style data representation and use it as fixtures or something. I saw something similar on Django Video Workshop by Douglas Napoleone. See the videos at http://www.linux-magazine.com/online/news/django_video_workshop. and http://www.linux-magazine.com/online/features/django_reloaded_workshop_part_1. Hope this one helps.
Hope you can work it out. I just started learning django, so I can just point you to resources.