[Fixed]-Python: Can dumpdata cannot loaddata back. UnicodeDecodeError

What worked for me is following these steps:

- Open the file in regular notepad
- Select save as
- Select encoding "UTF-8" (Not "UTF-8 (With BOM)")
- Save the file.

Now you can use loaddata.

However, this only works for files that are small enough for notepad to open.

0xff in position 0 looks like the start of a little-endian UTF-16 byte order marker to me. Notepad’s “Unicode” save mode is little-endian UTF-16, so that makes sense if you saved your json from Notepad after creating it. Notepad will keep the byte order marker even in utf-8, which could plausibly cause loaddata to fail to parse it.

If you don’t have your un-edited json still handy, you’ll need to remove the BOM – personally I’d use emacs, but another answer suggested this stand-alone Windows .exe:

http://www.bryntyounce.com/filebomdetector.htm

On windows, if you run your standard dumpdata command with -Xutf8 it has always solved this problem for me:

python -Xutf8 manage.py dumpdata app.mymodel > app/fixtures/mymodel.json

Here is an article for reference:
https://dev.to/methane/python-use-utf-8-mode-on-windows-212i

After good research, I got the solution.
In my case, datadump.json file was having the issue.

• Simply Open the file in notepad format
• Click on save as option
• Go to encoding section below & Click on "UTF-8"
• Save the file.

Now you can try running the command. You are good to go 🙂

For your reference, I have attached images below.

Notepad

Save as

UTF-8

I found one way to solve this issue by manually re-output a new binary json file with following code, rb stand for “read and binary”, wb for “write and binary”.

First, go to shell:

python manage.py shell

Second, rewrite the test.json to a binary file:

with open('path/to/test.json', 'rb') as f:
    data = f.read()
newdata = open('newfile.json', 'wb')
newdata.write(data)
newdata.close()
exit()

Then you can load the file:

python manage.py loaddata newfile.json

Above code works for me. Hope it can help you as well.

i encountered the same problem when loading data. it has a problem with encodings. install notepad ++. and change the encoding format to UTF-8

in the lower right corner you can see the current encoding. if it is not UTF- 8, you can simply change it to UTF-8 form the encoding menu tab.

this solution worked for me.

orginal post

If you are using newer versions of windows 10 you can use notepad to change the encoding from UTF-16 to UTF-8 simply by saving the file again and selecting the encoding option on the save dialog. See the example image below.