Troubleshooting PuTTYgen "Couldn't load private key (not a recognized key file format)" Error

Background

I’ve been using PuTTY to connect to my EC2 instances from my Windows computer. I always choose to download the SSH key file from AWS console directly. The downloaded key file is a PEM file, and it has human readable format like this:

-----BEGIN RSA PRIVATE KEY-----
...some PRIVATE data...
-----END RSA PRIVATE KEY-----

PuTTY cannot consume this key file as-is. As suggested in AWS Docs, PuTTYgen is the preferred tool for converting PEM files to PPK(PuTTY Private Key) files.

It always worked well and I had no problems with using PuTTYgen. But today I accidently deleted my PPK file and had to regenerate it. Fortunately I backed up the key file on Dropbox, and was able to send the content of the key file from my phone to my computer, via messenger. I opened Notepad, copy-pasted the content and then saved it as key.pem as I always did. And tried to load it from PuTTYgen, then I saw this error:

Troubleshooting

I thought it might be the problem with the file extension at first, so I changed it to .pem, .ppk, .key, etc. but it didn’t help. I opened it with Notepad again and checked if I made a mistake copy-pasting it, and it was not the reason too. I tried to add a newline at the file’s end but still PuTTYgen complains about the file format.

Then, I suddenly realised that I selected UTF-8 encoding when saving the key file. I always do that because if I save a file with ANSI encoding, the default option, Korean letters do not show up correctly in other editors - Notepad encodes CJK letters with CP949 while other editors expects UTF-8 as the default encoding.

I changed the file encoding to ANSI and PuTTYgen worked happily.

Actual reason

After that, I wondered why the file encoding matters even without CJK letters. I opened Notepad again, put this content and saved it as hello.txt with UTF-8 encoding:

hello
world

Then I opened a Python shell in the same directory and ran this script:

with open("hello.txt", "rb") as f:
    print(f.read())

It printed out following result:

b'\xef\xbb\xbfhello\r\nworld'

Yes, that was the reason, the UTF-8 BOM(Byte Order Mask).

Conclusion

So today I learned that Notepad inserts BOM when saving files with UTF-8 encoding. This is why one shouldn’t use Notepad!