I know a little bit about custom extensions. For one, it's really easy to name a custom extension on Windows. If you were to open notepad, type some stuff, and click Save As, you can select the filetype to all, then in the filename, type ".something" at the end, and bam: custom extension.
But here's the catch, it's not unique.
Since all you did was put text into your file, notepad, or other word editors can easily open it up, even though it was named .something, because all it contains is text.
So how to create a "unique" file extension?
That is about the extent of my knowledge. I know that, using C++, you would have to open a file for writing in binary mode using 'ios_base::out | ios_base::binary'. You would then proceed to make a unique "bit-system" within your file, essentially, you would assign bits that make your file unique. But it's not just about setting bits, it goes way into stuff that I'm not familiar with, like hashes.
So, bottom line, as far as naming an extension, just create a file with that extension.