The concept of binary files really "clicked" for me when I tried creating and reading .wav files. The extension doesn't matter, it just helps some OSs to know which program to open the file with by default. The extension usually refers to the file format, but it doesn't actually contribute to the content.
However, the WAVE format specification is what defines how to use this type of binary file:
http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/WAVE.html
The WAVE format is open, which means anyone can see what what bytes are placed where in the file. The file contains a header which tells us that this is a WAVE file and the size. It also contains a FORMAT chunk which defines properties such as the number of channels, the size of each sample, the bit-rate, etc. Then there is a DATA chunk which contains the wave-forms that make up the file.
Other formats such as the ms office formats (example: .xlsx files) or some game save data are proprietary. This means that the specification is not open to the public. You can try to reverse engineer the file format by viewing it in a hex editor and by making small changes to the file and seeing what effect this will have on the file. This can be difficult if the file-format is very complex.
For example: If I have a file which defines a level 1 character which looks something like this:
00 1D 74 F2 01 56
Then I level the character up and I see the new file looks like this:
00 1D 74 F2 02 56
Then I might assume that the 5th byte there represents the level. If I change that file to:
00 1D 74 F2 03 56
now I can load the character into the game and see if he leveled up.