Out if interest, did you want/need to handle a number of different archive types, or do you just need to use one? If you just to choose and use one, then you could use the built-in support.
The Win32 functions Duoas refered to are part of the File Management API: LZInit, LZOpenFile, LZRead, etc.
Edit: Note that the Win32 LZ functions are provided to support the unzipping of single, compressed files (NTFS uses the LZNT1 algorithm (a variant of the LZ77) for file compression on a per file basis.) While it is no doubt possible to use LZSeek and LZRead to decompress a structured archive (i.e. a compress directory subtree of files), it would be better to use a decent zip library for this purpose. If you went with the ZL functions, you'd be on you own when it comes to handling the directory structure, etc
File Management Functions
But if you need to handle a wider range of the file types, there's the 7-Zip SDK (also alluded to)
LZMA SDK (Software Development Kit)
plus assorted other libraries (you'll probably need more than one to cover all of the popular archive file types.)
including .tar, tar.gz, tar.bz2
.xz and .lzma
|will just try and read the encoding type|
The various compression schemes use a magic number as part of their file header, so you could check that to work out what the file probably is. But there's always a slim chance that the value is there randomly, of course. And I think (older) .zip files do turn up without a magic number, just to complicate things.
any ideas how I might go about creating my own type of zip style file
If you are thinking about implementing your own type of archive, Windows does provide a mechanism that allow you treat a file like file systems in its own right: Structured Storage. This API is pretty involvd and it also COM-based (Component Object Model) API, which could be a bit of a headache if you haven't come across it before.
COM Structured Storage
(this article mentions other structured storage libraries)
Your description of what you want to do does sound a bit like the GUI tools that come with 7-Zip and WinZip??
Component Object Model
COM: Component Object Mode Technologies