Best to just not include it in the first place if you do not require it. And it depends on what the header contains but you can expect a bigger file. If a dynamic link library is included it will slow down the program.
#including a header will not slow down your program or enlarge it.
It will, however, increase the compilation time as there is more code in the form of declarations/definitions to process. There may also be some additional time spent in the preprocessing phase of the compilation process.