I was assigned a problem involving using buffer overflow to access a different function than I was supposed to. I was able to figure out how to modify the point in stack that I need to change using a printf statement, however what doesn't make sense is that when I use the input "AABBCCDDEEFFGG\x86\x64\x00\x00" the stack changes to 78363878 I looked up the ascii codes and I assume that it's not converting \x86, but using the input x86 instead. I don't see why this is wrong, any help I could get would be appreciated.
If you are just typing in the string as written, then the backslash is treated as a literal backslash. You will want to input whatever character corresponds to the hex values you want (perhaps by using some file as input).