Lumpkin wrote: 

Finding interest rates and working with doubles isn't very hard. 
True, but finance is not all about rate of return, which is indeed just arithmetic (and exponentiation), even with all the complexities of the calendar conventions, quoting and compounding rules. Even fairly common numbers such as derivatives pricing or risk calculations require advanced statistics and calculus, and specialized mathematical models used in quantitative finance can get to PhD level math.
Lumpkin wrote: 

High performance? Why would it need to be that? 
Even the basic calculations above have to be lightning fast to make today's economy possible, but where finance goes to the extreme is arbitrage: suppose you've learned that someone wants to sell 500 contracts of frozen pork bellies for Nebraska delivery* in Chicago at .31500 USD per pound, while someone else wants to buy 100 of the same contracts in Kansas City at .31625 USD/lb. As any sensible person who has the means to do so, you're going to simultaneously sell 100 in Kansas City at .31625 and buy 100 in Chicago at .31500, making 5,000 USD instant profit.
Now suppose the office across the street saw that opportunity too, and they entered their orders
a microsecond before yours (orders are executed in the order received). Now they have 5,000 dollars, while you have 4 million pounds of pork bellies that will be waiting for you in Nebraska next month and you've tied up 1,260,000 USD cash that could be used for more trading.
[*] pork bellies were delisted in 2011, I just like to use them in examples
To get back on topic
aimforthehead wrote: 

I have spent so much time working on console applications that are about payroll and things like that, when I think I'm more interested in making games and whatnot
[...]
I guess I'm just looking for programmers to tell me it gets funner if you just stick with it

It's normal to see people change jobs every 5 years or so. If you get a job making games and, years later, get bored doing the same thing every day, you might switch to OS development, or programming robots in car factory, or making cell phone software, or, sure, finance, if you like the stress.