AI is an extremely broad and complex topic.... and "a RTS game" is an extremely generic description of what you're trying to do.
Best advice I can give you is try to write code which reflects human logic. IE: examine all visible units and move your units where they'd be most effective.
The first year program of the school I'll attend lists the following under AI
- Finite states machines
- Fuzzy logic
- Pathfinding with A*
- Chase and evade
So far I've only studied pathfinding and it's not that hard, so I'd recommend that to begin since it's something you definetly need in an RTS. I used http://www.policyalmanac.org/games/aStarTutorial.htm and the wikipedia page for A*.