One problem that jumps out immediately:
1 2

vx=rotateX(vx, vy, STEER); // < changing vx here
vy=rotateY(vx, vy, STEER); // < using NEW vx value here. Not intentional?
 
This might cause wonkiness. Also if your angles are outside of [0,360) these calculations simply won't work. So unless you are wrapping to make sure you're always within those bounds (I don't see where you are in this routine) this approach will just fail.
For example... an angle of 90 degrees is the same as 270, so you would need to decrease the angle.... but since 90 is less than 180 this code would be increasing it.
==========================================
The good news is there is a much simpler approach to this problem in Linear Algebra. It's the dot product (and perpendicular dot product).
Get used to these functions. Once you understand them, you'll find 1001 uses for them. They really are extremely versatile:
1 2

inline float dot(const Vector& a, const Vector& b) { return (a.x*b.x) + (a.y*b.y); }
inline float perpDot(const Vector& a, const Vector& b) { return (a.y*b.x)  (a.x*b.y); }
 
These functions are magic. They have the below properties:
1 2 3 4

dot(A,B) == cos( theta ) * length( A ) * length( B )
perpDot(A,B) == sin( theta ) * length( A ) * length( B )
// where 'theta' is the angle between vectors A,B
 
Since I don't know whether or not you really care, I'll spare you the details of what this means, and I'll just tell you that this can be applied as follows:
 let vector A be the movement vector of the enemy (the direction he's currently facing)
 let vector B be the vector between the player and the enemy
 dot(A,B) will be positive if the enemy is moving closer to the enemy, and negative if moving away.
 perpDot(A,B) will be positive if the player is to the right of the enemy, and negative if the player is to the left (** note I might have this backwards... positive might mean to the left.... I can never remember which is which... try it and see! **)
 perpDot(A,B) will be zero if A and B are parallel (ie: enemy is moving directly towards the player, or directly away from them).
For your purposes, you can use perpDot to determine if you need to steer left or right. If perpDot is zero (parallel), you can use dot() to determine whether or not you are facing the right way:
1 2 3 4 5 6 7 8 9 10 11

Vector shipDirection = /*the direction the ship is facing*/;
Vector targetDirection = targetPosition  shipPosition;
float pd = perpDot( shipDirection, targetDirection );
if( pd < 0 )
turnRight(); // or left if I have it backwards
else if( pd > 0 )
turnLeft(); // or right if I have it backwards
else if( dot(shipDirection, targetDirection) < 0 )
turnRight(); // need to pull a 180, pick a direction and go for it
//else we have no need to turn because we're heading right for them
 
EDIT: If you are interested in understanding more of why/how this works, I can get into more detail. Just let me know. I just didn't want to waste the time on it if you weren't interested.