I want to be able to run a program that assigns and edits users rights of a program. Its called the UAM. The UAM can run without going into the program that its part of so its kind of like its own application.
As you already did say, the client must create a UDP connection to the server. Obviously, the server must listen to specific UDP port so that the connection can form. Furthermore, both programs must agree on what data packages do flow over the connection.
Use a library/framework that provides the basic networking bits. Do not forget authentication and access control, for someone might try to simply push garbage into your service with telnet.
could you explain how it will actually work?
so lets say i create some library... how does this library allow me to access the UAM from a client computer? will the UAM dialog box actually appear on the client computer?
You have to. Right now if all you have is the code from that link then it doesn't do anything but register an entry in the Service Control Manager and respond to a few, not even all, of the messages the OS could send it.
You wouldn't create a library for this (you could but you wouldn't) you would use one of the dozen or so that are already available including Winsock2 which you already have.
As for communicating with the client, your service would be running a second thread that just sits and listens on whatever port for a connection attempt. The reason for the second thread is that the program will wait on the "Listen()" function until a connection attempt is made and services have to be able to respond to messages from the OS or they get slaughtered. I wouldn't worry about authentication yet, although keskiverto is absolutely correct about the telnet concern as it is this service runs under the local service account so that should be fine for most things you are doing. After a connection is made your applications are free to send text to each other just like any other sockets program using the send() and recv() functions. You could have a function in the service application that takes the text sent from the server and evaluates it to determine which command it should perform.
Having a dialog box actually appear and interface with the user involves IPC (Inner Process Communication) and is another issue entirely. We can try to address that after we have this first part done.