Multiple TCP Server, one program

[Appletree610]

Hey there,
I am as new in this forum as I am in Linux network programming.
So Hello, first of all.

I reached a point of beeing confused, after 5 different books and lots of google searching. Hopefully somebody can help me.

I am having a programm with a few, let's call them peers. Each peer should work as a TCP server and reply on request from clients. I searched for some solutions and I read a lot about multithreading, multiple processes - concurrent server, which in my mind are already a starting point. But all those solutions imply just ONE server that serves MULTIPLE clients (at the same time).
Is it possible to realize my idea considering processing times and process or thread limitations and so on?
I am think of having about 20-50 server-maybe more.
Am I able to work concurrent or just sequently?
And how could I start?

Hopefully my questions are easy to understand.
Appreciate each answer!

[kooth]

Hello right back at you!

What is your goal with having peer TCP servers exactly? Are you trying to control throughput? Does each server handle the same types of requests?

I find the book "UNIX Network Programming" very informative in deciding how to create your servers and clients. The book goes through several types of servers from servers that use fork() and exec(), to servers that have a new thread per request, and so on.

If you read this book and work through the exercises, your questions will definitely be answered.

[Appletree610]

Thank you for your reply.
Yes each server handles the same typ of requests but has different properties - means different responses to the requests.

I have this book and a few others. The problem was that I read a lot about fork pthread or daemon process etc. But i think it is hard for me to decide which solution would be the most efficent for establishing a lot of servers.

[kooth]

It might be helpful to consider the way you want your application to scale, and the cost of resources needed to handle a request. My guess that a multi-threaded approach would be the best solution as far as throughput is concerned. Since you have the book, you might benefit from doing the exercises in the Client/Server Design Alternatives chapters.

I hope this helps!

[kbw]

I am having a programm with a few, let's call them peers. Each peer should work as a TCP server and reply on request from clients.

I think I know where you're going with this. Are you trying to implement micro services in an SOA environment?

Anyway, there are a few ways to do this.

1. You can run each service in its own thread. So you use those example you've been looking at and let each thread listen on its server socket.

2. You can multiplex all the sockets in a single thread. It should be obvious (from the use of the term multiplex) that this is going to be more complicated. This can naturally be extended using Asyncronous I/O for performance, but I don't think you need this (yet).

I suggest you use 1 thread per server to get going. If you experience performance problems, you can consider doing something else. That way you can get your logic working without having to deal with complex comms code.

[Appletree610]

Thank you for your reply. I am planning on something similar to what you expected.

If I am doing multithreading would it be need a lot of resources?

I started thinking of this:

If I am have a "Server Manager" with a lot of Server, can I do it as follows?

CreateSockets(ServerList);
CheckSockets(SocketList, master_set);

Within the Server Manager:
1) for loop to create all server sockets (functions: socket/setsockopt/ioctl/bind/listen)

void CreateSockets(map<int,ServerType> ServerList)
{
	fd_set        master_set;
	map<int,ServerType>::iterator it;
	map<int,int> SocketList;
	for (it= ServerList.begin();it!= ServerList.end();it++)
	{
		listen_sd = socket(AF_INET, SOCK_STREAM, 0);
		if (listen_sd < 0)
		{
			perror("socket() failed");
			exit(-1);
		}
		rc = setsockopt(listen_sd, SOL_SOCKET,  SO_REUSEADDR,
				(char *)&on, sizeof(on));
		if (rc < 0)
		{
			perror("setsockopt() failed");
			close(listen_sd);
			exit(-1);
		}
		rc = ioctl(listen_sd, FIONBIO, (char *)&on);
		if (rc < 0)
		{
			perror("ioctl() failed");
			close(listen_sd);
			exit(-1);
		}
		memset(&addr, 0, sizeof(addr));
		addr.sin_family      = AF_INET;
		addr.sin_addr.s_addr = htonl(INADDR_ANY);
		addr.sin_port        = ((*it).second->Port);
		rc = bind(listen_sd,(struct sockaddr *)&addr, sizeof(addr));
		if (rc < 0)
		{
			perror("bind() failed");
			close(listen_sd);
			exit(-1);
		}
		rc = listen(listen_sd, 32);
		if (rc < 0)
		{
			perror("listen() failed");
			close(listen_sd);
			exit(-1);
		}
		SocketList.insert(make_pair(((*it).second->Port),listen_sd));
		FD_ZERO(&master_set);
		max_sd = listen_sd;
		FD_SET(listen_sd, &master_set);
	}
}

Next:
2) Some how wait for some events within the server manager (Select with a list of socket descriptors)

 void CheckSockets(map<int,int> SocketList, fd_set master_set)
 {
 	fd_set working_set;
	do
	{
		memcpy(&working_set, &master_set, sizeof(master_set));
		ready_descriptors = select(max_sd + 1, &working_set, NULL, NULL, NULL);
	
		if (ready_descriptors >0)
		{
			desc_ready = rc;
			for (i=0; i <= max_sd  &&  desc_ready > 0; ++i)
			{
				if (FD_ISSET(i, &working_set))
				{
					desc_ready -= 1;
					if (i == listen_sd)
					{
						do
						{
							new_sd = accept(listen_sd, NULL, NULL);
							if (new_sd < 0)
							{
								//error
							}
							FD_SET(new_sd, &master_set);
							if (new_sd > max_sd)
								max_sd = new_sd;
						} while (new_sd != -1);
					}
					else
					{
						do
						{
							//Go into server and recv and send ( Input Parameter = i)
							CheckServer(i);
						} while (TRUE);
					}
				}
			}
		}
		else {endserver=true;}
	}while(endserver=true;)

}
 

3) Go into the server and process the question (recv/send)????

 void CheckServer( int sd)
 {
	rc = recv(sd, buffer, sizeof(buffer), 0);
	
	//some stuff in between
	
	rc = send(i, buffer, len, 0); 
 }
 
 

Some parts are used and changed from the IBM nonblocking IO source Code.

[LB]

If you decide to go with the non-threaded solution, liblacewing works really well with multiple servers - it uses an event pump (and isn't threadsafe).

[kbw]

You don't want to poll a whole bunch of sockets like that. It might be better if you have a server per thread. So each thread does the whole socket/bind/listen/accept sequence in parallel with the others.

I take it the each thread is providing a different service, with it's own handlers and protocol, right? So you shouldn't just poll them as many have work to do, and the amount of time handling each request will vary.