having more than 1024 fds in a FD_SET?

[LuckyIsDog]

How can I accomplish this? FD_SETSIZE is 1024, so that means I cannot have more than 1024 file descriptors in an FD_SET. I am building a server that needs to handle more than that.

I am thinking about an FD_SET vector to accomodate all the connections I will be handling.

What troubles me is, how do I do select() on all of the FD_SETs if it is a blocking function?

[Cubbi]

For handling many connections, you may wish to read http://www.kegel.com/c10k.html old as it is. Specifically, all I/O multiplexing methods other than select() don't have this limitation, but they have other limits that come up when you reach 10k clients.

[rapidcoder]

Java webservers can handle easily 500k concurrent clients, so there is no real reason a C server couldn't. You can set a different limit for FDS using ulimit.

[LuckyIsDog]

Could you provide an example? It would be really helpful, thank you!

[LuckyIsDog]

Here's me theorizing a bit, but would this work?:

void ConnectionThread (void * p){

fd_set server_socket_set;
vector<fd_set> client_sockets_set;
vector<int> client_sockets_holder;
signed long client_sockets_set_size = client_sockets_set.size();
timeval nextset;
nextset.tv_usec = 300; //0.3 second timeout till next set is checked!

for(;;){

activity = select( UNKNOWN, &server_socket_set, NULL, NULL, nextset);

if ((activity < 0) && errno != EINTR){
    cout << "Could not select!" << endl;
    exit(EXIT_FAILURE);
}
if (FD_ISSET(master_socket, &server_socket_set)){
    // Handle incoming connection
}



for (int i=0; i<client_sockets_set_size; i++){
    activity = select( UNKNOWN, &client_sockets_set[i], NULL, NULL, nextset);
}


for(int i=0; i<clients_sockets_holder; i++){
    int clientNum = i;
    for (i=0; i<client_sockets_set_size; i++){
        if (FD_ISSET(client_sockets_holder[clientNum], &client_socket_set[i])){
             // HANDLE INCOMING READ or EOF 
        }

    }

}



}

So first off, when we hit the 1024 limit for client_sockets_set[0]. Then we move onto the next set until client_sockets_set[1] is full too then move on..

Then next we have a 0.3 second timeout for select to go through every fd_set in client_sockets_set. Maybe even have a smaller timeout for less latency (I don't know if this would be safe to)

I will test it out but would love an experienced programmer to give advice on it!

[Cubbi]

Experienced programmers don't use select() to monitor large sets of file descriptors! That's what OS-specific APIs (epoll. dev/poll, kqueue, pollset, I/O completion ports) are for, and the libraries that encapsulate them (e.g. libevent or boost.asio, where suitable)