Coding for background tasks/child processes

[mathewfer]

Hi C++/Unix/Linux guys,

I have a question on getting some part of the code in background (not sure correct word).

I am not a very experienced C/C++ coder

I have the below code which worked well but with a list of about 1000 entries in file DeviceList, UNIX (SunSolaris) server get really CPU intensive & not very responsive to prompt.

Code does below steps;

Binary/program is run with the parameter email address.

1st while loop read the file DeviceList & each line is then build with parameters to run with the same compiled program as the main one (one program)
The new command is then called via system with "nohup" alone with "-f" followed by "&" for background. This "-f" option code in turn calls a function - "main_comn(Device) which does some complex analysis on log files" Please note that there is ONE complied binary program - used differently depending on command line parameters.

2nd while loop wait till/loop wait till 1st file with "Devicename+Resultlog.log" is generated.

3rd while loop wait till number of "Resultlog.log" (that is generated per device - each line in file DeviceList)

Now all these code works as expected when run with few number of entries (say 50) in file but when the entries are about 200-300, UNIX box CPU utilisation becomes closer to 100% due to many processes created with "nohup "+tool_name+" -f "+Devicename+" 2>/dev/null 1>/dev/null &".

is there a better way to get a multiple child run at the same time to speed up checking all log files in parallel & improved method for main code to wait till all child process are done without utilizing CPU cycles?

I head about fork() & not used before.

Any suggestion/links to improve this code & learn more on child processes etc. are much appriciated.

#include <fstream>
#include <iostream>
#include <ctime>
#include <cstdlib>
#include <cstdio>
#include <string>
#include <sys/stat.h>
#include <sys/types.h>
#include <sstream>
#include <time.h>
#include <dirent.h>
#include <ctype.h>
#include <vector>
#include <unistd.h>

using namespace std;

int main(int argc, char** argv) {

    int nohupCounter=1;
    
    if (argc==2)
     {
       if (validateEmailAdd(argv[1]))
        {
            if (FileExistCheck("DeviceList"))
            {
                string DeviceList1="DeviceList";
                string Devicename;
                ifstream fileToSearch12;
                fileToSearch12.open (DeviceList1.c_str());

                 while(getline(fileToSearch12, Devicename))
                  {
                     if (nohupCounter<=nohupCounterMAX)
                     {
                         string run_Device_test_bg="nohup "+tool_name+" -f "+Devicename+" 2>/dev/null 1>/dev/null &";
                         system(run_Device_test_bg.c_str());
                         nohupCounter++;
                     }
                     else
                     {
                         string run_Device_test_bg="nohup "+tool_name+" -f "+Devicename+" 2>/dev/null 1>/dev/null &";
                         system(run_Device_test_bg.c_str());
                         nohupCounter=1;
                         sleep(30);
                     }
                  }
                
                NoDevices=noLineinAFile("DeviceList");
                int NoFilesChked=0;
                
                cout <<" "<<endl;
                cout <<" ---- In progress"<<endl;
                cout <<" "<<endl;
                
                FileList();
                
                while(!searchAStringinFile("File_list", "Resultlog.log")==1)
                {
                    sleep(1);
                    remove("File_list");
                    FileList();                   
                }
                
                 while(NoDevices!=NoFilesChked)
                  {
                    string CountNoDeviceFiles="ls *.log|wc -l > NuFilesChecked";
                    system(CountNoDeviceFiles.c_str());
                    
                    string wcNumber;
                    ifstream fileToCheck1;
                    fileToCheck1.open("NuFilesChecked");
                    getline(fileToCheck1,wcNumber);
                    
                    istringstream buffer(wcNumber);
                    buffer >> NoFilesChked;
                    
                    cout<<" --- Checked "<<NoFilesChked<<" Devices so far --- ";
                    cout << '\xd';
                   
                  }
             
            }
            else
            {
                cout <<" "<<endl;
                cout <<"Device list file is not generated, internal error & existing the tool"<<endl;
                cout <<" "<<endl;
                return 0;
            }

            cout<< " Done" << endl;
            
        }
     }
    else if (argc==3 && (strcmp(argv[1],"-f")==0 || strcmp(argv[1],"-f")==0))
     {
        string Device=argv[2];                              
              
        // Call main module
        main_comn(Device);
     }
    else
     {
        // Display usage
        usage(argv[0]);
     }

    return 0;
    // end of main function

}

Regards,

Mathew

[kbw]

That's crazy stuff. I have no idea where it's spending it's time.

You need a sleep in loop:

                 while(NoDevices!=NoFilesChked)
                  {

Maybe that'll take some unnecessary load off the file system and CPU.

[mathewfer]

Hi Kbw,

Thanks for replying.

Question on sleep(x) - does it consume CPU cycles? If not, it will help but are their any other disadvantages using sleep(x)?

Any help is much appreciated to understand this.

Mathew

[kbw]

sleep() uses no CPU, the thread doesn't get scheduled.

You use sleep() in the previous loop. You could increase the sleep times to 5 seconds, there's no need to check every second.

If you start the programs 10 at a time, 2 seconds apart, that would release the load on the file system too. But it's impossible to tell what these exact values should be, it depends on how long it takes for each of these child apps to run. My idea is not to spawn too many processes at a time, you want to let a few complete while your starting others.

[mathewfer]

Hi Kbw,


Thanks. I have started reading about "pthread" in my UNIX box in "pthread.h". I think after reading & learning it, I will be able to use it in my code to improve it.

Do you think this is better than using sleep()?

Mathew

[kbw]

No. don't use threads. Your app is slow because of too much concurrency. Threads increase concurrency.

You start some app, tool_name, to do stuff. Clearly it's quicker to run a few together than run one at a time. But it's also clear that you can't run hundreds together as the system become unresponsive. So there's some optimum number that can run together, and any further instances should be queued and run when one of the others completes.

The whole thing slows down because of some combination of CPU load and file system load. There comes a point when the system simply cannot do more and more without slowing down running processes.

Back to your app. You already have a throttle, nohupCounterMAX. But instead of waiting when you hit that limit, you plough on by resetting the counter, which is pointless. I guess once you fix that and set nohupCounterMAX to a suitable level, you'll be able to manage that threshold of instances.

[mathewfer]

Hi Kbw,

Thanks, yes, I too felt that threads will give more concurrency, after reading about it & it's purpose does not seems to suit my requirement.

Yes, I am going to play around with nohubCountMax counter value to be about 10-20 & sleep value.
I will report back my findings.

Mathew

[mathewfer]

Hi Kbw,

Good news!!, got it to work.

same code above with nohupCounterMAX=5 with 10 sec interval (seep(5)) did the trick. Now CPU utilisation does not hit peak more than 30% & I could finish checking (in module - main_comn(Device)) 100 devices in 3min.

With this, I achieved my objective.

Once again, thanks for all your replies.

Mathew