Fstream Related Doubt

[VishnuBK]

If I am running a program to write some information to a text file and I open the output file during the execution of the program, does it affect the output? As in will it continue to write in the same file? My program takes at least 15 hours to execute so please answer if your very sure.

[dhayden]

I'm not certain what you mean by "will it continue to write in the same file" but I'll point out that when opening a file for writing, you can open it to append to the end of the file or to overwrite the file.

My program takes at least 15 hours to execute

Modern computers are insanely fast. There's a good chance that your algorithm can be improved substantially. You might want to post a description of the problem and how you're solving it to see if anyone has ideas of a better way.

[Chervil]

It should be possible to open the file to read its contents while the program is still running. It's common practice to view the log file for example, generated by a long-running process.

The only minor issue is that because of buffering (depending on how the program was written) the file being viewed may not fully show all of the contents until the program completes and closes the file.

[Rabindra Hota]

What do you mean by opening the file during execution of the program;
If it is a manual opening of the log file, it will not effect the writing process.
If you fear to do s, copy the file to a new location and read it manually.

If you want to open it from code, best is use some lock mechanism in reading and writing process so one should wait for the other. anyway this kind simultaneous log reading and writing is a bad design.
Where such logging are helpful :-You need to send the ongoing status of your log to another process, where the process cannot enter your main program but only input is the log file and cannot wait till your main program ends and wants to read the log.

[VishnuBK]

@dhayden

The problem is as follows:
It's an astrophysics related project. There are two text files, One has 18,94,216 rows and 6 columns. The other has 59031 rows and 6 columns. I need to compare the value of 3 columns(in one row) of one file to 3 columns(in one row) of the other file. This has to be done line by line as in the values of 3 columns of the first row of the first file should be checked with every row of the other file.(I presume that 59031*18,94,216 operations it needs to do)
If they match, write values to a new text file.

I am using nested for loops(calling one 'i' and other 'j') The main part of the program has been copied here.


for(int i=1;i<=18,94,216;i++)
{
infile1>>plate>>mjd>>fibre>>emission_line>>linesigma>>sigma_error;

for(int j=1;j<=59031;j++)
{
infile2>>ra>>dec>>z>>plate2>>mjd2>>fibre2;

if(plate==plate2&&mjd==mjd2&&fibre==fibre2)
{
outfile<<ra<<" "<<dec<<" "<<z<<" "<<plate<<" "<<mjd<<" "<<fibre<<" "<<emission_line<<" "<<linesigma<<" "<<sigma_error<<endl;

count++;
i++;
}
}
infile2.seekg(0,ios::beg);
}

[dhayden]

Oh this is just a database join. It's actually quite easy to do efficiently. If you're running on Linux or MacOS, look at the sort and join command line utilities. If running on windows, see if you can get these from cygwin or something similar.

To do this efficiently, you sort the files by joined fields (plate, mjd and fibre). Then walk through each file comparing the joined fields. Since the files are sorted, you never have to back up in either file. The actual code can be a little tricky, but that's it in a nutshell. The UNIX sort command will sort your files and then the "join" command will join lines that match on specific fields.

[VishnuBK]

I wanted to this in plain C++. I sorted both tables in the following order(plate,mjd,fibre) and (plate2,mjd2,fibre2). This is the code I used.

#include <iostream>
#include<string>
#include<fstream>

using namespace std;

int main()
{
int plate,mjd,fibre,plate1,mjd1,fibre1,plate2,mjd2,fibre2,count=0;
string emission_line,emission_line1;
double linesigma,sigma_error,ra,dec,linesigma1,sigma_error1,z;
ifstream infile1,infile2;
ofstream outfile;
infile1.open("output.txt");
infile2.open("DR10sorted.txt");
outfile.open("Final Catalog_without_error4.txt");
infile1>>plate>>mjd>>fibre>>emission_line>>linesigma>>sigma_error>>plate1>>mjd1>>fibre1>>emission_line1>>linesigma1>>sigma_error1;

while( infile2>>ra>>dec>>z>>plate2>>mjd2>>fibre2)
{
while((plate==plate2&&fibre<fibre2)|| (plate<plate2))
{
infile1>>plate>>mjd>>fibre>>emission_line>>linesigma>>sigma_error>>plate1>>mjd1>>fibre1>>emission_line1>>linesigma1
>>sigma_error1;
}
while(plate==plate2&&mjd==mjd2&&fibre==fibre2)
{
outfile<<ra<<" "<<dec<<" "<<z<<" "<<plate<<" "<<mjd<<" "<<fibre<<" "<<emission_line<<" "<<linesigma<<" "<<sigma_error<<endl
<<ra<<" "<<dec<<" "<<z<<" "<<plate1<<" "<<mjd1<<" "<<fibre1<<" "<<emission_line1<<" "<<linesigma1<<" "<<sigma_error1<<endl;// Write to file
count++;
infile1>>plate>>mjd>>fibre>>emission_line>>linesigma>>sigma_error>>plate1>>mjd1>>fibre1>>emission_line1>>linesigma1
>>sigma_error1;


}

}



cout<<count<<endl;
infile1.close();
infile2.close();
outfile.close();
return 0;
}

It mostly works file but few objects that has to be matched is missing. Any idea where I went wrong?

[dhayden]

The code as written will match one line in DR10sorted.txt with each matching line in output.txt. But if there are multiple lines in DR10sorted.txt with the same plate, mjd and fiber values, they won't match.

You might try reducing the files to the lines that are supposed to match but don't. Then see what's happening in the program.

[dhayden]

What if there are 2 lines in output.txt and 2 lines in DR10sorted.txt that all have the same plate, mjd, and fibre values? You won't print out the 4 possible matches.

If that isn't the problem then try reducing the two input files to just the lines that area causing problems and then look at what's happening with a debugger. If you don't have a debugger then add some temporary code to output the value of each record as it's read. This should help you find the problem.

Let us know what the runtime is when it's working!

[VishnuBK]

Every thing works fine. I feel stupid putting up this question now. It was actually a sorting error. I sorted it in two parts in MS excel before as it did not fit in one sheet. This created a problem. I used sort command in cygwin and everything is fine. My aim was only to match one line from DR10sorted.txt to one line in output.txt. So everything is working perfectly fine now.

And the runtime came down from 19 hours to 77 seconds. :P Thank you for help. :)

Any ideas on improving the algorithm, just for learning sake?