extracting data from file

closed account (NCRLwA7f)

hey guys, I have a few data files that I would like to sort data out of, The originals got damaged and the only copy I have contain the data I need but it also contains symbols and other punctuation marks mixed in.
I tried using inFile >> and setting up conditions to copy every char to a new text file excluding the symbols I need. The only problem is that I'm not pulling in the blank spaces so now I have one continuous sentence.
Would I need to use getline? or can I check every character at a time with a different method?

  //example text sample
//from:
thisÚËﬁ‘ is a sample`VLB8.$
¸ÚËﬁ‘ ¿µ™üîâ~sh]RG<1& sentence98@
//to:
this is a sample sentence

Handy Andy (5051)

Hello DaRealFonz,

As a thought you could use "std::getline()" to read each line of the file and use a for loop to check each character with the "isalpha" and "isspace" functions. When you find a match add this character to a new string.

For something fancy you could set up something to add the necessary punctuation at the end.

Looks like some afternoon fun. I will see what I can come up with. If you could give me more than a one line example of what the file actually looks like that would help.

Hope that helps,

Andy

Edit: Forgot to mention "isalpha" and "isspace" is in the header file <cctype>.

Last edited on

tpb (1495)

Yes, use getline(fin, line). Then scan the line and only output what you want (including the spaces). Don't forget to output the newline at the end of the line.

tpb (1495)

This horrible hack of mine seems to work on your sample.
It removes all Unicode multi-byte characters.
It removes all characters with a '\x1e' or '\x1f' character after it.
It removes all characters that are not alphabetic or whitespace.

#include <iostream>
#include <iomanip>
#include <sstream>
using namespace std;

int nbytes(char ch) {
    int n = (unsigned char)ch;
    if      (n >= 240) return  4;
    else if (n >= 224) return  3;
    else if (n >= 192) return  2;
    else if (n >= 128) return -1;
    return 1;
}

int main() {

    istringstream sin(
    "thisÚËﬁ‘ is a sample`VLB8.$"
    "¸ÚËﬁ‘ ¿µ™üîâ~sh]RG<1& sentence98@"
    );

    char c, prev;
    sin.get(prev);
    while (sin.get(c)) {
        int n = nbytes(c);
        if (n != 1) {
            for (int i = 1; i < n; i++)
                sin.get(c);
            sin.get(c);
            if (c != '\xe')
                sin.unget();
        }
        else if (c != '\xe' && c != '\xf') {
            if (isspace(prev) || isalpha(prev))
                cout << prev;
            prev = c;
        }
        else {
            sin.get(prev);
        }
    }

//    while (sin.get(c))
//        cout << setw(3) << (int)(unsigned char)c << ' '
//             << setw(2) << nbytes(c) << ' ' << c << '\n';;
    
    cout << '\n';
}

Handy Andy (5051)

Hello DaRealFonz,

I tried tpb's program and it did not work for me. Sorry tpb.

I did come up with this that works to a point:

#include <iostream>
#include <iomanip>  //  setw(), fixed, setprecision.
#include <string>
#include <fstream>
#include <cctype>
#include <chrono>  // For sleep time code
#include <thread>  // For sleep time code

int main()
{
	wchar_t c{};

	std::string line, newLine;
	std::string iFileName{ "Input Data.txt" };

	std::ifstream inFile;

	inFile.open(iFileName);

	if (inFile.is_open())
	{
		std::cout << "\n File " << iFileName << " is open" << std::endl;
		std::this_thread::sleep_for(std::chrono::seconds(0));  // <--- Needs header files chrono" and "thread".
	}
	else
	{
		std::cout << "\n File " << iFileName << " did not open" << std::endl;
		std::this_thread::sleep_for(std::chrono::seconds(3));  // <--- Needs header files chrono" and "thread".
		exit(1);
	}

	while (std::getline(inFile, line))
	{
		std::cout << "\n" << line << std::endl;

		for (size_t lc = 0; lc < line.size(); lc++)
		{
			c = line[lc];

			if (c > 128) continue;

			if (std::isalpha(c))
				newLine += line[lc];

			if (std::isspace(c))
				newLine += line[lc];
		}
	}

	std::cout << "\n " << newLine << std::endl;

	std::cout << "\n\n Press Enter to continue";
	std::cin.get();

	return 0;
}

This works except for the fact that it picks out anything that is a letter even if you wish it not to be in the final output.

Something you can start with.

Hope that helps,

Andy

closed account (NCRLwA7f)

Ok, Just wondering if there was something already in the standard temp library. what tpd put up looks interesting, I will give it a try. I tried what you put Andy and it is a good start, it did however stop working midway....but I think its bc the file is way to big. regardless that gave me an idea might have to give it a couple passes to clean it up.

I'm thinking i'm just going to ignore every char that is not a space a-z,A-Z, and a period for now.

lastchance (6980)

#include <iostream>
#include <fstream>
#include <sstream>
#include <string>
#include <cctype>
#include <algorithm>
#include <iterator>
using namespace std;


bool filter( char c )
{
   const string allowed = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ .\n";    // choose what to keep

   bool ok = allowed.find( c ) != string::npos;
   return ok;
}


int main()
{
// ifstream in( "input.txt" )
// ofstream out( "output.txt" )
   stringstream in( "thisÚËﬁ‘ is a sample`VLB8.$¸ÚËﬁ‘ ¿µ™üîâ~sh]RG<1&sentence98@" );

   istream_iterator<char> it{ in >> noskipws };
   ostream_iterator<char> ot{ cout };
   copy_if( it, {}, ot, filter );
}

tpb (1495)

@Handy, In what sense did it not work?
I just copied the code from the post above, pasted it in an editor, saved it, compiled it, ran it and it printed:
this is a sample sentence

tpb (1495)

Try running this on your file. Set the input and output file names, of course.

#include <iostream>
#include <iomanip>
#include <fstream>
#include <cctype>
using namespace std;

int nbytes(char ch) {
    int n = (unsigned char)ch;
    if      (n >= 240) return  4;
    else if (n >= 224) return  3;
    else if (n >= 192) return  2;
    else if (n >= 128) return -1;
    return 1;
}

int main() {
    ifstream fin("inputfile");
    ofstream fout("outputfile");

    char c, prev;
    fin.get(prev);
    while (fin.get(c)) {
        int n = nbytes(c);
        if (n != 1) {
            for (int i = 1; i < n; i++)
                fin.get(c);
            fin.get(c);
            if (c != '\xe')
                fin.unget();
        }
        else if (c != '\xe' && c != '\xf') {
            if (isspace(prev) || isalpha(prev))
                fout << prev;
            prev = c;
        }
        else {
            fin.get(prev);
        }
    }
    fout << prev;
}

closed account (NCRLwA7f)

whooo ... hey I tried your code tpb, this is probably the best I have output the file, I like how it takes the unicode out yet leaves a white blank where the unicode was, this really improves redability.

lastchance you code was kind of what I was doing...but yours is shorter and practical.

I think I can get what I need from the files with what you guys have provided. Now I just have to sort through about 6 files with 50MB of text each :)

dhayden (5795)

If you're on a linux system, consider using sed:
sed -e '1,$s/[^a-zA-Z]+/ /g' < inputFile > outputFile

tpb (1495)

My analysis was based on the output of this program:

#include <iostream>
#include <iomanip>
#include <sstream>
using namespace std;

int nbytes(char ch) {
    int n = (unsigned char)ch;
    if      (n >= 240) return  4;
    else if (n >= 224) return  3;
    else if (n >= 192) return  2;
    else if (n >= 128) return -1;
    return 1;
}

int main() {
    istringstream sin(
        "thisÚËﬁ‘ is a sample`VLB8.$"
        "¸ÚËﬁ‘ ¿µ™üîâ~sh]R"
        "G<1& sentence98@"
    );

    char c;
    while (sin.get(c))
        cout << setw(3) << (int)(unsigned char)c << ' '
             << setw(2) << nbytes(c) << ' ' << c << '\n';
}

The output follows. As you can see, there are definitely unicode multi-byte characters and also a bunch of 14's and 15's that need to be removed. And if a 14 or 15 appears, the character before it needs to go, too, even if it's a normal alphabetic character.

Topic archived. No new replies allowed.

C++

Forum

extracting data from file