Breaking a string into tokens

[mherald81]

I need some help breaking a string into tokens. Here is what I have so far.

#include <iostream>
#include <string>
#include <fstream>

using namespace std;

int main ()
{
	ifstream infile;
	infile.open("test.txt");
	string line;
	char *str;
	char *temp;
	cout << "Splitting the line into tokens\n";
	while (!infile.eof())
	{
		getline(infile, line);
		str = new char[sizeof line];
		for (unsigned int i=0; i<= line.length(); i++)
			str[i] = line[i];
		temp = strtok (str,"\t");
		while (temp!=NULL)
		{
			cout << endl << temp;
			temp = strtok (NULL, "\t");
		}
		cout << endl;
	}
	cout << endl << endl;
	return 0;

	delete [] str;
} 

Here is a sample input that I have:(each char is separated by a \t)

1 1 0 0 A

Now my code breaks it up into tokens when it looks like, but what I want it to do is to break it up into and recognize blank, or empty. Something like this:

1 1 A

My code will print this for the first sample:
1
1
0
0
A

And this for the second sample:
1
1
A

What I want it to do is to print 0s for any blank/empty positions:

1
1
0
0
A

[helios]

I think it'd be easier to just check each character and compare it to '\t' or ' ' or whatever.

Also, did you know that you can access the internal string with std::string::c_str()?

[Duthomhas]

I agree. Please don't use strtok() in C++ programs. (The function is evil to begin with.)

There was a recent Article on it:
http://www.cplusplus.com/forum/articles/4860/
(Be aware that my example in this article does not account for empty fields --the same as strtok().)

You can also use getline() and a stringstream:
(This example does account for empty fields.)

#include <iostream>
#include <sstream>
#include <string>
using namespace std;

string trim( const string& s )
  {
  string result( s );
  result.erase( result.find_last_not_of( " " ) + 1 );
  result.erase( 0, result.find_first_not_of( " " ) );
  return result;
  }

int main()
  {
  string list_of_words;

  cout << "Please enter a list of words separated by commas\n> " << flush;
  getline( cin, list_of_words );

  list_of_words += " ";
  istringstream iss( list_of_words );
  string        word;
  int           cntr = 0;
  while (getline( iss, word, ',' ))
    {
    cout << "word " << ++cntr << ": " << trim( word ) << '\n';
    }

  cout << "\nGood bye!\n";
  return 0;
  }

Hope this helps.

[mherald81]

Thanks for the help and advice Duoas! I will give it a try using getline and a stringstream.

[anders43]

mherald81 you got a mem leak in your code and other bugs e.g.

str = new char[sizeof line]

Don't use sizeof for this, you use line.length() - as you do later when you traverse it causing memory corruption.

You don't delete the str in each iteration, so every time a new str is allocated. At the end of the file you delete the last allocated str.

I would suggest reading char by char instead, since your input is very straightforward.