getline max size = 256? how to bypass this?

[daven25]

Hi, I have a .csv file that I'm trying to read from. Each line has approximately 400-600 characters (sometimes more). When I'm trying to use the fstream to read the line and put it into an object that I've created, it only takes the first 300 characters or so. I've checked the csv file and there isn't any '\n' (newline) escape characters that would make it stop there.

I did some research and it appears there might be a limit on getline but I wasn't able to find a solution that worked for me? Is that true and if so, how do I get it to input the entire line? The weird thing is, it reads the first 40 lines just fine and some of them are longer. Thank you for your help. Here is my code:

#include <iostream>
using namespace std;
#include <string>
#include <fstream>
#include <vector>
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

int main()
{
     importZoho();
}

void importZoho()
{
	ifstream fin ("zohotest.csv");  // Fstream operator
	string input;                   // The string to put it in
	int noOfZoho = 0;               // Counter to keep track of how many objects I've created
	vector<Zoho> zoho(15000);  // A vector of objects I've created
	
	while(getline(fin, input))      // While there is another line, put it into input
	{
		zoho[noOfZoho].setLine(input);  // Sets the line in my object
		zoho[noOfZoho].setPartNumber(getZohoPN(input));   // Sets the partnumber in the object using a non member function "getZohoPN" that I've created
		zoho[noOfZoho].setPrice(getZohoPrice(input));    // Sets the price in the object using a non member function "getZohoprice" that I've created
                ++noOfZoho;
	}
}

Here is a sample of just one line from my csv where it's getting stuck and only taking the first 300 characters or so:

zcrm_438903000000066507,zcrm_438903000000048003,Wiley Electronics WEEB 11.5 Grounding Clip,590-0038 WEEB-11.5,zcrm_438903000000066367,TRUE,Wiley Electronics,,,,,,zcrm_438903000000048003,zcrm_438903000000078304,40788.5375,40975.61458,,USD,1,1.99,0,Sales Tax,TRUE,Each,2900,-4400,0,,-4400,"Wiley Electronics, WEEB Grounding clip for DPW PowerTube CRS Rail Brackets,
WEEB-11.5",No,,WEEB-11.5,590-0038,1.99

[daven25]

I copied and pasted the line and I guess it did it again. If you look at the sample line that I pasted, that's where it cuts off the string.

"WEEB-11.5",No,,WEEB-11.5,590-0038,1.99" is suppose to be on the same line as where it ends. I don't get it. I'm using notepad and I don't see any newline characters. When I use excel to save this, should I be saving in Unicode perhaps? Thanks for any help you provide.

[cire]

Notepad isn't exactly definitive. Using a hex editor to inspect the file might be more informative.

[o1vk4iN6]

^

Notepad only uses CRLF line endings while getline only needs LF which notepad would not see as a new line.

[daven25]

Thanks cire, I'm looking into hex editors right now. But it would suck if I have to use a hex editor everytime to preformat the csv file. I'm building the program so that others can use it to add new products to the data base.

>>xerzi... so what does that mean?

[o1vk4iN6]

It means notepad uses window's line ending of "\r\n" instead of just "\n". You can just use word pad or something better than notepad. If you need to know specific binary information a hex editor is preferred.

[cire]

I wouldn't suggest a hex editor to fix your formatting problems, but it is a reliable way to diagnose the problem and a generally useful thing to have around.

Windows systems use a combination of newline and linefeed to indicate a linebreak in text files. Depending on your compiler, a lone linefeed may be translated into a linebreak (and thus newline) when it's read from the file. Make sure whatever program you're generating your input with is emitting linefeed/newline combos to indicate linebreaks. Alternately, read your input infile in in binary mode and write it back out replacing single linefeeds with the combination, then use it normally in text mode.

[daven25]

Thank you Cire and Xerzi! I used a hex editor to diagnose the problem as you suggested and I think I've found it. Apparently, there is a hex key "0A" (actually, 14,000 of them) in my csv file. From what I understand, this is the value for a line break. I'm going to cheat and just delete all of the hex keys with that value. For now, it seems like it's importing everything. Hopefully I haven't overlooked anything. Thank you for your help. I will report back again after I finish building my program. Cheers!