Trying to Build a Tokenizer, problem defining Token class

I'm trying to make a vector to store all the tokens inside a .java file. What I've got so far is a class to define the type Token as a collection of 2 strings (the token type, and the token data), a class to Tokenize the strings, and the main class.
However inside my program I'm getting 2 error types:
1
2
3
4
5
  class Token
{
    public string type;
    public string characters;
}; //expected ':' before string at both variable definitions 


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
vector<Token*> Lexer(FILE * fl)
{
    char ch;
    vector<Token*> T;
    do
    {
        ch = getc(fl);
        if(isWhiteSpace(ch)){}
        //ignore
        else if(isalpha(ch)||ch=='_')
        {
            Token* A=new Token();
            string x;
            x+=ch;
            do
            {
                ch=getc(fl);
                x+=ch;
            }while(isalpha(ch)||ch=='_'||isdigit(ch));
            A->characters = x;
            if(x=="true"||x=="false")
            {
                A->type = "BoolL";
            }
            else if(x=="int"||x=="char"||x=="string"||x=="real"||x=="bool"||x=="unit")
            {
                A->type = "VarType";
            }
            else if(x=="and")
            {
                A->type = "MultiplierOP";
            }
            else if(x=="or")
            {
                A->type = "AdditionOP";
            }
            else A->type = "Identifier";
            
            T.insert(A);
            //no matching function for call to 'std::vector<Token*>::insert<Token*&>'
        }//identifier
        else if(isdigit(ch))
        {
            
        }//number
        else if(isOperator(ch)){}//operators
        else if(isPunct(ch)){}//punctuation
        else {}
    }while(ch!=EOF);
    return T;
}//lexer returns a dynamic list of tokens from a file stream
//the tokens still need to be organized into their own grammar rules beyond this method
//those grammar rules will be used to create the parse tree at the end 


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
int main(int argc, char** argv) 
{
    //first input text stream from file
    //tokenize
        //tokenizer will look through all characters in the given input stream to create the tokens
        //the tokens will be stored in a dynamic list to send to another function will which parse the tokens
        //and make up the grammar rules
        //the grammar rules will then be sent to a function to create the parse tree
    //parse each token against structure functions for each command type
    //structure functions will depend on some symbol in lookup table
    //store data in parse tree nodes
    //connect nodes into an n-tree
    cout << "Open what file?";
    string str;
    char c;
    cin >> str;
    FILE * f;
    f =fopen(str.c_str(),"r");
    vector<Token*> T = Lexer(f);
    cout << T.front()->type;
    fclose(f);
    
    return 0;
}


In my main I call the Lexer(FILE*) method to store the vector<Token*> data inside a variable in the main class for use in another method later on, there are no errors being picked up in that class, and I'm just using it to be able to test that I'm actually picking up the tokens for now
The Lexer function is incomplete and I still have to factor in the cases where data is a numerical token, an operator, etc.
expected ':' before string at both variable definitions

So put a colon behind public. You only need "public:" once though, as access specifiers apply to following statements. http://www.learncpp.com/cpp-tutorial/115-inheritance-and-access-specifiers/

no matching function for call to 'std::vector<Token*>::insert<Token*&>'

Check out the signature for std::vector::insert() here: http://www.cplusplus.com/reference/vector/vector/insert/
You probably want to use std::vector::push_back()
Topic archived. No new replies allowed.