CSE 109    Program 4    Due   9 PM Thursday 26 March 2008
          Late submission dates: 28, 29 March 2008

     In this programming assignment I ask you to develop a class Lex,
     which acts as a tokenizer for the FPCALC Language.  Your file p4.cc
     should contain exactly the code which can be found in the file
     /proj/csc109/p4/testLex.cc. When I test p4 I will discard your
     copy of p4.cc and use the code in /proj/csc109/p4/testLex.cc. THUS, YOU
     MUST NOT CHANGE THE CODE IN testLex.cc.

     Part of the program involves using the class Word, which I partially
     developed in class. You should finish developing the class Word. Among
     the methods you should implement are the logical operators (<, <=, >,
     >=, ==, and !=).  In /proj/csc109/p4 I have stored the files  word.cc
     and word.h. The expected output of the program is in a comment at the
     bottom of the program stored in /proj/csc109/p4/testLex.cc.

     First I describe how your class Lex should behave. It has the following
     methods(I don't indicate the paremeters or their types).  Note, the
     main job of Lex is take in a string and then return the various "tokens"
     in the string.

         setString()   store a string of characters to tokenize
         next()        return the next token in the string, unless there
                       are no more, in which case return DOLLAR (see below)
                       The list of possible tokens is below
         atEnd()       true if and only if there are no more tokens to
                       return
         reset()       go to some place in string and start the tokenizing
                       process
         word()        return the token as a Word
         position()    return your location in the string
         peek()        return what would be the next token when next() is
                       called


       1.  Your class should have public (static) constants, corresponding
        to various tokens (value of constant in parentheses, where, e.g.,
        MUL stands for 'mul, while SMUL stands for '*', LPAR stands for ')',
        etc.):
        JUNK(-1), NUMBER(0), IDENTIFIER(1),PLUS(2),MINUS(3),TIMES(4),
        DIVIDE(5),CLEAR(6), ADDTOMEMORY(7), REMOVEMEMORY(8), CLEARMEMORY(9),
        GETSCREEN(10), ENTER(11), LBRACK(12), RBRACK(13), LPAR(14), RPAR(15),
        PUSH(16), POP(17), STACK(18), JUMP(19), JUMPL(20), JUMPG(21),
        JUMPLE(22), JUMPGE(23), JUMPE(24), JUMPNE(25), COLON(26),DOLLAR(27),
        END(28);
        Note: The constants ADDTOMEMORY, REMOVEMEMORY, and CLEARMEMORY are
              obsolete and can be ignored. The constants LBRACK('[') and
              RBRACK(']') will be used later in the semester.
     2.  next()   Read the next token in the string and return its type,
        i.e., one of the constants from (1). Store the actual string read in
        a Word.  If it is a number, store the value of the number as well.
        If at the end of a string, REFUSE TO GO ANY FURTHER and return DOLLAR.
        (If at the end ofa string, repeated calls to next() should return
        DOLLAR.)  Calls to next() should ignore comments, which are denoted
        by the '$' (DOLLAR).

       3.  position() returns an int giving the number of characters read (so
        far) from the current string

     To submit your assignment, create a subdirectory of /cse109.091 called
     p4 (i.e., from your root directory /cse109.091/p4) for doing your work.
     Finally, indicate you wish the assignment to be collected by executing
     the unix command "touch DONE", which will create a file of 0 bytes
     named DONE.  Then all the files in the subdirectory will be collected.
     I will test your program using your Makefile and our copy of p4.

     Hints:
        (1)  The description of next() above states that comments
             should be ignored. Because they are denoted by the dollar sign
             this is automatically accomplished by having next() refuse to
             go past the dollar sign. At the same time, you should append
             the dollar sign to any string that is passed to instances of
             Lex.
        (2)  It will be very helpful to have a "lookup table" that stores
             each of the tokens, e.g., "read", "+", etc., along with their
             corresponding values.  You should use your class Table< , > from
             programming assignment 3.  But what if the classes you developed
             for programming assignment 3, Word and Table, do not function
             well. In that case, I provide the files table.t, check.h, and
             check.cc.
        (3)  To understand the expected output of p4.cc read the program
             in testLex.cc and see the comment at the bottom of testLex.cc
             (your p4.cc).
        (4)  The declarion and code for class Word (incompletely) developed
             in class is stored in /proj/csc109/p4/word.h and
             /proj/csc109/p4/word.cc

     As usual I warn you about unfair collaboration.  Read the syllabus if
     you are unclear about what constitutes unfair collaboration.