by::tokenDispatcher Class Reference

Token queue management for parser. More...

#include <tokenDispatcher.hpp>

Public Member Functions
void	add (nint newToken)
void	addFront (nint newToken)
nbool	pop (nint &output)
	Removes a token from the front of the queue.
ncnt	len () const
void	rel ()

Detailed Description

Token queue management for parser.

Manages a queue of tokens for lookahead and token reordering during parsing. Provides FIFO token dispatch with front insertion capability.

must return token in your rule after using dispatcher

if you don't return a token in flex file, dispatcher won't work. it's by implementation of flex. for example,

yourRule {
    PS.getDispatcher().add('+');
    // you didn't return here because you want to feed '+' back.
    // however, by implementation of flex, there is no chance
    // to trigger tokenDispatcher.
}
yourSecondRule {
    PS.getDispatcher().add('+');
    return STRVAL; // you returns some token. now next token will be '+'.
}

only takes token, not character

you may confuse that this class is just like unput(), but it's totally different. unput() affects stream itself. however, tokenDispatchers skips lexer and returns determined token forcely, not a single character. for example,

auto& disp = PS.getDispatcher();
disp.add('n');
disp.add('a');
disp.add('m');
disp.add('e');

you may think that above codes exactly same to ‘unput('e’); unput('m'); ....‘. but that's not true. if you use unput(), lexer will judge the token as STRVAL which contains string value, "name". however, above codes, returns 'n’ token individually to parser which makes it fail to continue.

Member Function Documentation

pop()

nbool by::tokenDispatcher::pop

(

nint &

output

)

Removes a token from the front of the queue.

Parameters

output

Reference to store the popped token ID.

Returns

true if a token was successfully popped, false if the queue was empty.

---

The documentation for this class was generated from the following file:

• /home/runner/work/byeol/byeol/module/core/worker/tokenDispatcher.hpp