External : Post-Processing

The tokens and nodes you generate can be manipulated external of the parser by you using your custom code.

With normal parser code, we first call AlgodalParser_GetTokenizeResult to generate the tokens. Then we pass those tokens to AlgodalParser_GetAnalyzeResult to generate the nodes.

The your can manipulate this however you feel for futher processing.

Important

There is absolutely no need to do post-processing if the parser does everything you need already. However, in rare cases, if the parser can’t do everything you want, then you can do post processing.

For example we want to parse Python. A python parser has to use logic to generate DEDENT token. Our parser does not support generating tokens from logic so we can do our post processing on our generated tokens.

# this is a rudimentary parser for testing (not meant to be a full parser)

@token:
@action alpha    <[+] @oneof $"ABCDEFGHIJKLMNOPQRSTUVWXYZ">
@action numeral <[+] @oneof $"0123456789">
@action a_other  < @first numeral | [+] "_" | alpha>
@action name     <[*] "_" alpha [*] a_other>

@action newline <@first "\r\n" | "\n">
@action space   <[+] @oneof " \t">

@action comment  <"#" [*] @not "\n" newline >

@action indent @post_create
@action dedent @post_create

@syntax:
@action if_cmd <name {"if"}>
@action def_cmd <name {"def"}>
@action pass <name {"pass"}>

@action prog <def_cmd name "(" name ")" ":" newline if_cmd name ":" newline pass>


@config
{
    @tokenizer [
        @reduce newline,
        name,
        space,
        @discard comment,
        ":",
        "(", ")",
        numeral,
        indent,
        dedent
        ],
    @entrypoint [prog],
    @objectpoint [
        prog,
        space,
        name,
        if_cmd,
        pass,
        numeral,
        def_cmd,
        indent,
        dedent
        ]
}

Note

The POST_CREATE command allows us to declare actions. These has no definition and do nothing. We create the tokens for these action in our post-process.

We then do our post processing

# code exerpt
struct AlgodalParser_TokenizeResult tresult = AlgodalParser_GetTokenizeResult(program, text, length, 0);
tresult = TokenPostProcessing_GeneratePythonTokens(&tresult, program, text);
AlgodalParser_PrintTokenizeResult(program, tresult, text, 0);

TokenPostProcessing_GeneratePythonTokens is a function provided in it’s extend file however you can write your own if it is not good enough for your needs. Take some go through the generated parser types like struct AlgodalParser_Token and struct AlgodalParser_Node. Understanding them will help you to write code to manipulate them.

If you write fancy libraries to post-process feel free to add it to our public repo https://github.com/Algodal/Parser-Generator-Tool-PR Please include test cases so we can be sure that it works.