Recognisers

What is a recogniser ?

A recognizer (not a compsci word) attempts to work out what to do when given a string or a token.

There are two types of string or token; words or literals (aka numbers). There are 3 things you can do with a word; take its XT and EXECUTE it, COMPILE, it or POSTPONE it. There are 2 things you can do with a (floating point) number; put it on the (float) stack, or (F)LITERAL it to put it on the (float) stack at run time.

A recognizer takes a token and tries to recognize it based on some characteristic. Can I find it in a dictionary? It’s word. Does it look like a floating point number? It’s a float. Etc.

If it recognizes the token, the recognizer may transform it into some usable value; like a number, or an NT (name token) and so on. It also returns a value (for instance, a pointer to a 3 element table of actions) that represents the kind of token it recognized, and that value tells the system what to do with the converted token when interpreting, compiling or postponing (for instance, selecting one of the 3 actions in the table and executing it).

The reason that it’s of so much interest is that it allows pluggable standardized recognizers for a variety of tokens. ANS Forth is readily extensible, except in this key area. [alexrivadpm]

Question: So the recognizer if successful is taking a token and tagging it with (value, action) and passing that back to forth to process.

Answer: Yes, that’s about the sum of it. [alexrivadpm]