The goal of a recognizer is to dynamically extent the Forth command interpreter and make it understand and handle new data formats as well as new synatax’s. The present, 2nd generation recognizers achieve this by generalizing the classic interpreter with an API to factor the components. Recognizers are portable across different forth’s.
Recognizers are not a new concept for forth. They have been discussed earlier.
Number Parsing Hooks in 2007.
Presentations held at Euroforth conferences
More recognizer examples are available at The Forth Net.
Most changes are only wording changes. Most importandly many of the key words are renamed to better ones. They now describe what they do in a less confusing way (I hope). No changes were made to the stack effect so all existing code continues to work with a simple search and replace.
non-normative wording changes
The data type token DT is not new actually. It got only a more prominent role since it is one result of the parsing process but is not related to recognizers themselves. It provides information about the data type and how to handle a certain data type (hence the name) inside the interpreter. It is not relevant where it comes from.
The gforth team suggests that the
DT>x words should have
a different stack effect.
DT>COMP ( i*x DT:TOKEN -- j*y XT )
The input is what comes from the recognizer parsing words, the
result should be feedable directly to
EXECUTE to achieve the
desired semantics. The words may change the result set too.
In the reference implementation this is not used however.
The primary purpose of this seems to be an optimization and an unification of the name token and the data type token for named words), so that DT>COMP becomes identical to NAME>COMPILE for name tokens.
The 1st formal RFD (pdf), (txt) was published at october, 3 2014. Version 2 (pdf), (txt) has been published on september, 20 2015. It improves the proposed standard section and adds a long chapter discussing the recognizer design based on feedback from version 1. The 3rd version has been started immediately after v2 due to a suggestion changing the POSTPONE action. Version 3 (pdf), (txt).
The papers linked below give some historical background information.