Our price for the intelligent tokenizer

My version is coming in at 120 lines of code, not likely to grow much or shrink much.  The swapping between file name mode and expression mode was easy, it was dealing with four extra possibilities of op code strings, adding the quoting/ escaping function, and then the configuration function. Enugh hooks to get bu for a couple of console versions without a major rewrite.  But, as always, consider this a cutnpste reference, and look to other solutions.  Look before leaping, we are setting a standard for snippet authors for years to come, it is an interactive process.

I integrated the new tokenizer with console loop and posted it, untested, uncompiled, on the right side under links.