When an engine like iPython uses the tokenizer

The console loop knows then we are in syntax mode and will not reset the args list and the default syntax engine immediately gets the tokenized result, but it is still tokenized. The syntax can check its own rules for line continuation or filenames and set the tokenizer controls appropriately, then call its own parsing routine. On first start, like :

Python with starting statements; // Python knows this is first time activated

Thus, on start and on each line, the Python manager can set the tokenizer controls.