tokenize-rt adds ESCAPED_NL for a backslash-escaped newline "token"; tokenize-rt adds UNIMPORTANT_WS for whitespace (discarded in tokenize); tokenize-rt normalizes string prefixes, even if they are ...

This is a problem for xdoctest because it uses tokenize to determine if a statement is "balanced" (i.e. if it is part of a line continuation or not). This is the magic I use to autodetect PS1 vs PS2 ...