This is an old revision of the document!
Table of Contents
Detokeniser
by Jon Ripley, January 2008, May 2011
The FNdetokenise function listed below accepts a string containing either a tokenised program line or tokenised code and returns a string containing detokenised code:
DEF FNdetokenise(S$)
IF S$ = "" OR S$ = CHR$0+CHR$255+CHR$255 THEN =""
PRIVATE Keywd$()
IF !^Keywd$()=0 THEN
DIM Keywd$(160)
Keywd$() = "AND","DIV","EOR","MOD","OR","ERROR","LINE","OFF",\
\"STEP","SPC","TAB(","ELSE","THEN","","OPENIN","PTR",\
\"PAGE","TIME","LOMEM","HIMEM","ABS","ACS","ADVAL","ASC",\
\"ASN","ATN","BGET","COS","COUNT","DEG","ERL","ERR",\
\"EVAL","EXP","EXT","FALSE","FN","GET","INKEY","INSTR(",\
\"INT","LEN","LN","LOG","NOT","OPENUP","OPENOUT","PI",\
\"POINT(","POS","RAD","RND","SGN","SIN","SQR","TAN",\
\"TO","TRUE","USR","VAL","VPOS","CHR$","GET$","INKEY$",\
\"LEFT$(","MID$(","RIGHT$(","STR$","STRING$(","EOF","SUM","WHILE",\
\"CASE","WHEN","OF","ENDCASE","OTHERWISE","ENDIF","ENDWHILE","PTR",\
\"PAGE","TIME","LOMEM","HIMEM","SOUND","BPUT","CALL","CHAIN",\
\"CLEAR","CLOSE","CLG","CLS","DATA","DEF","DIM","DRAW",\
\"END","ENDPROC","ENVELOPE","FOR","GOSUB","GOTO","GCOL","IF",\
\"INPUT","LET","LOCAL","MODE","MOVE","NEXT","ON","VDU",\
\"PLOT","PRINT","PROC","READ","REM","REPEAT","REPORT","RESTORE",\
\"RETURN","RUN","STOP","COLOUR","TRACE","UNTIL","WIDTH","OSCLI",\
\"","CIRCLE","ELLIPSE","FILL","MOUSE","ORIGIN","QUIT","RECTANGLE",\
\"SWAP","SYS","TINT","WAIT","INSTALL","","PRIVATE","BY","EXIT"
ENDIF
LOCAL flag%, O$, i%, ch%
IF ASCS$ = LENS$ AND ASCRIGHT$(S$,1) = 13 THEN
ch% = ASC MID$(S$, 2)+256*ASC MID$(S$, 3)
IF ch% O$=STR$ch%+" "
S$ = MID$(S$, 4, LENS$-4)
IF S$="" THEN =LEFT$(O$, LENO$-1)
ENDIF
FOR i%=1 TO LENS$
IF ASCMID$(S$,i%)=34 flag%EOR=1
IF flag% AND 1 THEN
O$ += MID$(S$, i%, 1)
ELSE
ch% = ASCMID$(S$, i%)
CASE TRUE OF
WHEN ch% >= 17 AND ch% < 128:O$ += CHR$ch%
WHEN ch% = &8D
ch% = ASCMID$(S$,i%+1)
lo% = ((ch% << 2) AND &C0) EOR ASC MID$(S$, i%+2)
hi% = ((ch% << 4) AND &C0) EOR ASC MID$(S$, i%+3)
O$+ = STR$(lo% + 256*hi%)
i% += 3
OTHERWISE O$ += Keywd$(ch% EOR 128)
ENDCASE
ENDIF
NEXT i%
=O$
Using FNdetokenise to list the error line
FNdetokenise is useful to list the source code of a line when one occurs.
Using FNdetokeniser to write a Quine
Using FNdetokenise it is possible to write a Quine, that is, a program which when run outputs its own listing:
P% = PAGE
WHILE (!P% AND &FFFFFF) <> &FFFF00
line$ = ""
FOR i% = P% TO P%+?P%-1
line$ += CHR$?i%
NEXT i%
PRINT FNdetokenise(line$)
P% += ?P%
ENDWHILE
END
In the author's opinion using this method or *LIST to write a Quine is cheating.
Using the built-in detokeniser
The BBC BASIC for Windows interpreter has a built-in detokeniser accessed using the *LIST command that detokenises a specified file and outputs the listing on the current output device. Whilst it is possible to write a simpler looking detokeniser utilising *LIST the resulting routine is significantly slower due to the number of file operations required.
DEF FNdetokenise(S$)
IF S$="" OR S$=CHR$0+CHR$255+CHR$255 THEN =""
IF LENS$>251 ERROR 100,"Line too long"
LOCAL T%, fIn$, hFile%, hOut%
DIM T% LOCAL 255
SYS "GetTempPath", 256, T%
SYS "GetTempFileName", $$T%, "tmp", 0, T%:fIn$=$$T%
SYS "GetTempPath", 256, T%
SYS "GetTempFileName", $$T%, "tmp", 0, T%
hFile% = OPENOUT$$T%:IF hFile% = 0 ERROR 100,"Cannot create temp file"
IF ASCS$=LENS$ AND ASC RIGHT$(S$,1)=13 THEN
BPUT#hFile%, S$+CHR$0+CHR$255+CHR$255;
ELSE
BPUT#hFile%, CHR$(LEN(S$)+4)+CHR$0+CHR$0+S$+CHR$13+CHR$0+CHR$255+CHR$255;
ENDIF
CLOSE#hFile%
hOut% = @vdu%?-130
hFile%=OPENUPfIn$:IF hFile%=0 ERROR 100,"Cannot create temp file"
OSCLI "Output "+STR$hFile%
OSCLI "List """+$$T%+""""
OSCLI "Output "+STR$hOut%
PTR#hFile%=0:S$=GET$#hFile%:CLOSE#hFile%
WHILE ASCS$=32:S$=MID$(S$,2):ENDWHILE
SYS "DeleteFile", fIn$
SYS "DeleteFile", $$T%
=S$
Do not use this version in production code.