toka.nbk: Home | Index | Next Page: parser.c | Previous Page: New Pages

 parser

Toka's parser is fairly simple. It only handles input coming from a file stream (which includes stdin). Input sources are implemented as a stack.

At the bottom is the stdin file. Generally on startup, the bootstrap.toka. A script would be at the third position, and any files that includes get added above it.

The parser will read from the top file until either end. or an EOF (end of file) is detected. At this point, it closes the file and drops to the next one on the stack.

Internally, the fundamental function for parsing is get_token(). This accepts two arguments, a pointer to a buffer to place the resulting string, and a delimiter character. Parsing ends when the delimiter character is encountered. get_token() also leaves a pointer to the resulting token on the stack. A Toka-level wrapper, parse(), makes use of get_token().

When parsing, a number of escape sequences are recognized. These are listed below:

  \n
  Embed a carriage return (ASCII 10) into the token
 
  \r
  Embed a line feed (ASCII 13) into the token

  \^
  Embed ASCII 27 into the token. Useful with VT100/ANSI terminal
  escape sequences

  \\
  Embed a \ character into the token

  \"
  Embed a quote into the token

Processing of escape sequences can be enabled or disabled by turning the escape-sequences variable on or off. For example:

  escape-sequences off
  " \\ hello \\" type cr
  escape-sequences on
  " \\ hello \\" type cr

Tips

• Tokens have a maximum size of 4096 characters
• If the delimiter = 10, get_token() will also break on encountering ASCII 13.
• If the delimiter = 32, get_token() will also break on encountering ASCII 10 or ASCII 13.

toka.nbk: Home | Index | Next Page: parser.c | Previous Page: New Pages

Notebook exported on Sunday, 1 July 2007, 19:54:43 PM EDT