Lil: A Scripting Language

Lil is part of the technology that powers Decker, a multimedia creative tool inspired by HyperCard. Decker uses Lil for adding custom behavior to decks and the widgets within. Lil is designed to be learned in layers, but it is a richly multi-paradigm language which incorporates ideas from imperative, functional, declarative, and vector-oriented languages.

on mode a do   
 r:()
 each x in a
  r[x]:1+r[x]
 end
 first extract key orderby value desc from r
end

mode[1,2,2,3,4,2,1]

Lil's implementation needs to be fairly small, as an interpreter is shipped with every standalone Decker document along with the rest of the runtime. At the same time, Lil should be prepared for the everyday needs of Decker users, with primitives to handle common use-cases and a minimum of boilerplate.

The language therefore tries to thread the needle between a design which is simple and a design which is ergonomically pleasant. There are a small number of datatypes, features and primitives which compose in many useful ways. The query syntax generalizes to manipulation of dictionaries, lists, and strings, and replaces many individual primitive operations in other languages with a single algorithmic framework. Simple things are easy, and complex things are possible.

• Lil, A Primer
• Types and Conversions
• Lil, the Imperative Language
• Lil, the Functional Language
• Lil, the Query Language
• Lil, the Formatting Language
• Lil, the Vector Language
• Lil, the Decker Language
• Appendix 1: Unary Primitives
• Appendix 2: Binary Primitives
• Appendix 3: Language Grammar

Lil, A Primer

Let's begin by looking at a few practical examples of Lil scripts you might encounter or compose while using Decker, to give you a loose impression of the language. Consider the following script on a button widget:

on click do
 
 go["DungeonEntrance" "BoxIn" 15]
end

The on NAME do ... end structure is how you define a function. Buttons in Decker call a function named "click" when they're clicked. The octothorpe (#) indicates a line-comment: the remainder of the line # adventure awaits! is a hint for human readers, but has no significance to Lil. The function go is built into Decker; it moves to a different card. The square brackets ([]) indicate that we're calling go and providing arguments. The arguments to go are the name of a destination card represented as a string, a second string indicating a transition animation, and a number indicating how many frames the transition lasts. Lil strings are enclosed in double-quotes (""). The arguments within the brackets are separated by whitespace, not commas!

In another button script, clicking the button is only meant to navigate to another card if the user has previously found a "key", represented as a checkbox widget named haskey on a card named inventory:

on click do
 if inventory.widgets.haskey.value
  play["DoorSqueak"]
  go["TheCrypt"]
 else
  alert["The door is locked!"]
 end
end

The if CONDITION ... else ... end structure is how Lil handles conditional logic. You can omit the else or add extra clauses with elseif. The "dot-notation" sequence (inventory.widgets.haskey.value) is how we access the hierarchical parts of a deck: the value of the haskey checkbox of the widgets contained in the inventory.

For more elaborate conditions you can form expressions using & (logical AND), | (logical OR), ! (logical NOT) and the familiar arithmetic comparison operators =, < and >. Note that Lil does not have "combined" operators like !=, >= or <=. Expressions in Lil are always carried out with uniform right-to-left operator precedence; use parentheses to group subexpressions:

on click do
 i:inventory.widgets
 goldCoins:i.money
 hasPass:i.trollPass.value
 if hasPass & (goldCoins.value>5)
  alert["The troll charges you 5 gold coins."]
  goldCoins.value:goldCoins.value-1
  go["TrollTavern"]
 else
  alert["The troll gruffly refuses you entry."]
 end
end

The colon : is Lil's assignment operator. You can stash numbers, strings, or even references to widgets in variables like i, goldCoins and hasPass to avoid repeating yourself. You can also modify the fields of widgets, like inventory.widgets.money, using assignments.

If you're looking to add simple interactive behaviors to your decks, you may already know enough to get started. The rest of this reference manual will focus on Lil itself, separately from its usage within Decker. You can find more practical examples and scripting tutorials among the .deck files packaged with Decker.

Types and Conversions

There are 8 types of value in Lil: numbers, strings, lists, dictionaries, tables, functions, interfaces, and the nil value.

• Numbers are floating-point values, possibly with a sign: 42 37.5 -29999.
• Strings are a sequence of characters, written enclosed in double-quotes. The special characters backslash, double-quote, and the newline character (\n) are preceded with a backslash escape: "apple" "foo\nbar". The binary fuse and format operators can both be used for concatenating strings.
• Lists are ordered sequences of values which can be indexed by numbers. The empty list is written as (), and the unary operator list constructs a single-element list from any value. Lists can be joined with the , operator.
• Dictionaries are ordered sequences of pairings between keys and values, both of which can be any type. Dictionaries can be made by the binary operator dict, or a list can be promoted to a dictionary by assigning at non-numeric keys. Dictionaries can be unioned together with the , operator. The keys operator gets the keys of a dictionary, and the range operator gets the values of a dictionary.
• Tables are a rectangular array of values for which every column has a string as its key. Tables can be made with the unary operator table or with the insert statement. The binary operators , (append), join (natural join), and cross (cartesian join) offer a selection of tabular joins.
• Functions have a name, take arguments, and return a result. They are declared with on, and may be freely passed around or stored in variables. first of a function gives its name, and keys of a function gives a list of its named arguments.
• Interfaces are opaque dictionary-like values used to represent system resources or Input/Output devices. Accessing or writing to an interface may produce side-effects.
• nil represents the absence of a value: an undefined variable, an out-of-bounds index (outdex), a failed attempt at parsing a value. The unless and fill operators can be used to substitue a default value for a nil. Otherwise, nil will happily coerce to contextual "empty" values.

Lil has a soft, spongy, dynamic type system in which values do their best to convert to a more relevant type as the need arises.

• When a number is required, nil becomes 0, strings are parsed, and lists or dictionaries attempt to convert their first element. Otherwise, the number 0 is used.
• When a string is required, nil becomes "", numbers are formatted, and lists are recursively converted to strings and joined. Otherwise, the empty string is used.
• When a list is required, nil becomes (), strings are treated as a list of their characters (each a length-1 string), dictionaries are treated as their value list, tables are treated as the list of their rows (each a dictionary), and anything else (number, function, interface) is enclosed in a length-1 list.
• When a dictionary is required, strings and lists become dictionaries from their indices to their elements, tables become dictionaries of their columns, and otherwise an empty dictionary is used.
• When a table is required, nil becomes the empty table, lists become a single-column "value" table, and dictionaries become two-column "key" and "value" tables. Anything else is interpreted as if it were a list.

When indexing into values, we will uniformly refer to elements:

• An element of a number is itself.
• An element of a string is a character, which itself is a length-1 string.
• An element of a list is one of its items.
• An element of a dictionary is one of its values.
• An element of a table is one of its rows, as a dictionary.
• An element of an empty string, list, dictionary, or table is nil.

The number zero (0), nil, and the empty string (""), empty list (()), or empty dictionary are all falsey, and any other value (including any table, function, or interface) is truthy.

Lil, the Imperative Language

Lil scripts are a sequence of expressions, including literal values, references or assignments to variables, conditionals (if), each and while loops, and function declarations (on).

Variable and function names may contain any alphanumeric characters (as well as ? and _), but must not start with a digit. Variable assignments use the symbol :, read as "gets" or "becomes":

a: 23
longer_name: "a string"

If a value has never been explicitly placed in a variable, it contains nil. You can access or assign to elements of a list or dictionary by subscripting with brackets:

(11,22,33)[1]
"String"[2]
l[4]:2
d["fruit"]:"apple"

For convenience, indexing may also be performed with "dot notation", where a name after the dot is treated as a string index. These expressions are equivalent:

d["fruit"]:"cherry"
d.fruit:"cherry"

If you index a list with integers within range of its count it will act like a list. If you index a list like a dictionary- with keys that are not integers within range of its count- it will become a dictionary.

b: 5              
b[0]: 5           
c                 
c.fruit: "yes"    

When the leftmost part of an assignment expression is an expression- not just a variable name- the value of that expression is amended and a new value with appropriate changes is returned:

(11,22,33)[1]:44  
"Cat"[1]:"ive"    
().baz:99         
                  
(11,22).baz:33    

Note the important distinctions between these similar-looking cases:

foo:11,22,33

foo[1]:44     

(foo)[1]:44   
              

(foo[1]):44   
              

If you index with a dot followed immediately by another dot or open-bracket it means "each element". Use this for drilling into nested lists or dictionaries:

t:("AB","CD","EFG")

t[1]        
t.[1]       

d.a.key:"apple"
d.b.key:"pear"

d           
d.a         
d.a.key     
d..key      

Lil has a number of unary and binary operators- see appendix 1 and 2 for details. Many, like + and *, should be familiar from mathematics or other programming langugages. The most important thing to remember in Lil is that expressions have uniform operator precedence: expressions are always carried out right-to-left, unless explicitly parenthesized:

3*2+5    
3*(2+5)  
(3*2)+5  

You can make decisions with if. The keyword if is followed by a conditional expression, one or more statements, and finally the keyword end. The statements inside the if ... end will only be executed if the conditional expression has a truthy value. You can also optionally include an elseif keyword (followed by a conditional expression) or just an else keyword to divide the body of the if into multiple cases.

if 1>2
 "narp"  
end

if 5
 "yarp"  
end

if 5
 "also yarp"  
else
 "also narp"  
end

if 1>2
 "narp"    
elseif 1<2
 "yarp"    
else
 "narp"    
end

You can iterate over the elements of a value with an each loop. The keyword each is followed by zero or more variable names, the keyword in, an expression giving the source to iterate over, one or more statements forming the "body" of the loop, and finally the keyword end. The source is always treated as a dictionary, and for each iteration of the loop the variable names provided are assigned to the value, key, and index of the current pairing in that dictionary. For a list, key and index will naturally be identical. Here are a few examples of each loops:

each x in 3,5,7
 x
end

each val key in d
 val,key
end

r:0
each x in range 10
 r:r+x
end

You can also repeat a body of code an indefinite number of times with a while loop. The keyword while is followed by a conditional expression, one or more statements forming the "body" of the loop, and finally the keyword end. The conditional expression is evaluated before each iteration of the loop, and the loop stops if this ever results in a falsey value. Here's an example:

a:5
while a>3
 print[a]     
 a:a-1
end

To declare a function, use the keyword on followed by a name, zero or more argument names, the keyword do, one or more statements comprising the "body" of the function, and finally the keyword end. To call a function, use its name and a set of bracketed expressions corresponding to the arguments it takes. Extra arguments are ignored, and missing arguments are bound as nil:

on pair x y do
 x,y
end

pair[3 5]    
pair[3]      
pair[3 5 7]  

pair[3,5]    
pair 3 5     

If you want to define a function that takes a variable number of arguments- a so-called variadic or ambivalent function- declare the function with a single named argument prefaced with .... When the function is called, this argument will contain a list of every argument provided to the function:

on several ...x do 1-x end
several[11 22 33]

Lil, the Functional Language

As we've seen above, thinking about Lil as an everyday garden-variety imperative language is perfectly sufficient for writing scripts. Some characteristics, though, make it well-suited to the functional style of programming, in which we aim to minimize mutation, compose our program from functions which do not have "side-effects", and make use of so-called "higher-order" functions- functions which take functions as arguments.

Values in Lil are truly values. They have copy-on-write semantics: an assignment to part of a value creates an entirely new value, leaving any other references to the original value unchanged:

a:1,2,3
b:a
b[1]:5
show[a]  
show[b]  

Statements in Lil are always expressions. That is, they always return a value and can be composed within larger expressions. Assignments evaluate to the value being assigned. if returns the value of the last statement in its taken half. (A missing else just means the falsey half evaluates to nil.) while likewise evaluates to the last statement on the last iteration of its body:

x:y:z                       
a: if x>5 99 else 33 end    
c: while b<100 b:b*2 end    

The each loop collects together the results of each iteration of the loop and returns them. If the input was a dictionary, the output will be a dictionary with the same keys. Otherwise, the output will be a list. Some other languages call this operation map:

each x in 3,5,7   
 x*100
end

t.foo:"one"
t.bar:"three"

each v in t       
 count v
end

You can pass functions to other functions by name:

on apply func do
 func["two"]
end
on twice x do
 x,x
end
apply[twice]    

Since on...end is an expression, you can also directly substitute it into the call, just as with "anonymous" or "lambda" functions in other languages.

apply[on thrice x do x,x,x end]

One advantage of functions always having a bound name is that there's no special combinator plumbing required for anonymous recursion. If the name never matters, consider using the name _ to signal your intent.

Lil uses lexical scope: variables will resolve to the closest nested binding available, and the local variables of a caller to a function will not be visible or modified by the callee (unless the callee's definition is nested in the caller):

global:333
on quux x do
 v:99
 x[77]
 print[v]           
end
on zami x do
 v:23
 print[global,v,x]  
end
quux[zami]

Furthermore, functions close over variables in their lexical scope, allowing for encapsulated "objects" with their own mutable state:

on counter x do
 on inc do
  x:x+1
 end
end
a:counter[100]
b:counter[200]
print[a[]]        
print[a[]]        
print[b[]]        
print[a[]]        
show[list x]      

Whenever an assignment is carried out, if the variable name in question has been assigned to in any surrounding lexical scope, the assignment will update the closest definition. If the name has never been assigned to before, a new local variable will be created at the current scope. The on, each, and query (select, update, extract) statements always create new local variables for their arguments, loop variables, or columns, and can thus shadow (take precedence over) outer local variables of the same name. When in doubt, use unique names; it's much less confusing!

It is also possible to explicitly define a local variable by using the keyword local, a name, a colon (:), and an expression giving the local's initial value:

duplicate:"Alpha"
on func do
 local duplicate:"Beta"
 show[duplicate] 
end
func[]
show[duplicate] 

Finally, if you're mentally wed to the idea of expressing algorithms recursively, you may be pleased to discover that Lil supports tail-call elimination. If a recursive function calls itself (or another function) as the final operation in the function, it will not consume extra stack space:

on addrec x y do
 if x>0
  1+addrec[x-1 y]  
 else
  y
 end
end

on addtail x y do
 if x>0
  addtail[x-1 y+1]  
 else
  y
 end
end
addtail[80000 5]  

Lil, the Query Language

So far, we haven't looked much at tables. While one can operate on tables by indexing them or iterate over their contents with an each loop, the real power of tables comes from Lil's query syntax, which resembles a simplified version of SQL. Unlike some of their SQL counterparts, select, extract, update, and insert are all pure operations which strictly return a new value as a result.

A query consists of an operation, a sequence of column expressions (possibly with names), any number of where, by, and/or orderby clauses, the keyword from, and then finally an expression giving the source table. If the source is non-tabular, it will be converted into its table equivalent automatically. The operations are:

• select: query a table and obtain a table of results.
• extract: query a table and obtain a list or dictionary of results.
• update: produce an amended version of a table, preserving its overall structure and order.

If a query has multiple clauses, they're carried out right-to-left, just like Lil expressions. You can think of each clause as taking a table and a column expression, evaluating the expression within the context of the table (binding each table column as a variable), and using the resulting column to transform the input table into a different table. The clauses are:

• where x: filter the table to only the rows of x which are truthy.
• orderby x dir: sort the rows of the table such that they would put the values of x in order; dir can either be asc (ascending) or desc (descending). Lists are sorted lexicographically, numbers are sorted by value, and anything else is sorted lexicographically by its string equivalent.
• by x: group the rows of the table by the dictinct values of x, producing 1 or more "subtables". Once the source table has been grouped, all subsequent clauses and result column expressions are applied to the subtables individually. If more than one by clause appears in a query, the subtables will be "ungrouped" into a single table before re-grouping.

When column expressions are evaluated, several "magic" columns and variables are automatically defined:

• index: integer list; the original row number in the input table for each row in the current table.
• gindex: integer list; the row number for each row in the current group (subtable); this is "reset" after each clause is carried out.
• group: integer list; the group number for each row in the current group (subtable); this is 0 for all rows in an ungrouped query.
• column: dictionary; all of the columns in the current table or subtable, keyed by name.

When "ungrouping" or collecting the final result columns, the table is rectangularized: each group will have as many result rows as the column of maximum count. Grouping may change the order of rows- see update if you want to preserve it.

Given a simple table:

people.name:"Alice","Sam","Thomas","Sara","Walter"
people.age:25,28,40,34,43
people.job:"Developer","Sales","Developer","Developer","Accounting"
people:table people

Let's perform some selections:

select from people       










select name from people      










select firstName:name dogYears:7*age from people     










select name where name like "S*" from people     







select name index orderby name asc from people     










select name job by job orderby name asc from people     










select name job orderby (job join name) asc from people     

All result columns will be repeated to match the count of the longest result. Thus, if all the new columns yield a non-listy result, the group is effectively collapsed into a summary row:

select employed:(count name) job by job from people










select employed:(count name) job:(first job) by job from people

When computing columns, you're working with lists of elements, and taking advantage of the fact that primitives like < and + automatically "spread" to lists. When performing comparisons, be sure to use = rather than ~! If you want to call your own functions- say, to average within a grouped column- write them to accept a list like so:

on avg x do (sum x) / count x end

select job:(first job) avg_age:avg[age] by job from people

The same applies to any functions called in a where, by, or orderby expression. The predefined aggregation functions sum, prod, raze, min, and max may come in handy!

The update statement has the same syntax and clauses as select, but behaves differently: results are merged with the original table, and the original order of rows is preserved. orderby can be used with update, but will only impact the inputs to column expressions, not the order of the result rows.

update job:"Engineer" where job="Developer" from people

As with select you can compute new named columns- values will be filled in with nil for rows masked off by a where clause.

update manager:random[name] where job="Developer" from people

The fill primitive can be useful for substituting default values into the nil elements of columns or entire tables at once:

t:update manager:random[name] where job="Developer" from people

select name "manager":"Julia" fill manager from t










"n/a" fill t

When mixing queries with other types of code, it may be very useful to execute a query and get back simple lists, strings, or numbers. The extract statement is another variation on select which unpacks its result directly into a value, instead of a result column in a table:

jobs:extract first job by job from people

extract gives you access to a variety of useful operations for tables, dictionaries, or even lists:

extract value orderby value asc from jobs              


extract index orderby value asc from jobs              


extract value orderby index desc from jobs             


extract list index by value from "ABBAAC"              


extract list value by floor index/3 from "ABCDEFGHI"   


extract first value by value from "ABBAAC"             

If names are specified, all results are collected into a dictionary:

extract a:first age b:last age orderby age asc from people

If no columns are specified, the result will be the first column of the table:

extract orderby value asc from "BEDAC"

The insert statement adds new rows to a table. It is followed by a sequence of one or more column names terminated by the with keyword, and then a series of expressions supplying the values of each row. The insert statement concludes with end, when creating a new table from scratch, or into followed by an expression evaluating to a "source" table when appending to an existing table.

insert name job age with "John" "Writer" 32 end






insert name job age with "John" "Writer" 32 into people

As a special case, if the value to insert into is a number, treat it as an empty table; insert ... into 0 is equivalent to insert ... end.

Lil also offers two basic joining operations: join (natural join), and cross (cartesian/cross join):

jobs:insert job salary with
 "Sales"      85000
 "Developer"  75000
 "Accounting" 60000
 "Facilities" 50000
end









people join jobs










guests:insert name with "Alice" "Joan" "Oscar" "Thomas" end









select a:name b:name_ where name < name_ from guests cross guests

For a row-wise join- concatenating tables with the same columns- you can use ,. You can likewise take or drop rows or columns from a table. If the left argument to take is a list of numbers, it picks out those rows very much like a generalization of select:

"name" take people










("age","job") take people










3 drop people







(0,2,3) take people

The flip of a table transposes its data, promoting the first column (or, if it exists, a column named key) from the original table to column keys for the result:

expenses:insert kind jan feb with
 "tax"   11 55
 "gas"   22 66
 "power" 33 77
 "food"  44 88
end









flip expenses

The key column makes the flip of a table a reversible operation. If you don't need the original keys, you can discard them with drop:

"key" drop flip expenses

In some situations you may wish to construct or query tables containing columns which have names that are reserved keywords (like count or range) or are not valid Lil identifiers (like a name with spaces). To define such a column, enclose the name in double-quotes:

denormal: select "with \"escapes":index "count":value from "ABC"








insert "pet name" "pet species" with
 "Galena"  "Chicken"
 "Pippi"   "Chicken"
 "Chester" "Toad"
end

To reference these columns in a query, every column expression has the variable column bound to the entire table (or subtable, when grouping) within column expressions. As usual, you can access specific columns by indexing this table with a string:

select where column["with \"escapes"]>0 from denormal







first extract column from denormal

Extracting column may occasionally be useful for debugging complex queries!

In general, use select for narrowing down or summarizing tables, update for making changes to a table while preserving or extending its existing structure, extract for pulling data out of a table for use elsewhere, and insert to make new tables or append to existing ones.

It may also be helpful to think in terms of how the shape of output tables relates to input tables for various table operations:

Lil, the Formatting Language

The parse and format primitives are used for breaking strings apart into Lil values, and formatting Lil values into a customizable string representation, respectively. Both take as their left argument a format string with a concise notation for controlling these processes. In many cases, these format strings are symmetrical: the parse of the format under the same Format string will be an identity operation, and vice versa:

f: "0x%04h"       
f format 123      
f parse "0x007b"  

Format strings consist of a sequence of patterns and literals. Patterns always begin with a % character, may contain several optional flags, and end in an alphabetic character. Pattern flags configure the details of how each pattern behaves, and may appear in the structure %[name]*-0N.DX, where N and D may be 1 or more digits 0-9, and X is a pattern type. All flags are optional. If present, their meanings are as follows:

• name may contain any number of non-] characters. If any pattern in a format string has an explicit name, any other patterns without an explicit name are assigned a default name corresponding to their index, starting from 0.
• * this pattern will be validated, but will not produce output values. When formatting, instead of consuming an input value, these patterns will use nil.
• - specifies left-justification for N, as decribed below. The default is right-justification. The %r/%o patterns use - to invert the set of valid characters.
• 0 specifies padding with 0 for N, as described below. The default is padding with spaces.
• N indicates that the value will be padded to at least N characters, using the justification and padding character selected with - and 0.
• .D for %f/%c, the value should be formatted with exactly D decimal places. The %r/%o patterns use D to indicate a number of valid characters. For any other type, the formatted string will be truncated at D characters, respecting justification.

Pattern types are as follows:

When parsing, each pattern will be matched against input in sequence, consuming some number of input characters and producing output values, and literals will be expected; if at any point a pattern or literal fails to match against input, parsing will cease, and any subsequent patterns in the format string will yield nil. Thus, a given format string will always yield a fixed number of results, no matter the input. In the case where there is exactly one value-yielding pattern in the format string, parse output will simply be that value instead of a list.

"%f %s %i" parse "12 apples"                        
"%f %ss" parse "12 apples"                          
("amount","noun") dict "%f %ss" parse "12 apples"   
"[%s]" parse "[something]"                          

The right argument to parse may be list of strings; in this case the output will be a list of rows, where each row contains the values from parsing one input string. The flip of this would give columns instead. Here's an example of parsing a table of fixed-width records:

form: "%6s%6c%2i"
data: "apple  $1.00 1\ncherry $0.3515\nbanana $0.75 2"

r: form parse "\n" split data


table r








flip r


t: table ("name","price","amt") dict flip r








("\n",form) format t              

When formatting, literals will be included in the output string and each pattern will control conversion of one value from the input list, with a few exceptions as explained below. Missing arguments will be interpreted as nil.

"%i,%a,%i" format 1,(list 65,66,67)  

The left argument to format may be a list of strings; in this case it is a series of alternating delimiters and format strings, with the last item always being a format string. Each format string is "pushed" one layer deeper into the right argument of format, and (if present), the delimiter will be used to fuse these intermediate strings, allowing a simple statement to recursively format a complex structure:

                    ()  format 11,22                   
                "%03i"  format 11,22                   
          (list "%03i") format 11,22                   
           (":","%03i") format 11,22                   
    ("<%s>",":","%03i") format (list 11,22),(list 33)  
("@","<%s>",":","%03i") format (list 11,22),(list 33)  

Recursive formats will "explode" tables into lists of row-lists:

t: insert alpha beta with "one" 11 "two" 22 end







(list "%u - %i") format t

If N is specified, the %s,%a and %b patterns will read up to N characters of input when parsing. Otherwise, the next character in the format string will be interpreted as a delimiter, and input characters will be read until that delimiter is encountered. (Note that if %s/%a/%b is immediately followed by a pattern, the delimiter will be %!) If %s appears at the end of the format string, it will simply read the remainder of the input string.

The %p pattern operates on a dictionary with numeric fields for year, month, day, hour, minute, second:

"%p" parse  "2021-02-03T04:05:58Z"  
"%p" format ().year:1984            

The %n pattern can be used for progressive parsing. It also offers a way of finding the first index of a given character in a string:

data: "one,two,three"
"%s,%n" parse data         
"%s,%n" parse 4 drop data  

"%*sA%n" parse "BBCABA"    

The %m and %z patterns can be used to identify the success or failure of an entire pattern.

"%i%m" parse "23"                      
"%i%m" parse "0"                       
"%i%m" parse "orange"                  

The %r pattern is followed by one or more "valid" characters, the count given by D (or 1 by default). This pattern matches and collects input if and only if the input characters are within this set of valid characters. The - flag inverts this behavior, such that the pattern matches and collects only characters which are not in the valid set. If N is specified, exactly N characters must be matched; otherwise %r will accept zero or more valid characters. The %o pattern is exactly like %r, but if N is unspecified it will accept zero or one valid characters:

"%.2r01"     parse "01110201"     
"%*.2r01%z"  parse "010","012"    
"%r-%i"      parse "----45"       
"%o-"        parse "A","-A","--A" 
"%*o-%i"     parse "-45"          
"#%-r\n\n%s" parse "# comment\nA" 

The %q and %v patterns are useful for manipulating Lil source code, parsing/matching and formatting Lil string literals and Lil variable names, respectively:

"%v[%q]%m" parse "func[\"foo\"]"    
"%q" format "a string"              

The %j pattern can be used to format or parse data as JSON. When formatting JSON, Lil dictionary keys will be cast to strings, tables will be cast to their rows (as by rows), and anything other than a number, string, list, or dictionary will become a JSON null:

"%j" format (11,22) dict (33,44)    
"%j" format list 11,22              
"%j" format table 11,22             
"%j" format on x do 2+x end         

When parsing JSON, the value true will become the number 1, false will become the number 0, and null will become nil. This JSON parser is highly tolerant and will among other things accept non-string JSON values as dictionary keys, single-quoted strings, missing , and : delimiters, and some missing trailing delimiters. Postel's Law, baby!

"%j" parse "[true,false,null,1]"   
"%j" parse "{11:22,33:44"          
"%j" parse "{'foo':22}"            

The %J pattern handles a superset of JSON called Lil Object-Value Encoding (LOVE) which can represent all of Lil's basic datatypes and certain Interface types losslessly. Beyond the behavior of %j, %J supports dictionaries with keys of any Lil type, represents tables in <> (otherwise like a string-keyed dictionary of columns), and recognizes "Data Block" encodings for Images, Sounds, and Arrays, as described in The Decker File Format:

"%J" format ((list 11,22),(list 33,44)) dict "one","two" 
"%J" format insert a b with 11 22 33 44 end              
"%J" format image[3,4]                                   

If any patterns specify names, the result of parse will be a dictionary, and format will likewise expect a dictionary as its right argument:

"%[one]i %[two]i" parse "34 56"                     
"%[one]i %[two]i" format ("one","two") dict 34,56   

The like operator offers another way of performing string matching, based on so-called glob patterns, which are a simplified subset of regular expressions. A Lil glob pattern is a string which may contain the special characters ., *, # or backticks. In a Lil glob pattern, a dot (.) matches any single character, an octothorpe (#) matches any single digit (0-9), a star (*) matches zero or more of any character, a backtick "escapes" a subsequent special character, treating it like a normal character, and any other character matches exactly itself. (You may find that SQL engines you're familiar with offer similar features with their own set of special characters.)

For example, we can use . and # to describe a "mask" of characters to accept within a broader pattern:

"Apple" like "A..le"                       
"(555)-867-5309" like "(###)-###-####"     

We can use backtick escapes when we need to match against a special character:

"2*3" like "#`*#"                          

And we can use * to perform prefix, infix, or postfix matches for fuzzy searching:

"The Best Orange" like "The*"      
"The Best Orange" like "*Best*"    
"The Best Orange" like "*Orange"   

If the left argument to like is a list of strings, it will produce a list of results indicating whether each string matched the pattern. This allows it to be used conveniently in queries:

t:("widget","plastic dingus","whatsit","extruded plastic dingus","dingus")

t like "*dingus"


select where value like "*dingus" from t

If the right argument to like is a list of patterns, the result will match if any of the provided patterns match:

t:("apple pie","key lime pie","banana cream pie","apple computer")

t like ("apple*","banana*")


select where value like ("apple*","banana*") from t

Lil, the Vector Language

Lil has a number of features influenced by "Vector-oriented" languages like APL, J, K, and Q. The essence of vector-oriented languages is thinking about manipulating entire data structures at once, rather than manipulating their elements serially.

The most obvious vector-oriented feature in Lil is conforming, in which a number of primitive operators like + and - can be applied either to single numbers or entire lists. This functionality is essential to how Lil manipulates columns within queries.

Let's start with a unary operator. Applied to a list, - "penetrates" to each list element:

-(5)                
-(10,-35)           

With a binary operator, a non-list item will "spread" and be paired with each element in a list. Given two lists, corresponding element from each list are "paired up":

100+10              
100+(10,20)         
(100,200)+10        
(100,200)+(10,20)   

The same pattern is carried out recursively; you can operate on arbitrarily high-dimensional data this way. Sometimes you may still need an each or flip to get your operands to line up the way you want:

2*(list 1,2,3),(list 3,4)


2 cross 3


each x in 2 cross 3 (27,19)+x end


flip(27,19)+flip 2 cross 3

Conforming is why Lil has two different equality operators: = (equals) conforms, and ~ (match) does not. Consider these cases:

22=11,22,33          
22~11,22,33          

(11,22,33)=11,22,33  
(11,22,33)~11,22,33  

()=11,22             
()~11,22             

In many situations, = and ~ are equivalent. Prefer ~ when you don't need conforming behavior; it signals your intent more clearly to a reader, since it is easy to tell without context that the result will be a single number.

If two lists do not have the same length, Lil will truncate or repeat the right argument (y) to correspond to the length of the left argument (x), as if by (count x) take y:

(11,22,33,44)+(100,200)               
(11,22,33,44)+(100,200,300,400,500)   
(100,200)+(11,22,33,44)               

This behavior can be useful for finding relative relationships in a list, comparing it to a "shifted" copy of itself:

v:1,2,2,5,3,6,7,7

(1 drop v)=v                          
(1 drop v)>v                          
(1 drop v)-v                          

If you're familiar with Q or K, you might recognize this pattern as similar to the applications of the eachprior adverb.

Another application is applying a "mask" pattern to an entire list:

(11,22,33,44,55)*(0,1)                
(11,22,33,44,55)*(1,0)                

Yet another useful consequence of this behavior is that enlisting the right argument when conforming two lists can be used to compute a matrix giving the cartesian product: every combination of the elements of the left and right lists. Compare this with cross, which produces a flat list of pairings:

(11,22,33) +      100,200             
(11,22,33) + list 100,200             
      (100,200) cross 11,22,33        
sum @ (100,200) cross 11,22,33        

You can also conform dictionaries. Lil takes the union of keys in the dictionaries and applies the operator between dictionary elements (or nil if the entry is missing). As when taking the union of dictionaries with ,, conforming "prefers" the order of keys in the left argument:

x:("White","Brown","Speckled") dict 10,34,27
y:("Brown","White","Blue"    ) dict  9,13,35

x+y  
y+x  

If you use a conforming operator between a dictionary and non-dictionary value, the non-dictionary value will be spread to every element of the dictionary and then conformed recursively as usual. Thus, if either argument is a dictionary, the result will be a dictionary:

d: ("Alpha","Beta") dict (list 5,7),(list 3)

d+100      
(10,20)*d  

Combining conforming operators with reducing operators like sum and raze offers many elegant and direct solutions to problems. Compare each of these approaches to counting how many times a value needle can be found in a list haystack:

needle:   "apple"
haystack: ("frog","apple","chicken","toadstool","apple","rice","fish")

c:0 each x in haystack if x~needle c:c+1 end end c     

c:0 each x in haystack c:c+x~needle end c              

count extract value where value=needle from haystack   

count needle take haystack                             

sum needle=haystack                                    

Transforming iterative code into parallel, vector-oriented algorithms makes Lil much faster and more efficient, and may result in simpler and clearer code. Consider the following two approaches for replacing values in an array that are less than 5 with the number "99":

each v in x                
 if v<5 99 else v end
end

m:x<5                      
(99*m) + x*!m              

The @ operator is another powerful tool. Given a data structure on the left and a list of indices on the right it picks the element at each index. This operation can be used to replicate, filter, or permute the elements of the source:

"ABC" @ 0,0,1,2,1,2,0


("AB" dict 11,22) @ "BAAB"

If the left argument is a function, it is applied to each element of the right argument, like a more concise each loop:

on triple x do x,x,x end

each x in 11,22,33 triple[x] end


triple @ 11,22,33

This also works if the "left argument" is a primitive unary operator, "pushing" the operator onto each element of a list or dictionary:

first "Cherry","Olive","Orange","Lime"


first @ "Cherry","Olive","Orange","Lime"


count ("Alpha","Beta") dict (list 11,22,33),(list 44,55)


count @ ("Alpha","Beta") dict (list 11,22,33),(list 44,55)

Lil, the Decker Language

Lil contains a number of language features and builtins specifically intended for interaction with the Decker environment.

The send statement is followed by the name of a function, and then a set of arguments in brackets, just as if the function were being called. Instead of calling the named function directly, send finds the next closest binding (or the most recently shadowed binding, if you like) for the name and calls that function. This is particularly useful for event handlers which need to "bubble" events up to a more general handler, like a card's link definition deferring to the deck's link definition.

The send statement can also be handy to override built-in functions like go[] with your own code, so that you have a chance to perform some work before, after, or instead of their default behavior. Defining these overrides in the deck's script will make them apply for events triggered from any card or widget:

on go x do
 
 if x~"Random"
  x:random[deck.cards]
 end

 
 send go[x]
end

An interface appears similar to a dictionary, but indexing or assignment through an interface may have side-effects, and the values in fields may change over time:

sys                  
"%e" format sys.now  
"%e" format sys.now  

time:sys             
time.now             

Interfaces cannot be defined from Lil programs- they are furnished by a host application like Decker. Consult Decker's manual for a description of the interfaces you can use in your scripts. Many "utility" interfaces offer considerably more efficient ways of performing operations than with raw Lil alone: of particular note are the Array interface, which provides a mutable data structure for manipulating and parsing binary data, the Bits interface, which provides functions for performing bitwise arithmetic, and the Image interface, which offers a variety of ways to manipulate two-dimensional arrays of bytes in bulk.

It is not possible to enumerate the keys of an interface; they may have infinitely many keys, populated on the fly. Thus, the in operator will always return 0 when an interface is its right argument, and the keys operator will always return ().

Interfaces can be compared with ~ and = using reference equality and concatenated into lists using , like any other datatype:

sys~sys              
sys=(sys,sys,123)    

All interfaces will accept the "type" index, which behaves like typeof. This is handy if you ever have a list, dictionary, or table containing interfaces:

sys.type             
(sys,sys,sys)..type  

Accessing an invalid index will return nil, and attempting to write to an invalid (or read-only) index will return the expression to the right, like any other indexed assignment:

sys.bogus            
sys.bogus:123        

Appendix 1: Unary Primitives

The unary arithmetic primitives - (negation), ! (logical not), floor, cos, sin, tan, exp (the exponential function), ln (natural log), and sqrt (square root) conform, and generalize to dictionaries, lists, and numbers.

The unary aggregation primitives sum, prod, raze, min, and max take a list and collapse it into a single result as if by combining every element of the list with the binary primitives +, *, ,, & and |, respectively.

The raze of a table x will convert it into a dictionary as if by x[(keys x)[0]] dict x[(keys x)[1]].

typeof gives the name of the type of the argument; one of the strings { "number", "string", "list", "dict", "table", "function", "nil" }, or, in the case of an interface, the name of that interface type.

count gives the number of elements in a value. The count of a number is always 1. The count of nil (per its listy interpretation) is always 0.

first and last pick the first or last elements of a value. The first of a function is the function's name.

range of a number produces a list of the integers [0,x). Applied to anything else, it will produce a list of the values of its dictionary equivalent.

keys produces a list of the keys of a value's dictionary equivalent. The keys of a function are a list of its argument names.

list of anything produces a list of length 1 containing that value.

flip will transpose the elements of a list of lists- swap the x and y axes. For example, flip (list 1,2,3),(list 4,5,6) is ((1,4),(2,5),(3,6)). The result will always be rectangular: sublists that are too short will be padded with nil, and any non-list values will be spread to take up an entire column in the result. Applied to a table x, flip will produce a new table using a column named key (or the first column) as new column keys, and all other columns will become rows. The original keys will become a column named key in the resulting table. This operation can be useful for "pivoting" categorical data.

rows converts a table into a list of dictionaries, each representing a row of the table.

cols converts a table into a dictionary of uniform-length lists, each representing a column of the table.

table constructs a table. Given a list-of-dictionaries (as produced by rows) or a dictionary, it will reconstitute a table, taking the union of any dictionary keys as columns and extending any short columns as with take. A list of lists will be interpreted as a list of rows, padded to rectangularity with nil if needed, using default column names {c0, c1... cn}. A list containing non-dictionary and non-list values will be converted into a table with a single value column.

mag computes the magnitude of a vector, or the euclidean distance between a point and the origin. If its argument is anything except a number or list of numbers it will descend recursively through the argument before computing a magnitude, similar to how other unary primitives conform. For example, mag ((list 9,0),(list 3,4),(list 0,7)) is (9,5,7). The magnitude of a single number is its absolute value.

heading computes the angle in radians from the origin toward a point. This is similar to the atan2(y,x) quandrant-disambiguating arctangent provided in many math libraries. Like mag, heading will descend recursively through any non list-of-number arguments, so it can be applied to points, lists of points, and so on.

unit computes a unit vector (with mag of 1) pointing toward an angle in radians. This operation conforms. If the angle is between 0 and 2 pi and the original mag was 1, unit is the inverse of heading.

Appendix 2: Binary Primitives

The binary arithmetic and comparison primitives +, -, *, /, % (modulus), ^ (exponentiation/power), <, >, =, & (minimum), and | (maximum) conform, and generalize to dictionaries, lists, and numbers.

The modulus operator % takes its arguments in the opposite order of common notation- the divisor is the left operand. Thus, 5 % 3,4,5,6,7 is (3,4,0,1,2). In common usage, this order will require fewer parentheses given Lil's right-to-left precedence rule.

The = comparison operator always produces the number 1 or 0 as a result. If either operand is an interface or a nil value, they are equal only if both are identical. If both arguments are numbers, they are compared numerically, and otherwise they are converted to strings and compared.

The < and > comparison operators always produce the number 1 or 0 as a result. If either operand is a number, they are compared numerically. Otherwise, the arguments are treated as strings and compared lexicographically. Note the differences between how these operators handle nil or mixed-type operands and the behavior of =!

The & and | operators calculate the minimum or maximum of their arguments. As a consequence, for the numbers 0 and 1 they are equivalent to logical "AND" and logical "OR"- thus their notation. Applied to strings, they compare values lexicographically in the same fashion as < and >.

x ~ y is the match operator. It produces the number 1 if x and y are identical values. Unlike =, match does not convert arguments or automatically "spread" to list elements; x and y must have identical types to begin with. This is particularly important if you want to e.g. check whether an item is the empty list: ()=1,2 yields (), but ()~1,2 yields 0.

x , y is the concatenation operator. It is used for joining items together into lists. If x is a dictionary, y will be converted to a dictionary, and the operator will take the union of their key-value mappings, preferring any bindings in y over x when both are present. If applied to two tables, their rows will be concatenated, with any missing columns supplied as nil. Note that using , on two strings will result in a list of two strings, whereas "" fuse x,y will concatenate the strings together.

x @ y is spread-indexing: x is indexed with each element of y. For example, (11,22,33) @ 0,1,0,1,0 is (11,22,11,22,11). With an appropriate index y, this operator can be used to reorder, duplicate, or filter elements of a list x. The expression x @ y is essentially equivalent to each v in y x[v] end, and so it can also be used as shorthand for any each loop that would otherwise simply be applying a function to each element of its source. This also works if x is a unary primitive: count @ "one","two","three","four" is (3,3,5,4).

x split y breaks a string y apart on any instance of the string x, resulting in a list of strings.

x fuse y conversely combines the strings in the list y with the string x, resulting in a string. Combining split and fuse can allow one to replace instances of one substring with another.

x like y returns a truthy result if a string x matches a glob pattern y. If y is a list of patterns, the result will be truthy if any of the patterns match. If x is a list, the pattern or patterns will be applied to each string of x and return a list of 0 or 1 values.

x dict y constructs a dictionary from a list of keys x and a list of values y. If y isn't long enough, trailing keys will be associated with nil: "ABC" dict 11,22" is {"A":11,"B":22,"C":nil}.

x take y and x drop y are very general operators for filtering and reshaping data. Their behaviors depend on the type of the left and right arguments, as summarized in the table below. When x is a number, take will repeat elements from y if it isn't long enough. For non-numeric x values, take can be thought of as set intersection, while drop can be thought of as set difference.

Note that if you represent sets as dictionaries, the take, drop and , operators can provide set intersection, disjunction, and union, respectively:

x:"AB" dict ()    
y:"BC" dict ()    

(keys x) take y  
(keys x) drop y  
x,y              

x limit y returns up to x items from y. Equivalent to if x<count y x take y else y end.

x window y divides a list or string y into slices of length mag x. If x is a negative number, the slices will overlap. For example, 3 window "ABCDEF" is ("ABC","DEF"), while -3 window "ABCDEF" is ("ABC","BCD","CDE","DEF"). If there aren't enough elements in y, a positive x may result in a final slice with fewer than x elements.

x in y returns the number 1 or 0 depending upon whether y appears in x. If y is a string, look for the string x anywhere in y. If y is a list, look for x as an element of y. If y is a dictionary, x must be a key of that dictionary. If y is a table, it must likewise be a key (column name) of that table. In all other cases, in returns 0. If x is a list, consider each element of x and return a list of 1 or 0.

x unless y returns x unless y is not nil; otherwise it returns y. This "nil-coalescing operator" is handy for providing default values when indexing into dictionaries or lists. For example, () unless foo[x] will evaluate to the empty list if foo contains no value with the key x.

x fill y recursively replaces nil elements within y with x. You can think of this as a more general version of unless in many cases; bulk-replacing missing or invalid elements from a list, dictionary, or table with an appropriate default value.

x join y performs a natural join upon the tables x and y. The intersection between column names in x and y are used as the matching key. For any combination of rows in x and y where the matching key is identical, the result will contain a row containing the union of the values in that row. The resulting table's column names will be the union of the names in x and y. Natural joins neatly capture foreign-key relationships between tables, provided that column names are consistent. If join is applied to non-table arguments, it produces a list by pairing adjacent elements from x and y. If either argument is a number, it is considered range of that argument, and otherwise it is interpreted as a list. For example, "ABC" join 3 gives (("A",0),("B",1),("C",2)).

x cross y performs a cartesian product (or cross join) of the tables x and y. Every row from x is concatenated with every row in y, producing (count x)*(count y) resulting rows. The resulting table will contain all the columns of both x and y, with any collisions in y gaining an underscore (_) suffix to disambiguate. A cross product filtered by a query can be used to perform inner joins. If cross is applied to non-table arguments, it produces a list of all pairings of elements from x and y. If either argument is a number, it is considered range of that argument, and otherwise it is interpreted as a list. Thus, 2 cross "ABC" gives ((0,"A"),(1,"A"),(0,"B"),(1,"B"),(0,"C"),(1,"C")).

x parse y breaks apart a string or list of strings y based on a format string x. x format y constructs a formatted string from a value or list of values y based on a format string y. The parse and format primitives are generally (but not strictly) inverses, given the same format string.

Appendix 3: Language Grammar

The following is a slightly hand-waved EBNF description of Lil's syntax. The production ALPHA is taken as any alphabetic character (upper- or lowercase), and the production NON_ESC is any DeckRoman character (except backslash). All whitespace is interchangeable in Lil, any amount of whitespace may appear between tokens, and line comments begin with #.

MONAD   := '-'|'!'|'floor'|'cos'|'sin'|'tan'|'exp'|'ln'|'sqrt'|'count'|'first'|'last'|'sum'|'min'|'max'|
           'raze'|'prod'|'range'|'keys'|'list'|'rows'|'cols'|'table'|'typeof'|'flip'|'mag'|'unit'|'heading'
DYAD    := '+'|'-'|'*'|'/'|'%'|'^'|'<'|'>'|'='|'&'|'|'|','|'~'|'@'|'split'|'fuse'|'dict'|'take'|'drop'|
           'in'|'join'|'cross'|'parse'|'format'|'unless'|'limit'|'like'|'window'|'fill'
DIGIT   := '0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9'
NUMBER  := '-'? DIGIT+ '.'? | DIGIT* '.' DIGIT+
STRING  := '"' (NON_ESC|'\\'|'\"'|'\n')* '"'
LITERAL := NUMBER | STRING | '(' ')'
NAME    := (ALPHA|'_'|'?') (ALPHA|'_'|'?'|DIGIT)*
ITER    := (( each' NAME* 'in' ) | 'while') EXPR* 'end'
ON      := 'on' ( '...' NAME |  NAME+ ) 'do' EXPR* 'end'
IF      := 'if' EXPR* ('elseif' EXPR*)* ( 'else' EXPR* )? 'end'
CLAUSE  := ('where' EXPR) | ('by' EXPR) | ('orderby' EXPR ('asc'|'desc'))
QUERY   := ('select'|'extract'|'update')((NAME|STRING ':')?EXPR)* CLAUSE* 'from' EXPR
INSERT  := 'insert' (NAME|STRING)* 'with' EXPR* (('into' EXPR) | 'end')
SEND    := 'send' NAME '[' EXPR* ']'
INDEX   := ('.' NAME? | '[' EXPR* ']')*
ACCESS  := NAME INDEX ( ':' EXPR )?
LOCAL   := 'local' NAME ':' EXPR
TERM    := LITERAL | ITER | ON | IF | QUERY | INSERT | SEND | '(' EXPR ')' | MONAD EXPR | ACCESS | LOCAL
EXPR    := TERM ( INDEX ( ':' EXPR )? | DYAD EXPR )?
PROGRAM := EXPR*