GitHub - punkbrwstr/pynto: Time series analysis in Python using the concatenative paradigm

GitHub

11 min read Original article ↗

pynto logo

pynto: Data analysis in Python using stack-based programming

pynto is a Python package that lets you manipulate a data frame as a stack of columns, using the the expressiveness of the concatenative/stack-oriented paradigm.

How does it work?

With pynto you chain together functions called words to formally specify how to calculate each column of your data frame. The composed words can be lazily evaluated over any range of rows to create your data frame.

Words add, remove or modify columns. They can operate on the entire stack or be limited to certain columns using a column indexer. Composed words will operate in left-to-right order, with operators following their operands in postfix (Reverse Polish Notation) style. More complex operations can be specified using quotations, anonymous blocks of words that do not operate immediately, and combinators, higher-order words that control the execution of quotations.

What does it look like?

Here's a program to calculate deviations from moving average for each column in a table using the combinator/quotation pattern.

>>> import pynto as pt
>>> ma_dev = (                             # create a pynto expression by concatenating words
>>>     pt.load('stock_prices')            # append columns to stack from the built-in database
>>>     + ~(pt.dup + pt.ravg(20) + pt.sub) # quotation: copy column, calc moving avg, subtract
>>>     + pt.map                           # use the map combinator to apply the quotation
>>> )                                      # to each column in the stack
>>>
>>> df = ma_dev.rows['2021-06-01':]        # evaluate over a range of rows to get a DataFrame
>>> pt.db['stocks_ma_dev'] = df            # save the results back to the database

Why pynto?

  • Expressive: Pythonic syntax; Combinatory logic for modular, reusable code
  • Performant: Memoization to eliminate duplicate operations
  • Batteries included: Built-in time series database
  • Interoperable: Seamless integration with Pandas/numpy

Get pynto

Reference

The Basics

Composing words

Words are composed using the + operator. Each word resolves from the pt namespace.

>>> # Compose words that add a column of 10s to the stack, duplicate the column,
>>> # and then multiply the columns together
>>> ten_squared = pt.c(10) + pt.dup + pt.mul

Constant literals

Add constant-value columns to the stack using pt.c(value). You can also add constants directly with + using numeric literals. pt.r(n) adds whole number-value constant columns from 0 to n - 1.

>>> ten_squared = pt.c(10) + pt.dup + pt.mul    # using pt.c()
>>> ten_squared = 10 + pt.dup + pt.mul          # using numeric literal with +

Row indexers

To evaluate your expression, you use a row indexer. Specify rows by date range using the .rows[start:stop (exclusive):periodicity] syntax. None slicing arguments default to the widest range available. int indices also work with the .rows indexer. .first, and .last are included for convenience.

>>> ten_squared.rows['2021-06-01':'2021-06-03','B']                   # evaluate over a two business day date range
                 c
2021-06-01     100.0
2021-06-02     100.0

Quotations and Combinators

Combinators are higher-order functions that allow pynto to do more complicated things like branching and looping. Combinators operate on quotations, expressions that are pushed to the stack instead of operating on the stack. Use the ~ operator to create a quotation from a word or expression. The map combinator applies a quotation at the top of the stack over each column below in the stack.

>>> (pt.c(9) + pt.c(10) + ~(pt.dup + pt.mul) + pt.map).last
                 c         c
2021-06-02      81.0     100.0

~ supports single words, expressions, and nesting:

>>> ~pt.neg                                        # quote a single word
>>> ~(pt.dup + pt.mul)                             # quote an expression
>>> ~(~(pt.c(100) + pt.sub) + pt.call) + pt.map   # nested quotation
>>> ~~quoted_expr                                  # unquote (convert quotation back to word)

Headers

Each column has a string header. hset sets the header to a new value. Headers are useful for filtering or arranging columns.

>>> (pt.c(9) + pt.c(10) + ~(pt.dup + pt.mul) + pt.map + pt.hset('a','b')).last
                 a         b
2021-06-02      81.0     100.0

Column indexers

Column indexers specify the columns on which a word operates, overriding the word's default. Use .cols[indexer] to set the column indexer. Positive int indices start from the bottom (left) of the stack and negative indices start from the top.

By default add has a column indexer of [-2:]

>>> (pt.r5 + pt.add).last
              c    c    c    c
2021-06-02  0.0  1.0  2.0  7.0

Change the column indexer of add to [:] to sum all columns

>>> (pt.r5 + pt.add.cols[:]).last
               c
2025-06-02  10.0

You can also index columns by header, using regular expressions

>>> (pt.r3 + pt.hset('a,b,c') + pt.add.cols['(a|c)']).last
              b    a
2025-06-02  1.0  2.0

Use .copy to keep the original columns and .discard to remove non-selected columns:

>>> (pt.r5 + pt.pull.cols[2:4].copy).values[0]     # copy selected, keep originals
>>> (pt.r5 + pt.pull.cols[2:4].discard).values[0]   # move selected, discard rest
>>> (pt.r5 + pt.pull.copy).values[0]                # copy with default selector

Defining words

Words are composed using the + operator.

>>> squared = pt.dup + pt.mul
>>> ten_squared2 = pt.c(10) + squared    # same thing

Words can also be defined globally in the pynto vocabulary.

>>> pt.define['squared'] = pt.dup + pt.mul

The Database

pynto has built-in database functionality that lets you save DataFrames and Series to a Redis database. The database saves the underlying numpy data in native byte format for zero-copy retrieval. Each DataFrame column is saved as an independent key and can be retrieved or updated on its own. The database also supports three-dimensional frames that have a two-level MultiIndex.

>>> pt.db['my_df'] = expr.rows['2021-06-01':'2021-06-03']
>>> pt.load('my_df').rows[:]
              constant  constant
2021-06-01      81.0     100.0
2021-06-02      81.0     100.0

pynto built-in vocabulary

Column Creation

Word Description Parameters Column Indexer
c Pushes constant columns for each of values [-1:0]
day_count Pushes a column with the number of days in the period [-1:0]
from_pandas Pushes columns from Pandas DataFrame or Series pandas pandas: pd.DataFrame | pd.Series, round_: bool = False [-1:0]
load Pushes columns saved to internal DB as key key: str [-1:0]
nan Pushes a constant nan-valued column [-1:0]
period_ordinal Pushes a column with the period ordinal [-1:0]
r Pushes constant columns for each whole number from 0 to n - 1 n: int [-1:0]
randn Pushes a column with values from a random normal distribution [-1:0]
timestamp Pushes a column with the timestamp of the end of the period [-1:0]
year_frac Pushes a column with the actual/actual year fraction of the period [-1:0]

Combinators

Word Description Parameters Column Indexer
call Applies quotation [:]
call0 Shortcut for applying quotation without consuming anything from stack [-1:0]
cleave Applies all preceding quotations num_quotations: int = -1 [:]
compose Combines quotations num_quotations: int = 2 [:]
hmap Applies quotation to stacks created grouping columns by header [:]
ifexists Applies quotation if stack has at least count columns count: int = 1 [:]
ifexistselse Applies top quotation if stack has at least count columns, otherwise applies second quotation count: int = 1 [:]
ifheaders Applies top quotation if list of column headers fulfills predicate predicate: Callable[[list[str]], bool] [:]
ifheaderselse Applies quotation if list of column headers fulfills predicate, otherwise applies second quotation predicate: Callable[[list[str]], bool] [:]
map Applies quotation in groups of every every: int | None = None [:]
partial Pushes stack columns to the front of quotation [-1:]
repeat Applies quotation times times times: int = 2 [:]

Cumulative

Word Description Parameters Column Indexer
cadd Addition [-1:]
cavg Arithmetic average [-1:]
cdif Lagged difference [-1:]
clag Lag [-1:]
cmax Maximum [-1:]
cmin Minimum [-1:]
cmul Multiplication [-1:]
cret Lagged return [-1:]
cstd Standard deviation [-1:]
csub Subtraction [-1:]
cvar Variance [-1:]

Data cleanup

Word Description Parameters Column Indexer
ffill Fills nans with previous values, looking back lookback before range and leaving trailing nans unless not leave_end lookback: int = 10, leave_end: bool = True [:]
fill Fills nans with value value: float [:]
fillfirst Fills first row with previous non-nan value, looking back lookback before range lookback: int = 5 [-1:]
join Joins two columns at date date: datelike [-2:]
sync Align available data by setting all values to NaN when any values is NaN [:]
zero_first Changes first value to zero [-1:]
zero_to_na Changes zeros to nans [-1:]

Header manipulation

Word Description Parameters Column Indexer
halpha Set headers to alphabetical values [:]
happly Apply header_func to headers_ header_func: Callable[[str], str] [:]
hformat Apply format_spec to headers format_spec: str [:]
hindex Slice headers from start to stop start: int | None, stop: int | None [:]
hreplace Replace old with new in headers; set regex=True for regex replacement old: str | re.Pattern, new: str = '', regex: bool = False, flags: int = 0 [:]
hset Set headers to *headers [:]
hsetall Set headers to *headers repeating, if necessary [:]

One-for-one functions

Word Description Parameters Column Indexer
abs Absolute value [-1:]
dec Decrement [-1:]
div_last Divides all columns by the final column [:]
sub_last Subtracts the final column from all preceding columns [:]
exp Exponential [-1:]
expm1 Exponential minus one [-1:]
inc Increment [-1:]
inv Multiplicative inverse [-1:]
lnot Logical not [-1:]
log Natural log [-1:]
log1p Natural log of increment [-1:]
neg Additive inverse [-1:]
ntile Row-wise buckets bucket_count: int = 2 [:]
percentile Row-wise percentile rank [:]
rank Row-wise rank [:]
sign Sign [-1:]
sqrt Square root [-1:]

Quotation

Word Description Parameters Column Indexer
q Wraps the following words until p as a quotation, or wraps quoted expression as a quotation [-1:0]

Resample methods

Word Description Parameters Column Indexer
resample_avg Sets periodicity resampling method to avg [:]
resample_first Sets periodicity resampling method to first [:]
resample_firstnofill Sets periodicity resampling method to first [:]
resample_last Sets periodicity resampling method to last [:]
resample_lastnofill Sets periodicity resampling method to last with no fill [:]
resample_max Sets periodicity resampling method to max [:]
resample_min Sets periodicity resampling method to min [:]
resample_sum Sets periodicity resampling method to sum [:]
set_periodicity Changes column periodicity to periodicity, then resamples periodicity: str | Periodicity [-1:]
set_start Changes period start to start, then resamples start: datelike | int [-1:]

Reverse Cumulative

Word Description Parameters Column Indexer
rcadd Addition [-1:]
rcavg Arithmetic average [-1:]
rcdif Lagged difference [-1:]
rclag Lag [-1:]
rcmax Maximum [-1:]
rcmin Minimum [-1:]
rcmul Multiplication [-1:]
rcret Lagged return [-1:]
rcstd Standard deviation [-1:]
rcsub Subtraction [-1:]
rcvar Variance [-1:]

Rolling Window

Word Description Parameters Column Indexer
ewm_mean Exponentially-weighted moving average window: int = 2 [-1:]
ewm_std Exponentially-weighted standard deviation window: int = 2 [-1:]
ewm_var Exponentially-weighted variance window: int = 2 [-1:]
radd Addition window: int = 2 [-1:]
ravg Arithmetic average window: int = 2 [-1:]
rcor Correlation window: int = 2 [-2:]
rcov Covariance window: int = 2 [-2:]
rdif Lagged difference window: int = 2 [-1:]
rlag Lag window: int = 2 [-1:]
rmax Maximum window: int = 2 [-1:]
rmed Median window: int = 2 [-1:]
rmin Minimum window: int = 2 [-1:]
rret Lagged return window: int = 2 [-1:]
rstd Standard deviation window: int = 2 [-1:]
rvar Variance window: int = 2 [-1:]
rzsc Z-score window: int = 2 [-1:]

Row-wise Comparison

Word Description Parameters Column Indexer
equal Equal to ignore_nans: bool = False [-2:]
greater Greater than ignore_nans: bool = False [-2:]
greater_equal Greater than or equal to ignore_nans: bool = False [-2:]
less Less than ignore_nans: bool = False [-2:]
less_equal Less than or equal to ignore_nans: bool = False [-2:]
not_equal Not equal to ignore_nans: bool = False [-2:]

Row-wise Reduction

Word Description Parameters Column Indexer
add Addition ignore_nans: bool = False [-2:]
avg Arithmetic average ignore_nans: bool = False [-2:]
coalesce Returns the first non-nan value across selected columns for each row [:]
div Division ignore_nans: bool = False [-2:]
max Maximum ignore_nans: bool = False [-2:]
med Median ignore_nans: bool = False [-2:]
min Minimum ignore_nans: bool = False [-2:]
mod Modulo ignore_nans: bool = False [-2:]
mul Multiplication ignore_nans: bool = False [-2:]
pow Power ignore_nans: bool = False [-2:]
std Standard deviation ignore_nans: bool = False [-2:]
sub Subtraction ignore_nans: bool = False [-2:]
var Variance ignore_nans: bool = False [-2:]

Row-wise Reduction Ignoring NaNs

Word Description Parameters Column Indexer
nadd Addition ignore_nans: bool = False [-2:]
navg Arithmetic average ignore_nans: bool = False [-2:]
ndiv Division ignore_nans: bool = False [-2:]
nmax Maximum ignore_nans: bool = False [-2:]
nmed Median ignore_nans: bool = False [-2:]
nmin Minimum ignore_nans: bool = False [-2:]
nmod Modulo ignore_nans: bool = False [-2:]
nmul Multiplication ignore_nans: bool = False [-2:]
npow Power ignore_nans: bool = False [-2:]
nstd Standard deviation ignore_nans: bool = False [-2:]
nsub Subtraction ignore_nans: bool = False [-2:]
nvar Variance ignore_nans: bool = False [-2:]

Stack Manipulation

Word Description Parameters Column Indexer
drop Removes selected columns [-1:]
dup Duplicates columns [-1:]
hsort Sorts columns by header [:]
id Identity/no-op [:]
interleave Divides columns in parts groups and interleaves the groups parts: int = 2 [:]
keep Removes non-selected columns [:]
nip Removes non-selected columns, defaulting selection to top [-1:]
pull Brings selected columns to the top [:]
rev Reverses the order of selected columns [:]
roll Permutes selected columns [:]
swap Swaps top and bottom selected columns [-2:]