import zeki as zk

Every part of zeki's Python API as a railroad diagram. Follow a track from left to right: any path through it is valid code. Click a square box to jump to its own diagram.

code, typed as shown
another diagram
a value of your own
what a step gives
KindConstantArrayJagged
floatFloatCFloatAFloatJ
intIntCIntAIntJ
boolBoolCBoolABoolJ
datetime.dateDateCDateADateJ
strStringCStringAStringJ

Overview

A model builds collections of simulated values, combines them lazily, correlates them with copulas, and reads the results. Nothing is computed until a value is read.

Schemas

A collection's type lists its dims and then its value columns, so an annotation says what a variable holds: Collection[ReserveDims, FloatA[Literal["gross_ibnr"]], FloatC[Literal["payment_pattern"]]].

column-type

Every value column has a kind and a shape. Nothing converts between kinds or shapes implicitly (an array of length 1 is not a constant), but a constant broadcasts over the arrays and jagged values it meets. The name is a Literal so type checkers see it; at runtime zk.FloatA["gross"] gives the same class.

zk. Float Int Bool Date String C constant: one value per row A array: length values per row J jagged: length entries, each of any size [Literal[ "column" ]]

dims

A collection's dims: a Dims class, NoDims for a collection reduced to a single row, or a list of Dims.

DimsClassDimsClass zk.NoDims no dims: one row [ dimdim , ]

dims-class

Dims shared by many collections are declared once, as a class; each Dim takes its attribute's name. A Dims class extends its bases' dims. Iterating over the class gives its Dims in order.

class DimsClass (zk.Dims): dim_name = zk.Dim( str int float )

dim

One named axis, with str, int or float dim cells. Two dims are equal when their names and kinds are. A float dim holds the parameters a row's values are sampled from, never NaN.

zk.Dim( "name" , str int float ) .name .kind .is_int .is_float

value-columns-class

Several value columns declared once, wherever a column type is accepted at runtime; each ValueColumn takes its attribute's name. In annotations, a tuple of column types does the same job: Collection[ReserveDims, *Reserves].

class Columns (zk.ValueColumns): column_name = zk.ValueColumn( zk.FloatA, … )

Building collections

A collection starts from its dim rows with Collection.of and gains columns with with_columns, is grown a row at a time, or is derived from other collections. Its rows are sorted by dim row, and each array or jagged value cell is a node in the lineage.

dim-row-data

One dim row per row: a tuple of its dim cells in dim order, or a bare dim cell when there is one dim - or a polars DataFrame with a column per dim.

[ ( dim cell , ) dim cell one dim , ]

filling

Growing a collection in place. add takes one row: its dim cells in dim order, then a value per column in column order, or each by name. A dim row already present, or a value that doesn't fit its column, raises ValueError and leaves the collection as it was.

collection .add( dim cell , , cell-valuecell-value name = dim cell cell-valuecell-value , )

cell-value

What one value cell holds. A Value read from a collection keeps its lineage, so a copula applied to what it was built from reorders it too.

scalar constant column 1-D array array column ( counts , values ) jagged column valuevalue value-operationvalue-operation

value

One value cell's Value, read without computing it: an instance of the column's type, such as FloatC["premium"], holding the cell's node. Raises ValueError if the dim row or column is missing, or the column has another shape.

collectioncollection .get_constant( → FloatC, IntC, … .get_array( → FloatA, IntA, … .get_jagged( → FloatJ, IntJ, … ( dim cell , ) , "column" )

value-operation

Values combine as columns do, with Values of their kind and with scalars, giving another Value. Shapes broadcast as for columns, and the result is named after the left operand unless it's the scalar.

valuevalue + - * / // valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar arithmetic valuevalue < <= > >= == != valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar compare → bool zk.min( zk.max( valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar , valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar valuevaluevaluevaluevaluevaluevaluevalue scalarscalarscalarscalar ) elementwise

Operations

Binary operations align their operands by dim (name and kind), not position, and each operand has a single column. The result takes the left operand's column name. Every operation is strict: a dim row of the shared dims on one side only raises, and extend_default adds missing ones. A NoDims collection broadcasts to every row.

operation

What can follow a collection to give a new one. Arithmetic works on floats and ints (ints divide to floats, and // rounds down), < <= > >= on floats, ints and dates, and == and != on every kind. A comparison gives a bool collection, which has no truth value: if a < b: raises. ** is for floats. -collection negates every value. Reductions keep every column; jagged columns reduce their entries before they reduce across rows.

+ - * / // ** operandoperand arithmetic << <=<= >> >=>= ==== !=!= << <=<= >> >=>= ==== !=!= operandoperand compare → bool .clamp( operandoperand , operandoperand ) hold between bounds .abs() .sqrt() .exp() .log() floats every value .sum() .mean() .min() .max() reduce every row → NoDims .jagged_sum( .jagged_min( .jagged_max( .jagged_count( name ) reduce a jagged column's entries → array .filter( dim-exprdim-expr , ) keep the rows where all hold group-bygroup-by group rows by their dim rows .extend_default( collectioncollection dim rows , float_default= scalar and int_, bool_, date_, string_ ) add a row of defaults per missing dim row .get_column( "column" column-typecolumn-type ) .rename({ "old" : "new" column-typecolumn-type , }) .select_values( .drop_values( "column" column-typecolumn-type , ) .drop_dims( "dim" Dim , ) .with_columns( value-exprvalue-expr name = value-exprvalue-expr , ) .validate( dimsdims , column-typecolumn-type ) pick, rename, add or check columns

operand

The other side of an operation. A Python int counts as a float alongside a float column; otherwise operands are of one kind.

group-by

Each expression grouped by becomes a dim of the result, named after the dim it refers to, its alias, or its keyword; every other dim is reduced away. Grouped by nothing, the result has no dims: collection.sum() is group_by().sum().

.group_by( dim-exprdim-expr "dim" name = dim-exprdim-expr , ) .agg( name = value-exprvalue-expr , ) .sum() .mean() .min() .max()

dim-expr

An expression over a collection's dim cells, resolved only when it's used: a condition filters a collection, and one giving str or int dim cells groups it. Python's and, or, not and x if c else y raise TypeError; use &, |, ~ and zk.when.

zk.dim( "dim" ) ~ dim-exprdim-expr zk.when( dim-exprdim-expr , ).then( dim-operanddim-operand ) .when( dim-exprdim-expr , ).then( dim-operanddim-operand ) .otherwise( dim-operanddim-operand ) == != < <= > >= dim-operanddim-operand compare .is_in([ dim cell , ]) .is_between( dim-operanddim-operand , dim-operanddim-operand ) test & | dim-operanddim-operand combine conditions + - * // dim-operanddim-operand arithmetic .replace( mapping ) same kind .replace_strict( mapping , default ) map dim cells .alias( "name" ) name the group-by dim

call

A column computed by calling a function once per row, in with_columns: it returns a scalar, a 1-D array or a (counts, values) pair, the same shape every row. Each argument is resolved for the row: a dim expression gives the row's dim cell, a value expression of a constant column the row's value, and anything else is passed as it is.

zk.float( zk.int( function , keyword = dim-exprdim-expr value-exprvalue-expr object )

value-expr

An expression over value columns, for with_columns and GroupBy.agg. Every column of an agg must reduce across the group's rows; reduce both sides of an operation, or neither.

zk.value( "column" column-typecolumn-type ) callcall - value-exprvalue-expr value-whenvalue-when + - * / // value-operandvalue-operand arithmetic == != < <= > >= value-operandvalue-operand compare, for zk.when .clamp( lower , upper ) hold between bounds .pad_end( length , fill= scalar ) extend each array to a longer length .jagged_sum() .jagged_min() .jagged_max() .jagged_count() reduce each jagged entry .sum() .mean() .min() .max() .count() → int reduce the group's rows .alias( "column" ) name the column

Reading

A collection is lazy: describing it computes nothing. collect() - or zk.collect_all for several at once, in one pass of the engine - evaluates it, with every copula registered so far applied, into a zk.Collected holding the values as data.

collect

A Collected has the collection's dims, rows and column types, and no arithmetic. It is a snapshot: a copula registered afterwards doesn't change it.

collectioncollection .collect() zk.collect_all( collectioncollection , ) → tuple, one each → zk.Collected

read

to_polars gives a column per dim and then per value column: the kind's dtype for a constant, pl.Array for an array and pl.Array of pl.List for a jagged column. A dim and value column sharing a name raise ValueError. to_pandas takes a single constant or array column. A jagged column reads as a (counts, values) pair.

collectedcollected .to_numpy( name ) → ndarray, or (counts, values) .to_polars() → pl.DataFrame .to_list() → [(dim row dict, value row dict)] .to_pandas() → Series or DataFrame .get_constant( .get_array( .get_jagged( dim row , name ) → value, ndarray, or (counts, values) .dims .column_names .types .value_lengths len( collectedcollected ) .row_count structure

describe

A collection's structure is always known without computing anything. explain() shows a value's lineage, including the copulas it goes through and whether they have been applied yet.

collectioncollection .explain() → str: inputs, ties, copulas, formulas .dims .column_names .types .kinds .value_lengths .value_length( name ) len(collection) is .row_count .row_count .ndim .is_constant one column .is_empty() .jagged_shape( name ) → JaggedShape, not the counts describe, without computing valuevalue

dim-rows

A collection's dim rows without its values: a snapshot that rows added later don't change. Its zeros build new collections over the same rows.

collectioncollection .dim_rows .to_list() → [tuple] .dims .row_count .zeros( see collection

Dependency

A copula reorders a float array collection's cells against each other. Every input tied to a cell is reordered by its permutation, so values derived from them, before or after, stay consistent.

correlate

Correlates the cells of a FloatA column, lazily: the copula is applied when something depending on it is collected. Without a seed, one is derived from the context's seed and the cells' lineage, so results don't depend on statement order. Cells that already share an input can't be correlated. Warns with zk.StaleReadWarning if values already collected depend on the inputs it will reorder.

zk.correlate( collectioncollection , copulacopula , seed= int )

copula

Elliptical copulas take a function giving the correlation between two cells, from their rows' dim rows as tuples: called once per pair, giving a positive-definite matrix with values in [-1, 1]. Archimedean copulas take one theta for every pair and give positive dependence only.

zk.Copula. gaussian( correlations ) student_t( correlations , degrees_of_freedom ) clayton( θ ≥ 0, lower tail gumbel( θ ≥ 1, upper tail frank( θ ≥ 0, no tail joe( θ ≥ 1, upper tail ali_mikhail_haq( 0 ≤ θ < 1, weak theta )

context

The ambient settings a model is built under; only the root seed copula seeds derive from. Contexts nest, a setting left out being inherited, and are local to each thread and asyncio task. The default seed is 0.

with zk.Context( seed= int ): model code zk.current_context() .seed

Engine settings

Process-wide settings and counters.

settings

timings splits zeki's time into evaluate, copula, export and import, each exclusive of the others; building lazy operations and the caller's own code aren't included.

zk.set_num_threads( int ) zk.num_threads() zk.set_cache_limit( bytes ) 0 turns caching off zk.clear_cache() zk.cache_info() → entries, bytes, limit, hits, misses zk.timings() → seconds per activity zk.timing_counts() → starts per activity zk.reset_timings()

The engine

The DAG underneath the collections, built directly for float arithmetic in zeki.engine (from zeki import engine): float operations over arrays of one length, compiled once and evaluated in one parallel pass. It skips the collections' lineage, dims and kinds, so a large program of plain float arithmetic builds and evaluates with the least overhead. Most models want the collections.

graph

A Graph holds arrays of one length. Inputs are copied in once, so the compiled program owns its data. Compiling freezes the outputs into one program; the graph stays usable, so it can be compiled again for other outputs.

engine.Graph( length ) .input( 1-D float64 array ) → expr, length values .constant( float ) → expr, the value everywhere .compile( exprexpr [ exprexpr , ] ) → compiled .length .num_nodes including any no output reaches structure

compiled

Evaluates every output in one pass. Compiled from one Expr, eval gives a 1-D array; from a list, one row per output.

graph.compile(…)graph.compile(…) .eval() → ndarray, on every core .eval_serial() → ndarray, on one thread .length .num_outputs structure