Skip to content

Collection

Collection

Collection(dims: Any, /, *columns: Any)

Bases: Generic[D, *Columns]

A dimensioned collection of constant, array and jagged values, in one or more uniquely named value columns.

Collection[SomeDims, FloatA[Literal["gross"]], FloatC[Literal["rate"]]] annotates a collection with its Dims class and its columns' types, in order; a set of columns named once as a tuple can be unpacked: Collection[SomeDims, *SomeColumns]. Operations keep what they don't change: arithmetic with a scalar or a collection of the same dims keeps the dims, the left operand's column name and the larger shape; a reduction keeps the columns. Where the result's dims or columns can't be known statically the type says so, and validate checks a collection against dims and column types and returns it typed by them.

Every column has a type: a kind - Float, Int, Bool, Date or String, stored at its native width - and a shape - Constant (one value per row), Array (an array of length values per row) or Jagged (per row, length entries each holding its own number of values). Columns of one collection can differ in type and length.

Binary operations align operands by dimension (name and kind), not position. Each operand must have a single value column - get_column isolates one, and nothing broadcasts across value columns; the result takes the left operand's column name (a float operand takes the collection's name). Every operation is strict: a dim row of the shared dims on one side only raises ValueError, never dropped or filled. extend_default adds missing dim rows, so collections covering different rows line up.

Operands must be of the same kind, as nothing is converted implicitly (a Python int literal does count as a float alongside a float column). + - * // apply to floats and ints, / to floats and ints (ints dividing to floats, as in Python), min/max to floats, ints and dates, < <= > >= to floats, ints and dates, and == != to every kind, the comparisons giving bool collections, as Polars' expressions do; zk.when picks between branches of any one kind. A collection has no truth value, so if a < b: raises, and since == compares values, a collection isn't hashable.

Shapes aren't converted either, but a constant (including a Python scalar) broadcasts over an array or jagged value. Arrays combine with arrays of the same length - an array of length 1 is not a constant - and an array applies to a jagged value of its length entry by entry. Jagged values combine value by value, so their counts must agree row by row.

A collection holds lineage, not values: each value cell is a node owned by the values that refer to it, and operations build nodes. An array or jagged cell is lazy - its values are computed only when it's collected: collect(), or zk.collect_all for several collections at once, evaluates into a zk.Collected, whose to_numpy, to_list and to_polars read them. A constant cell keeps its lineage too - a constant column summed across rows (premium.sum()) remembers the cells it came from, and explain shows them - but, being one number, its value is computed as soon as it's built. Lineage is freed with the values, so there is no context to reset. See correlate and explain.

A collection starts from its dim rows, Collection.of(dims, dim_rows), with no value columns, and with_columns adds them - computed from the dims row by row with zk.float or zk.int, or from other columns; data already in arrays is read by a function the same way. A collection is also grown in place a row at a time with Collection(dims, *columns) and add, for columns of any kind. zk.concat puts another collection's columns alongside. Values already derived from a collection keep the rows they were built from.

An empty collection over dims with the columns columns declare - column types such as FloatA[Literal["gross"]], in order - to fill a row at a time with add. An array or jagged column's length is set by the first row added.

row_count property

row_count: int

The number of rows: one per distinct dim row. len(collection).

dim_rows property

dim_rows: DimRows[D]

The collection's dim rows - its dims and each row's dim cells - without its values, to build new collections over the same rows: collection.dim_rows.zeros(FloatA[Literal["name"]], length=...). A snapshot of the rows when it is read: rows added to the collection afterwards aren't in it.

value_lengths property

value_lengths: list[int | None]

Each column's value_length(name), in order, with None for a constant column.

ndim property

ndim: int

The number of dims; 0 for a scalar.

is_constant property

is_constant: bool

Whether the single value column is constant (one value per row). Raises ValueError if there are several columns.

kinds property

kinds: list[type]

The value columns' kinds, in order: float, int, bool, datetime.date or str.

types property

types: list[type[Value]]

The value columns' types, in order: zk.FloatA, zk.StringC and so on.

column_names property

column_names: list[str]

The value columns' names, in order.

dims property

dims: list[Dim]

The dims, in row order: rows sort by the first, then the second, and so on.

add

add(*dim_row_and_values: Any, **by_name: Any) -> None

Add one row, in place: its dim row in dim order, then one value per column in column order - positionally, or by dim and column name. A constant column takes a scalar, an array column a 1-D array, and a jagged column a (counts, values) pair of 1-D arrays; every array or jagged value of a column has the same length. Any column also takes a Value read from a collection with get_constant, get_array or get_jagged, of its kind and shape: the value cell then holds that Value's node, so it keeps its lineage - a copula applied to what it was built from reorders it too. Raises ValueError at once for a dim row already in the collection or a value that doesn't fit its column, leaving the collection as it was.

extend_default

extend_default(
    dim_rows: Collection | DimRows | Sequence[object],
    *,
    float_default: float = 0.0,
    int_default: int = 0,
    bool_default: bool = False,
    date_default: date = min,
    string_default: str = "",
) -> Collection[D, *Columns]

This collection with a row of defaults for each dim row of dim_rows it doesn't already have; its own rows are kept as they are. dim_rows is another collection, or its dim_rows, over the same dims (in any order), or a sequence of dim rows in this collection's dim order (a bare dim cell for a collection with one dim). Each new cell is its column kind's default, as a constant, an array of it, or a jagged value of empty entries. Operations between collections are strict, so this is how to line up collections that cover different rows - e.g. a.extend_default(b) + b.extend_default(a).

of staticmethod

of(
    dims: type[Dims] | Sequence[Dim], dim_rows: _DimRows
) -> Collection[Any]

A collection over dims (a Dims class or a sequence of Dims) and dim_rows, with no value columns yet: with_columns adds them. dim_rows is a sequence of dim rows in dim order (a bare dim cell for one dim), or a polars DataFrame with a column per dim, whose other columns are ignored. Float dims can hold the parameters each row is sampled from, which zk.float passes a function::

class Parameters(zk.Dims):
    line = zk.Dim(str)
    mean = zk.Dim(float)
    sd = zk.Dim(float)

rng = np.random.default_rng(seed)
gross = zk.Collection.of(Parameters, parameters).with_columns(
    gross_ibnr=zk.float(rng.lognormal, zk.dim("mean"), zk.dim("sd"), nsims)
)

drop_dims("mean", "sd") then leaves the values by line alone. Data already in arrays is read by a function too, keyed by the dims: zk.float(lambda line: gross[line], zk.dim("line")).

validate

validate(dims: Any, /, *columns: Any) -> Any

Check the collection has exactly the dims of dims and the value columns columns declare - column types such as FloatA[Literal["gross"]], in any order - and return it typed by them, with its dims in dims's order (the same object if they already are). Raises ValueError on a mismatch.

get_column

get_column(
    name: str | type[ValueColumns],
) -> Collection[D, Value]

A single value column (never a dim column), as in Polars' get_column. It returns a collection holding just the value column the name class declares, which must have that type, alongside all of this collection's dim columns, sharing its dims and dim rows.

This is how to isolate one value column for arithmetic, which needs a single value column in each operand: gross.get_column(GrossIbnr) * rate.

select_values

select_values(
    *columns: str | type[ValueColumns],
) -> Collection[D, *tuple[Any, ...]]

The collection with just the value columns columns, in that order - each a name, or a column type such as FloatA[Literal["gross"]], whose type the column must have. The dims and dim rows are shared. Typed by the column types given; with names, the columns are typed as any.

drop_values

drop_values(
    *columns: str | type[ValueColumns],
) -> Collection[D, *tuple[Any, ...]]

The collection without the value columns columns - each a name, or a column type whose type the column must have - the rest in order. The dims and dim rows are shared, so dropping every value column leaves the dim rows alone. What's left can't be typed statically, so its columns are typed as any - validate restates them.

drop_dims

drop_dims(*dims: str | Dim) -> Collection[Any, *Columns]

The collection without the dims dims - each a name or a Dim - over the same values, its other dims in order. The dims dropped must be redundant, every row keeping a dim row of its own without them, as parameter dims are once the values sampled from them are built: sampled.drop_dims("mean", "sd"). Raises ValueError for a dim the collection doesn't have, or rows that only the dropped dims told apart - group_by combines those. The dims left aren't known statically - validate restates them.

with_columns

with_columns(
    *expressions: ValueExpr, **named: ValueExpr
) -> Collection[D, *tuple[Any, ...]]

This collection with columns computed from its own, as in Polars: a column per value expression - in expressions, named by its alias or the first column it refers to, or by keyword in named - computed row by row from this collection as it is before any is added: with_columns(net=zk.value("gross") * zk.value("rate")). A column named like an existing one replaces it in place, and the rest follow in order; no name may repeat. Another collection's columns are put alongside with zk.concat.

A value expression's type isn't known statically, so the columns are typed as any - validate restates them.

rename

rename(
    mapping: Mapping[str, str | type[ValueColumns]],
) -> Collection[D, *tuple[Any, ...]]

The collection with its value columns renamed by mapping, as in Polars: rename({"gross": "net"}). Each old name maps to its new one, a str or a column type such as FloatA[Literal["net"]], whose type the column must have. All are renamed at once, so names can swap; the columns keep their order. Operations keep the left operand's column name, so a derived collection is renamed before it's validated against its own column types - a single column renamed to a column type keeps its type statically.

is_empty

is_empty() -> bool

Whether the collection has no rows: row_count == 0.

value_length

value_length(name: str) -> int | None

The length of the value column name's Values - len() of each value cell: an array's number of values, or a jagged value's number of entries (its outer length) - or None if the column is constant. Raises ValueError if there's no such column. Typed by the column: an int for an array or jagged column, None for a constant one.

jagged_shape

jagged_shape(name: str) -> JaggedShape

The jagged column name's shape, row by row: its entries hold as many values as this column's do, however copulas move them - the counts themselves aren't known until the column is collected. Pass it to dim_rows.zeros(FloatJ[...], jagged_shape=...) for zeros that combine with this column value by value. A snapshot, like dim_rows. Raises ValueError if there's no such column, or it isn't jagged.

abs

abs() -> Collection[D, *Columns]

Every value's absolute value, in every column. Floats and ints only.

sqrt

sqrt() -> Collection[D, *Columns]

Every value's square root, in every column. Floats only: raises ValueError for any other kind.

exp

exp() -> Collection[D, *Columns]

e to the power of every value, in every column. Floats only: raises ValueError for any other kind.

log

log() -> Collection[D, *Columns]

Every value's natural log, in every column: -inf at 0 and NaN below. Floats only: raises ValueError for any other kind.

sum

sum() -> Collection[NoDims, *Columns]

Every column summed across all the rows, value by value: a collection with no dims and a single row, its columns keeping their names, kinds and shapes. Reduce to some of the dims with group_by(...).sum(). Jagged columns don't reduce across rows; reduce their entries first, e.g. with jagged_sum. An empty collection raises a ValueError.

mean

mean() -> Collection[NoDims, *Columns]

Every column's mean across all the rows; as for sum.

min

min() -> Collection[NoDims, *Columns]

Every column's minimum across all the rows; as for sum. For the elementwise minimum of two collections, use zk.min.

max

max() -> Collection[NoDims, *Columns]

Every column's maximum across all the rows; as for sum. For the elementwise maximum of two collections, use zk.max.

jagged_sum

jagged_sum(name: str) -> Collection[D, Value]

The sum of each entry's values in the jagged column name: a collection of that column alone, as arrays of the same length. An empty entry sums to 0.

jagged_min

jagged_min(name: str) -> Collection[D, Value]

The least of each entry's values in the jagged column name; as for jagged_sum. An empty entry gives 0.

jagged_max

jagged_max(name: str) -> Collection[D, Value]

The greatest of each entry's values in the jagged column name; as for jagged_sum. An empty entry gives 0.

jagged_count

jagged_count(name: str) -> Collection[D, Value]

How many values each entry of the jagged column name holds: a collection of that column alone, as int arrays of the same length.

clamp

clamp(
    lower: Collection[Any, Any] | Scalar,
    upper: Collection[Any, Any] | Scalar,
) -> Collection[Any, Any]

The values held between lower and upper, row by row and value by value; see zk.clamp.

filter

filter(*predicates: DimExpr) -> Collection[D, *Columns]

The rows where every one of predicates holds - conditions on their dim rows, such as zk.dim("class_of_business").is_in(["motor", "home"]) and zk.dim("underwriting_year") == 2026, implicitly and-ed together as in Polars - with every dim and column kept. Values are shared, not copied.

group_by

group_by(
    *by: DimExpr | Dim | str, **named_by: DimExpr | Dim
) -> GroupBy[Any, *Columns]

The collection grouped by by, for GroupBy.agg to reduce. Each of by - a dim expression such as zk.dim("class_of_business") or zk.when(zk.dim("year") < 2025).then(zk.dim("year")).otherwise(2025), a Dim, or a dim's name - becomes a dim of the result, named after the first dim it refers to (or its alias), or by keyword in named_by. Every other dim is reduced away::

collection.group_by(zk.dim("class_of_business")).agg(
    total_claims=zk.value("gross_reserves").sum(),
)

Grouped by nothing, every row is one group, and the result has no dims (NoDims): collection.sum() is collection.group_by().sum().

collect

collect() -> Collected[D, *Columns]

Evaluate the collection, with every copula registered so far applied, into a zk.Collected: its values as data, to read (to_numpy, get_array, ...) or convert (to_list, to_polars, to_pandas). A Collection is lazy: nothing else evaluates it. zk.collect_all evaluates several collections in one run.

get_constant

get_constant(
    dim_row: tuple[str | int, ...], name: str
) -> (
    FloatC[Any]
    | IntC[Any]
    | BoolC[Any]
    | DateC[Any]
    | StringC[Any]
)

The Value of the constant column name at dim_row - a tuple of the row's dim cells, () for a collection with no dims - as an instance of the column's type, such as FloatC["premium"], holding the value cell's node in the lineage. Raises ValueError if the collection has no such dim row or column, or if the column isn't constant.

Typed from the column's Literal name for the first 8 columns; a column further along gives any constant value.

get_array

get_array(
    dim_row: tuple[str | int, ...], name: str
) -> (
    FloatA[Any]
    | IntA[Any]
    | BoolA[Any]
    | DateA[Any]
    | StringA[Any]
)

The Value of the array column name at dim_row, as an instance of the column's type, such as FloatA["gross"], holding the value cell's node in the lineage. Raises ValueError as get_constant does.

get_jagged

get_jagged(
    dim_row: tuple[str | int, ...], name: str
) -> (
    FloatJ[Any]
    | IntJ[Any]
    | BoolJ[Any]
    | DateJ[Any]
    | StringJ[Any]
)

The Value of the jagged column name at dim_row, as an instance of the column's type, such as FloatJ["large"], holding the value cell's node in the lineage. Raises ValueError as get_constant does.

explain

explain() -> str

A readable account of how the values came to be, without computing them: the inputs and operations they depend on, the tie groups those inputs form, the copulas that reorder them (applied or not yet), and their formulas. Cells built by the same operations - a column computed the same way in every row - share one formula, written with input for the inputs (and c for constants that differ between cells), followed by one example cell in full::

formula, every cell: ((input + input) * 0.5)
  e.g. ('home',) = ((input 1 + input 2) * 0.5)

A collection of several columns explains each in turn.