Skip to content

Dependency and copulas

A copula makes simulated values dependent by reordering their simulations. zeki applies copulas for you, and keeps track of everything built from the values a copula reorders, so a model stays consistent however it's written.

Correlating a collection

zk.correlate(collection, copula) correlates the rows of a float array collection: each row's simulations are reordered so the rows have the dependency the copula describes. Elliptical copulas take a function giving the target correlation between two rows, from their dimension rows:

gross = sample(1)

correlations = {
    frozenset({"home", "motor"}): 0.3,
    frozenset({"home", "property"}): 0.6,
    frozenset({"motor", "property"}): 0.2,
}


def correlation(row_a: tuple, row_b: tuple) -> float:
    """The target correlation between two rows, given their dim rows."""
    if row_a == row_b:
        return 1.0
    return correlations[frozenset({row_a[0], row_b[0]})]


zk.correlate(gross, zk.Copula.gaussian(correlation))

The function is called once per pair of rows, and the correlations must form a positive-definite matrix. zeki reorders the simulations with the Iman-Conover method, so each row keeps exactly the values it had, only in a new order.

Everything built from the values follows

zk.correlate is lazy, like the rest of zeki. It registers the copula on the inputs the collection depends on: the arrays you added, or that zk.float returned. When anything depending on those inputs is collected, they are reordered first, so every value built from them, before or after the copula was registered, reads the reordered simulations:

gross = sample(2)
net = gross * 0.8  # built before the copula is registered
zk.correlate(gross, zk.Copula.gaussian(correlation))
recovery = (gross - 30_000.0).clamp(0.0, 50_000.0)  # built after it

gross_values, net_values = zk.collect_all(gross, net)
# net is still gross * 0.8, simulation by simulation: both read the reordered inputs

This is how sim-tying behaves in currently available commercial capital modelling products, without you keeping track of which values are tied to which.

Operations tie their operands

Once c = a + b exists, simulation i of c is simulation i of a plus simulation i of b. If a copula later moved a but not b, c would silently stop being a + b. So an operation ties its operands' inputs as it's built: a copula that reorders one reorders the other by the same permutation.

It follows that two values that already depend on a common input can't be correlated with each other, since no reordering could move one without the other:

pair = sample(3)
home = pair.get_array(("home", 2026), "gross")
motor = pair.get_array(("motor", 2026), "gross")
combined = home + motor  # ties home's inputs to motor's: combined stays home + motor

with pytest.raises(ValueError, match="common input"):
    zk.correlate(pair, zk.Copula.gaussian(correlation))  # home and motor can no longer move independently

A reduction across rows, such as sum() or a group-by, ties the rows it combines in the same way:

rows = sample(4)
total = rows.sum()  # a reduction across rows ties every row's inputs too
with pytest.raises(ValueError, match="common input"):
    zk.correlate(rows, zk.Copula.gaussian(correlation))

So correlate rows before combining them. Ties last as long as the inputs do, not as long as the value that created them: deleting c doesn't untie a and b, because values computed from c may still exist.

Nested copulas

Because a copula ties its targets too, copulas compose. A common pattern correlates classes within each underwriting year, then correlates the years' totals. A year's total ties its class rows, so the second copula moves them together, keeping the correlations the first gave them:

rng = np.random.default_rng(9)
reserves = zk.Collection(Business, Gross)
for class_of_business in ["home", "motor", "property"]:
    for year in [2024, 2025, 2026]:
        reserves.add(class_of_business, year, rng.lognormal(10.0, 0.5, SIMULATIONS))

# 1. Within each year, correlate the classes.
for year in [2024, 2025, 2026]:
    zk.correlate(reserves.filter(zk.dim("underwriting_year") == year), zk.Copula.gaussian(correlation))

# 2. Correlate the years' totals. A year's total ties its class rows, so this copula moves them
#    together, keeping the correlations step 1 gave them.
by_year = reserves.group_by(zk.dim("underwriting_year")).sum()
zk.correlate(by_year, zk.Copula.gaussian(lambda year_a, year_b: 1.0 if year_a == year_b else 0.4))

Copula families

zk.Copula.gaussian(correlation)
zk.Copula.student_t(correlation, degrees_of_freedom=4.0)  # tail dependence
zk.Copula.clayton(theta=2.0)  # lower-tail dependence
zk.Copula.gumbel(theta=1.5)  # upper-tail dependence
zk.Copula.frank(theta=5.0)  # no tail dependence
zk.Copula.joe(theta=1.5)  # upper-tail, stronger than Gumbel for the same Kendall's tau
zk.Copula.ali_mikhail_haq(theta=0.5)  # weak dependence only
Copula Parameters Dependence
gaussian A correlation function No tail dependence
student_t A correlation function, degrees_of_freedom Tail dependence, growing as degrees_of_freedom falls
clayton theta >= 0 (0 is independence) Lower tail. Kendall's tau is theta / (theta + 2).
gumbel theta >= 1 (1 is independence) Upper tail. Kendall's tau is 1 - 1 / theta.
frank theta >= 0 (0 is independence) No tail dependence
joe theta >= 1 (1 is independence) Upper tail, stronger than Gumbel's for the same Kendall's tau
ali_mikhail_haq 0 <= theta < 1 (0 is independence) Weak: Kendall's tau below 1/3

The Archimedean copulas (Clayton, Gumbel, Frank, Joe and Ali-Mikhail-Haq) take one theta for every pair of rows, and cover positive dependence only.

Seeds and reproducibility

A copula's random draws come from a seed. Without one, zeki derives it from the root seed of the current zk.Context and the lineage of the values being correlated, so a model gives the same results however its statements are ordered. Set the root seed with zk.Context, or pass a seed directly:

with zk.Context(seed=42):
    gross = sample(6)
    zk.correlate(gross, zk.Copula.gumbel(theta=1.5))  # seed derived from 42 and gross's lineage

zk.correlate(sample(7), zk.Copula.clayton(theta=2.0), seed=1234)  # or an explicit seed

zk.Context is held per thread and per async task.

Seeing what a value depends on

collection.explain() describes how a collection's values came to be, without computing them: the inputs and operations they depend on, the groups those inputs are tied into, and the copulas that reorder them, applied or not yet:

gross = sample(8)
net = (gross * 0.8).filter(zk.dim("class_of_business") == "motor")
zk.correlate(gross, zk.Copula.gaussian(correlation))
print(net.explain())
1 cell over 1 input and 1 operation
tie groups: 1 (inputs per group: 1)
copulas, oldest first:
  1. Gaussian over 3 cells, seed 8638896654103472097: reorders 1 input of these (not yet applied)
formula, the cell: (input * 0.8)
  e.g. ('motor', 2026) = (input 1 * 0.8)

Cells built the same way, such as a column computed identically in every row, share one formula.

Why there's no global context

Every lazy system has to decide where its graph, and side effects such as copulas, live. There are roughly three answers:

Approach Used by What happened
One global graph, plus a reset function TensorFlow 1 (tf.get_default_graph(), tf.reset_default_graph()) The most disliked part of TensorFlow 1: stale nodes in notebooks, confusing errors and trouble with threads. TensorFlow 2 dropped it.
A scoped context entered with with PyMC (with pm.Model():), decimal.localcontext, numpy.errstate, and the model files of commercial capital modelling products Fine for settings that really do apply to everything in scope; clumsy when every operation has to find "the" context.
The graph lives in the values PyTorch autograd (tensor.grad_fn), Dask, Polars' LazyFrame, JAX No reset needed: lineage is freed when the last value referring to it is. Works with threads, and each value is self-contained.

zeki takes the third approach. A copula's question, "if this input is reordered, what else must move with it?", depends only on the input, so the state lives on the inputs rather than in a context. Values own their lineage, so in a notebook, rebinding or deleting a variable frees what only it used, and there's nothing to reset. Only settings that genuinely apply to everything, such as the root seed, live in a context: zk.Context.

Caching across reruns is solved separately, as Dask, Bazel and Nix solve it: by keying results on a hash of how they're computed rather than on object identity. Rebuilding the same model gives the same keys, so a rerun reuses earlier results without any context kept alive in between. See The result cache.

Memory and copulas nothing can apply

A registered copula holds on to the values it targets until it's applied. If every value depending on a copula's inputs is deleted before anything is collected, the copula can never be applied. zeki frees such copulas, and the values they hold, after every collect, whenever zk.correlate registers a copula, and on a full gc.collect(), so you rarely need to think about it. zk.pending_copulas() counts the copulas not yet applied, and zk.free_unreachable_copulas() frees the unreachable ones on demand.