Dependency and copulas¶
A copula makes simulated values dependent by reordering their simulations. zeki applies copulas for you, and keeps track of everything built from the values a copula reorders, so a model stays consistent however it's written.
Correlating a collection¶
zk.correlate(collection, copula) correlates the rows of a float array collection: each row's
simulations are reordered so the rows have the dependency the copula describes. Elliptical copulas
take a function giving the target correlation between two rows, from their dimension rows:
gross = sample(1)
correlations = {
frozenset({"home", "motor"}): 0.3,
frozenset({"home", "property"}): 0.6,
frozenset({"motor", "property"}): 0.2,
}
def correlation(row_a: tuple, row_b: tuple) -> float:
"""The target correlation between two rows, given their dim rows."""
if row_a == row_b:
return 1.0
return correlations[frozenset({row_a[0], row_b[0]})]
zk.correlate(gross, zk.Copula.gaussian(correlation))
The function is called once per pair of rows, and the correlations must form a positive-definite matrix. zeki reorders the simulations with the Iman-Conover method, so each row keeps exactly the values it had, only in a new order.
Everything built from the values follows¶
zk.correlate is lazy, like the rest of zeki. It registers the copula on the inputs the
collection depends on: the arrays you added, or that zk.float returned. When anything depending
on those inputs is collected, they are reordered first, so every value built from them, before or
after the copula was registered, reads the reordered simulations:
gross = sample(2)
net = gross * 0.8 # built before the copula is registered
zk.correlate(gross, zk.Copula.gaussian(correlation))
recovery = (gross - 30_000.0).clamp(0.0, 50_000.0) # built after it
gross_values, net_values = zk.collect_all(gross, net)
# net is still gross * 0.8, simulation by simulation: both read the reordered inputs
This is how sim-tying behaves in currently available commercial capital modelling products, without you keeping track of which values are tied to which.
Operations tie their operands¶
Once c = a + b exists, simulation i of c is simulation i of a plus simulation i of
b. If a copula later moved a but not b, c would silently stop being a + b. So an
operation ties its operands' inputs as it's built: a copula that reorders one reorders the other
by the same permutation.
It follows that two values that already depend on a common input can't be correlated with each other, since no reordering could move one without the other:
pair = sample(3)
home = pair.get_array(("home", 2026), "gross")
motor = pair.get_array(("motor", 2026), "gross")
combined = home + motor # ties home's inputs to motor's: combined stays home + motor
with pytest.raises(ValueError, match="common input"):
zk.correlate(pair, zk.Copula.gaussian(correlation)) # home and motor can no longer move independently
A reduction across rows, such as sum() or a group-by, ties the rows it combines in the same way:
rows = sample(4)
total = rows.sum() # a reduction across rows ties every row's inputs too
with pytest.raises(ValueError, match="common input"):
zk.correlate(rows, zk.Copula.gaussian(correlation))
So correlate rows before combining them. Ties last as long as the inputs do, not as long as the
value that created them: deleting c doesn't untie a and b, because values computed from c
may still exist.
Nested copulas¶
Because a copula ties its targets too, copulas compose. A common pattern correlates classes within each underwriting year, then correlates the years' totals. A year's total ties its class rows, so the second copula moves them together, keeping the correlations the first gave them:
rng = np.random.default_rng(9)
reserves = zk.Collection(Business, Gross)
for class_of_business in ["home", "motor", "property"]:
for year in [2024, 2025, 2026]:
reserves.add(class_of_business, year, rng.lognormal(10.0, 0.5, SIMULATIONS))
# 1. Within each year, correlate the classes.
for year in [2024, 2025, 2026]:
zk.correlate(reserves.filter(zk.dim("underwriting_year") == year), zk.Copula.gaussian(correlation))
# 2. Correlate the years' totals. A year's total ties its class rows, so this copula moves them
# together, keeping the correlations step 1 gave them.
by_year = reserves.group_by(zk.dim("underwriting_year")).sum()
zk.correlate(by_year, zk.Copula.gaussian(lambda year_a, year_b: 1.0 if year_a == year_b else 0.4))
Copula families¶
zk.Copula.gaussian(correlation)
zk.Copula.student_t(correlation, degrees_of_freedom=4.0) # tail dependence
zk.Copula.clayton(theta=2.0) # lower-tail dependence
zk.Copula.gumbel(theta=1.5) # upper-tail dependence
zk.Copula.frank(theta=5.0) # no tail dependence
zk.Copula.joe(theta=1.5) # upper-tail, stronger than Gumbel for the same Kendall's tau
zk.Copula.ali_mikhail_haq(theta=0.5) # weak dependence only
| Copula | Parameters | Dependence |
|---|---|---|
gaussian |
A correlation function | No tail dependence |
student_t |
A correlation function, degrees_of_freedom |
Tail dependence, growing as degrees_of_freedom falls |
clayton |
theta >= 0 (0 is independence) |
Lower tail. Kendall's tau is theta / (theta + 2). |
gumbel |
theta >= 1 (1 is independence) |
Upper tail. Kendall's tau is 1 - 1 / theta. |
frank |
theta >= 0 (0 is independence) |
No tail dependence |
joe |
theta >= 1 (1 is independence) |
Upper tail, stronger than Gumbel's for the same Kendall's tau |
ali_mikhail_haq |
0 <= theta < 1 (0 is independence) |
Weak: Kendall's tau below 1/3 |
The Archimedean copulas (Clayton, Gumbel, Frank, Joe and Ali-Mikhail-Haq) take one theta for
every pair of rows, and cover positive dependence only.
Seeds and reproducibility¶
A copula's random draws come from a seed. Without one, zeki derives it from the root seed of the
current zk.Context and the lineage of the values being correlated, so a model gives the same
results however its statements are ordered. Set the root seed with zk.Context, or pass a seed
directly:
with zk.Context(seed=42):
gross = sample(6)
zk.correlate(gross, zk.Copula.gumbel(theta=1.5)) # seed derived from 42 and gross's lineage
zk.correlate(sample(7), zk.Copula.clayton(theta=2.0), seed=1234) # or an explicit seed
zk.Context is held per thread and per async task.
Seeing what a value depends on¶
collection.explain() describes how a collection's values came to be, without computing them: the
inputs and operations they depend on, the groups those inputs are tied into, and the copulas that
reorder them, applied or not yet:
gross = sample(8)
net = (gross * 0.8).filter(zk.dim("class_of_business") == "motor")
zk.correlate(gross, zk.Copula.gaussian(correlation))
print(net.explain())
1 cell over 1 input and 1 operation
tie groups: 1 (inputs per group: 1)
copulas, oldest first:
1. Gaussian over 3 cells, seed 8638896654103472097: reorders 1 input of these (not yet applied)
formula, the cell: (input * 0.8)
e.g. ('motor', 2026) = (input 1 * 0.8)
Cells built the same way, such as a column computed identically in every row, share one formula.
Why there's no global context¶
Every lazy system has to decide where its graph, and side effects such as copulas, live. There are roughly three answers:
| Approach | Used by | What happened |
|---|---|---|
| One global graph, plus a reset function | TensorFlow 1 (tf.get_default_graph(), tf.reset_default_graph()) |
The most disliked part of TensorFlow 1: stale nodes in notebooks, confusing errors and trouble with threads. TensorFlow 2 dropped it. |
A scoped context entered with with |
PyMC (with pm.Model():), decimal.localcontext, numpy.errstate, and the model files of commercial capital modelling products |
Fine for settings that really do apply to everything in scope; clumsy when every operation has to find "the" context. |
| The graph lives in the values | PyTorch autograd (tensor.grad_fn), Dask, Polars' LazyFrame, JAX |
No reset needed: lineage is freed when the last value referring to it is. Works with threads, and each value is self-contained. |
zeki takes the third approach. A copula's question, "if this input is reordered, what else must
move with it?", depends only on the input, so the state lives on the inputs rather than in a
context. Values own their lineage, so in a notebook, rebinding or deleting a variable frees what
only it used, and there's nothing to reset. Only settings that genuinely apply to everything, such
as the root seed, live in a context: zk.Context.
Caching across reruns is solved separately, as Dask, Bazel and Nix solve it: by keying results on a hash of how they're computed rather than on object identity. Rebuilding the same model gives the same keys, so a rerun reuses earlier results without any context kept alive in between. See The result cache.
Memory and copulas nothing can apply¶
A registered copula holds on to the values it targets until it's applied. If every value depending
on a copula's inputs is deleted before anything is collected, the copula can never be applied.
zeki frees such copulas, and the values they hold, after every collect, whenever zk.correlate
registers a copula, and on a full gc.collect(), so you rarely need to think about it.
zk.pending_copulas() counts the copulas not yet applied, and zk.free_unreachable_copulas() frees
the unreachable ones on demand.