In John Wentworth and David Lorell’s paper “Natural Latents”, they aim to propose mathematical conditions under which translation between agents is guaranteed to be possible, and claim that these are the most general conditions under which translatability is guaranteed.

You can see their paper on arXiv or on the AI Alignment Forum, and read precursor posts here and here.

This paper has become a central part of my undergraduate thesis work, and as part of that work I present my own rendition of the math in their paper, with the aim of being maximally legible to readers with diverse mathematical backgrounds. Conceptual discussions are mostly outside of the scope of this post for now—we will focus on the definitions, theorems, and proofs.

Contents

1. Mediators, Redunds, and Natural Latents
   • Definition: Mediator
   • Definition: Redund
   • Definition: Natural Latent
2. Theorem: Mediator Determines Redund
   • Proof
3. Probabilistic Generative Models: Observables and Latents
   • Sensory Data
   • Observables
   • Latents
4. Theorem: Guaranteed Translatability
   • Agreement on Shared Observables
   • Statement
   • Proof
   • Corollary: Bi-directional Translation with Natural Latents
5. Appendix: Elementary Information Theory
   • Entropy
   • Mutual Information
   • Total Correlation

If you are not intimately familiar with entropy, mutual information, and total correlation, I recommend reading the Appendix before proceeding. The main text assumes familiarity with these concepts.

Mediators, Redunds, and Natural Latents

Definition: Mediator

We say that a variable $Z$ mediates over a sequence of discrete random variables $X$ when $\{X_1,X_2,\dots ,X_n\}$ are approximately conditionally independent given $Z$.

Intuitively, this means that once we know $Z$, learning the value of any $X_k$ doesn’t significantly change our uncertainty about the other $X_i$s. All the shared information between the $X_i$ variables flows through $Z$.

Formally, we require that the conditional total correlation is less than some constant $\epsilon$:

$$\mathrm{TC}(X_1,\dots ,X_n\mid Z)=\sum _{i=1}^{n}H(X_i\mid Z)-H(X_{1:n}\mid Z)\le \epsilon$$

Z is a mediator over X

$Z$ is an exact mediator if $\epsilon =0$; an approximate mediator up to error $\epsilon$ otherwise. We normally assume mediators are approximate, so we just say “mediator.” We think of $\epsilon$ as small, but our bounds are global, so there are no actual assumptions needed.

Definition: Redund

When a random variable $A$ is a function of another random variable $B$, we know the value $A$ takes as soon as we know $B$. Therefore the new information that we get is exactly zero: $H(A\mid B)=0$

When this only holds approximately, the conditional entropy is small, and we say that $A$ is approximately a function of or approximately determined by $B$:

$$H(A\mid B)\le \epsilon \text{for some small }\epsilon$$ If $Z$ is approximately a function of each element of a sequence $X$, then we say it is a redund over $X$. This means that all the information contained in $Z$ is also contained in each $X_i$.

$$H(Z\mid X_i)\le \epsilon ,\forall X_i\in \{X_1\dots X_n\}$$

Z is a redund over X

$Z$ is an exact redund if $\epsilon =0$; an approximate redund up to error $\epsilon$ otherwise. We normally assume redunds are approximate, so we just say “redund.” Again, we think of $\epsilon$ as small, but our bounds are global.

Definition: Natural Latent

$\Lambda$ is a natural latent over $X=\{X_1,X_2,\dots ,X_n\}$ up to errors ${\epsilon }_{\mathrm{med}}$ and ${\epsilon }_{\mathrm{red}}$ iff it satisfies both the mediation and redundancy conditions up to those errors:

$$\mathrm{TC}(X\mid \Lambda )\le {\epsilon }_{\mathrm{med}},H(\Lambda \mid X_j)\le {\epsilon }_{\mathrm{red}}\forall j$$

Λ is a natural latent over X

Theorem: Mediator Determines Redund

Let $X$ be a sequence of $n\ge 2$ discrete random variables $X_1,\dots ,X_n$, and assume that for some random variables $Z_{\text{med}}$ and $Z_{\text{red}}$ and for some constants ${\epsilon }_{\mathrm{med}}$ and ${\epsilon }_{\mathrm{red}}$ the following inequalities hold:

$$\mathrm{TC}(X\mid Z_{\mathrm{med}})\le {\epsilon }_{\mathrm{med}},H(Z_{\mathrm{red}}\mid X_i)\le {\epsilon }_{\mathrm{red}}\forall i$$ Then:

$$H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})\le {\epsilon }_{\mathrm{med}}+2{\epsilon }_{\mathrm{red}}$$

Proof

Pick some distinct $X_j,X_k$ where $j<k$. By the definition of conditional mutual information,

$$I(Z_{\mathrm{red}};X_k\mid Z_{\mathrm{med}})=H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})-H(Z_{\mathrm{red}}\mid X_k,Z_{\mathrm{med}})$$ $H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})$ is what we want to bound, so we put it on one side:

$$H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})=I(Z_{\mathrm{red}};X_k\mid Z_{\mathrm{med}})+H(Z_{\mathrm{red}}\mid X_k,Z_{\mathrm{med}})$$ We bound the mutual information term using monotonicity:

$$I(X_k;Z_{\mathrm{red}}\mid Z_{\mathrm{med}})\le I(X_k;Z_{\mathrm{red}},X_j\mid Z_{\mathrm{med}})$$ Now we can use the chain rule for mutual information to break apart the right hand side:

$$I(X_k;Z_{\mathrm{red}},X_j\mid Z_{\mathrm{med}})=I(X_k;X_j\mid Z_{\mathrm{med}})+I(X_k;Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})$$ Substituting, we get an upper bound on $H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})$:

$$H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})\le I(X_k;X_j\mid Z_{\mathrm{med}})+I(X_k;Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})+H(Z_{\mathrm{red}}\mid X_k,Z_{\mathrm{med}})$$ Now we will bound each of these terms individually.

First, by monotonicity we have

$$I(X_k;X_j\mid Z_{\mathrm{med}})\le I(X_k;X_1,\dots ,X_{k-1}\mid Z_{\mathrm{med}})$$ And by the nonnegativity of mutual information we have

$$I(X_k;X_{<k}\mid Z_{\mathrm{med}})\le \sum _{i=1}^{n}I(X_i;X_{<i}\mid Z_{\mathrm{med}})=\mathrm{TC}(X\mid Z_{\mathrm{med}})$$ Which by assumption is bounded by ${\epsilon }_{\mathrm{med}}$.

Second, bounding $I(X_k;Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})$ by the entropy of its second argument gives

$$I(X_k;Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})\le H(Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})$$ By monotonicity,

$$H(Z_{\mathrm{red}}\mid X_j,Z_{\mathrm{med}})\le H(Z_{\mathrm{red}}\mid X_j)$$ Which by assumption is bounded by ${\epsilon }_{\mathrm{red}}$.

Third, simply by monotonicity and the redundancy assumption,

$$H(Z_{\mathrm{red}}\mid X_k,Z_{\mathrm{med}})\le H(Z_{\mathrm{red}}\mid X_k)\le {\epsilon }_{\mathrm{red}}$$ So we have bounded each term, and substituting gives

$$H(Z_{\mathrm{red}}\mid Z_{\mathrm{med}})\le {\epsilon }_{\mathrm{med}}+{\epsilon }_{\mathrm{red}}+{\epsilon }_{\mathrm{red}}={\epsilon }_{\mathrm{med}}+2{\epsilon }_{\mathrm{red}}$$

Probabilistic Generative Models: Observables and Latents

In this section, we formalize how agents represent the world using probabilistic generative models. This framework will allow us to precisely state and prove the guaranteed translatability theorem.

If you don’t need the measure-theoretic details of how observables and latents are formally constructed from sensory data, you can skim or skip this section and jump directly to the Guaranteed Translatability theorem. The key intuition is that agents break up raw sensory data into chunks called observables and posit hidden variables (latents) to model these features.

We assume that a Bayesian agent learns a probabilistic generative model (PGM) which it uses to make predictions about the world. Traditionally, a PGM represents an agent as having a joint distribution over a set of hidden latent variables $Z_1,\dots ,Z_n$ and observables $X_1,\dots ,X_m$. Here we will begin with a single random variable, and show how first multiple observable random variables and then latent variables are introduced. Throughout this section, we assume all random variables are discrete.

Note that we will reuse some symbols from the previous section, and while we try to choose them in an intuive way, they have new meanings here.

Sensory Data

We define the random variable $D$ as follows. Let $(\Omega ,\mathcal{F},P)$ be a probability space. Then let

$$D:\Omega \to \mathcal{D}$$ be a measurable function, where:

• $\Omega$ is a sample space of outcomes.
• $\mathcal{F}$ is the $\sigma$-algebra on $\Omega$.
• $P:\mathcal{F}\to [0,1]$ is the probability measure.
• $\mathcal{D}$ is a countable set representing all possible sensory data.
• $\mathcal{A}$ is the power set of $\mathcal{D}$, denoted $\mathcal{P}(\mathcal{D})$.
• The distribution of $D$ is a Probability Mass Function (PMF) $P_D(d)=P(D=d)$ for $d\in \mathcal{D}$.

The agent can have a predictive model encoded as a distribution $P_D$.

Observables

Given sensory data $D$, we can define a sequence of observable random variables $X_1,\dots ,X_m$. We denote this sequence as $X$ and say that the random variables are measurable functions of $D$:

$$\forall i\in \{1,\dots ,m\},\exists f_i:\mathcal{D}\to {\mathcal{X}}_i\text{s.t.}{X}_i=f_i\circ D.$$ where each ${\mathcal{X}}_i$ is a countable set. The $i$-th observable random variable is then $X_i:=f_i\circ D:\Omega \to {\mathcal{X}}_i$.

The joint observable vector is $X=(X_1,\dots ,X_m):\Omega \to {\mathcal{X}}_1\times \cdots \times {\mathcal{X}}_m,$ where $X(\omega )=(f_1(D(\omega )),\dots ,f_m(D(\omega )))$.

Each $X_i$ has a well-defined PMF derived from $P_D$:

$$P_{X_i}(x)=\sum _{d\in f_i^{-1}(\{x\})}{P}_D(d).$$

Latents

Now we assume that in order to find $P_D$ and the joint distribution $P_X$, the agent uses a latent variable model.

The agent posits discrete latent random variables $Z=(Z_1,\dots ,Z_n):\Omega \to {\mathcal{Z}}_1\times \cdots \times {\mathcal{Z}}_n$, where each ${\mathcal{Z}}_j$ is a countable set.

We define the agent’s model using the joint PMF $P_{D,X,Z}$. Since $X$ is a deterministic function of $D$, the joint distribution of all variables is a degenerate extension of the distribution over $D$ and $Z$.

For any specific outcomes $d\in \mathcal{D},z\in \mathcal{Z}$, and $x\in \mathcal{X}$, the joint probabilities satisfy:

$$P_{D,X,Z}(d,x,z)=\{\begin{array}{ll}{P}_{D,Z}(d,z)& \text{if }x=(f_1(d),\dots ,f_m(d))\\ 0& \text{otherwise}\end{array}$$

Including $X$ in the joint distribution is technically redundant because $X$ adds no new information once $D$ is known (though $D$ may contain information not captured by $X$). However, we include $X$ to keep the relationship between raw data, features, and latents explicit.

Motivation of Latent Variables

Modeling $P_D$ directly is usually difficult because sensory data is high-dimensional and complex. However, it is often easier to use a generative process where we sample a simple latent factor $z$ from a prior $P_Z$, and we generate data $d$ conditioned on that factor using $P_{D\mid Z}$.

We have:

• Observable space: The countable set $\mathcal{D}$ (and by extension $\mathcal{X}$).
• Latent space: The countable set $\mathcal{Z}$.
• Random variables: Discrete variables $D$ and $Z$.

From these, we define the joint probability mass function (PMF):

$$P_{D,Z}(d,z)=P(D=d,Z=z)$$

The marginal distributions are obtained by summing over the other variable (Law of Total Probability):

$$P_D(d)=\sum _{z\in \mathcal{Z}}{P}_{D,Z}(d,z),P_Z(z)=\sum _{d\in \mathcal{D}}{P}_{D,Z}(d,z).$$

We can factor the joint distribution into a prior and a likelihood (conditional probability):

$$P_{D,Z}(d,z)=P_{D\mid Z}(d\mid z)P_Z(z)$$

This factorization allows us to express the complex data distribution $P_D$ as a mixture of (hopefully) simpler distributions:

$$P_D(d)=\sum _{z\in \mathcal{Z}}{P}_{D\mid Z}(d\mid z)P_Z(z)$$

Theorem: Guaranteed Translatability

This theorem establishes when translation between two agents’ latent variables is guaranteed to be possible. Two agents observe the world and build their own models with their own latent variables. If they agree on certain shared observables and their latent variables approximately satisfy the natural latent conditions (mediation and redundancy) then there exists an approximate mapping between the latent variables in their models.

The forward direction of this theorem shows that one agent’s mediators can determine another’s redunds, and the reverse direction shows that in the special case of two observables, being determined by all mediators implies being a redund.

Agreement on Shared Observables

We now consider two Bayesian agents, $\text{Alice}$ ($A$) and $\text{Bob}$ ($B$), who learn probabilistic generative models $M^A$ and $M^B$, respectively. $\text{Alice}$ derives observables $X^A=(X_1^A,\dots ,X_m^A)$. $\text{Bob}$ derives observables $X^B=(X_1^B,\dots ,X_n^B)$. These need not be the same family — their feature maps $f_i^A$ and $f_j^B$ may differ.

Some of $\text{Alice}$′s and $\text{Bob}$′s observables may be essentially the same, in the sense that they are functions of each other. We call these shared observables and denote them by $Y$.

Specifically, we define subsequences $Y^A\subseteq X^A$ and $Y^B\subseteq X^B$ of length $k$, such that for each $i\in \{1,\dots ,k\}$, there is a bijection:

$$g_i:{\mathcal{Y}}_i^A\to {\mathcal{Y}}_i^B$$

These variables correspond to the same feature of the data $D$ (e.g., $Y_i^B=g_i\circ Y_i^A$).

Important: The set $Y$ need not contain all observables that are shared between the agents. We only require that each element of $Y$ is indeed shared—that is, for each variable we include in $Y^A$ and $Y^B$, there exists the bijection described above. The agents may have additional shared observables beyond those in $Y$, but our analysis focuses on a particular subset $Y$ of shared observables.

While the variables may be linked in reality, the agents’ models of these variables may differ. We say that $\text{Alice}$ and $\text{Bob}$ agree on the observables in $Y$ if their marginal distributions over $Y$ are identical (under the mapping $g$).

Formally, for all $i$ and for all outcomes $y\in {\mathcal{Y}}_i^A$:

$$P^A(Y_i^A=y)=P^B(Y_i^B=g_i(y))$$

Since both agents assign the same probability mass to corresponding outcomes, the entropy of the shared observables is identical for both agents:

$$H(Y^A)=H(Y^B)$$ Consequently, $Y^A$ and $Y^B$ are interchangeable in entropy expressions, and we will often omit the superscripts (writing simply $H(Y)$) when the distinction is not needed.

Statement

Assume $\text{Alice}$ and $\text{Bob}$ agree on observables as defined above. Let $Z^B$ be any mediator in $\text{Bob}$′s model:

$$\mathrm{TC}(Y^B\mid Z^B)\le {\epsilon }_{{\text{med}}^B}$$

Then, for a redund $Z^A$ over $Y^A$ in $\text{Alice}$′s model, $Z^A$ is approximately determined by $Z^B$. Formally, if $Z^A$ is a redund up to error ${\epsilon }_{{\mathrm{red}}^A}$:

$$H(Z^A\mid Y_j)\le {\epsilon }_{{\text{red}}^A},\forall Y_j\in Y^A$$ Then:

$$H(Z^A\mid Z^B)\le {\epsilon }_{{\text{med}}^B}+2{\epsilon }_{{\text{red}}^A}.$$

Furthermore, when there are exactly two shared observables $Y_1$ and $Y_2$, the implication works in reverse as well. If $Z^A$ is determined by any mediator $Z^B$ in $\text{Bob}$′s model up to error ${\epsilon }_{\det }$, then $Z^A$ must be a redund up to error ${\epsilon }_{\det }$.

Formally, if for all $Z^B$ such that $\mathrm{TC}(Y^B\mid Z^B)\le {\epsilon }_{{\text{med}}^B}$, we have $H(Z^A\mid Z^B)\le {\epsilon }_{\det }$ for some constant ${\epsilon }_{\det }$, then:

$$H(Z^A\mid Y_j^A)\le {\epsilon }_{\det }\forall Y_j^A\in \{Y_1^A,Y_2^A\}$$

Proof

We are given that $\text{Alice}$ and $\text{Bob}$ agree on observables and that $\text{Bob}$′s latent $Z^B$ mediates over $Y^B$ up to error ${\epsilon }_{{\text{med}}^B}$:

$$\mathrm{TC}(Y^B\mid Z^B)\le {\epsilon }_{{\text{med}}^B}\text{\hspace{1em}(A)}$$

Forward Direction

Assume $\text{Alice}$′s latent $Z^A$ is a redund over her shared observables $Y^A$:

$$H(Z^A\mid Y_j^A)\le {\epsilon }_{{\text{red}}^A},\forall Y_j^A\in Y^A\text{\hspace{1em}(B)}$$

Since $\text{Alice}$ and $\text{Bob}$ agree on observables, the redundancy bound holds for $\text{Bob}$′s observables as well:

$$H(Z^A\mid Y_j^B)\le {\epsilon }_{{\text{red}}^A},\forall Y_j^B\in Y^B\text{\hspace{1em}(C)}$$

Now we apply the Mediator Determines Redund Theorem to the system $(Y^B,Z^B,Z^A)$.

• $Z^B$ is the mediator over $Y^B$ (from B).
• $Z^A$ is the redund over $Y^B$ (from D).

Substituting the bounds directly into the theorem gives:

$$H(Z^A\mid Z^B)\le {\epsilon }_{{\text{med}}^B}+2{\epsilon }_{{\text{red}}^A}$$

Reverse Direction

We restrict ourselves to the case $Y=(Y_1,Y_2)$. We assume that for any mediator $Z^B$ in $\text{Bob}$′s model (where $\mathrm{TC}(Y^B\mid Z^B)\le {\epsilon }_{{\text{med}}^B}$), $Z^A$ is determined by that mediator up to error ${\epsilon }_{\det }$:

$$H(Z^A\mid Z^B)\le {\epsilon }_{\det }$$

We must show that $Z^A$ is a redund.

In the 2-variable case, the observables mediate themselves. If $\text{Bob}$ chooses $Z^B=Y_1^B$, the conditional total correlation is zero:

$$\mathrm{TC}(Y_1^B,Y_2^B\mid Y_1^B)=I(Y_1^B;Y_2^B\mid Y_1^B)=0$$

Since $Y_1^B$ is a valid mediator (with ${\epsilon }_{{\text{med}}^B}=0$), our assumption requires that it determines $Z^A$:

$$H(Z^A\mid Y_1^B)\le {\epsilon }_{\det }$$ By symmetry, $Y_2^B$ is also a valid mediator, so:

$$H(Z^A\mid Y_2^B)\le {\epsilon }_{\det }$$

Using the agreement on observables assumption to map back to $\text{Alice}$′s variables, we have:

$$H(Z^A\mid Y_j^A)\le {\epsilon }_{\det }\forall j\in \{1,2\}$$

So $Z^A$ is a redund over $Y^A$ up to error ${\epsilon }_{\det }$.

Corollary: Bi-directional Translation with Natural Latents

The theorem above establishes one-way translatability: if Bob has a mediator and Alice has a redund, then Bob’s latent determines Alice’s. However, when both agents use approximately natural latents, we get an approximate bijection between their latents.

Statement: Suppose Alice and Bob agree on shared observables $Y$, and both $Z^A$ and $Z^B$ are natural latents over their respective shared observables. That is:

$Z^A$ is both a mediator over $Y^A$ (with $\mathrm{TC}(Y^A\mid Z^A)\le {\epsilon }_{{\text{med}}^A}$) and a redund over $Y^A$ (with $H(Z^A\mid Y_j^A)\le {\epsilon }_{{\text{red}}^A}$ for all $j$)

$Z^B$ is both a mediator over $Y^B$ (with $\mathrm{TC}(Y^B\mid Z^B)\le {\epsilon }_{{\text{med}}^B}$) and a redund over $Y^B$ (with $H(Z^B\mid Y_j^B)\le {\epsilon }_{{\text{red}}^B}$ for all $j$)

Then both conditional entropies are small:

$$H(Z^A\mid Z^B)\le {\epsilon }_{{\text{med}}^B}+2{\epsilon }_{{\text{red}}^A}$$

$$H(Z^B\mid Z^A)\le {\epsilon }_{{\text{med}}^A}+2{\epsilon }_{{\text{red}}^B}$$

Proof: Apply the forward direction of the Guaranteed Translatability theorem twice.

Alice → Bob: Since $Z^B$ mediates $Y^B$ and $Z^A$ is a redund over $Y^A$ (hence over $Y^B$ by agreement), we have $$H(Z^A\mid Z^B)\le {\epsilon }_{{\text{med}}^B}+2{\epsilon }_{{\text{red}}^A}$$

Bob → Alice: Since $Z^A$ mediates $Y^A$ and $Z^B$ is a redund over $Y^B$ (hence over $Y^A$ by agreement), we have $$H(Z^B\mid Z^A)\le {\epsilon }_{{\text{med}}^A}+2{\epsilon }_{{\text{red}}^B}$$

Appendix: Elementary Information Theory

We will giving a few elementary definitions and identities from information theory. We will always deal with discrete random variables, as this avoids continuous pathologies and is arguably the correct level of description for the real world anyway. This will assume very minimal prior knowledge, but for an authoritative source refer to Elements of Information Theory 2nd Edition by Cover and Thomas.

So $X$ is a discrete random variable which takes values $x_1,x_2,\cdots ,x_n$, and we write $$P(X=x_i)=p(x_i)$$ A joint distribution $(X,Y)$ takes values $(x_1,y_1),(x_1,y_2),\dots ,(x_2,y_1),(x_2,y_2),\dots ,$ and $$P(X=x_i,Y=y_j)=p(x_i,y_j)$$

All of what follows can be expressed in terms of discrete random variables with probability mass functions.

Entropy

The entropy of a random variable is the average information we get from sampling it. $$H(X)=-\sum _{x}p(x)\log p(x)$$

Since probabilities are between zero and one, we have $\log p(x)\le 0$, so we multiple by $-1$ to get a nonnegative number. Each term $p(x)$ is the probability of outcome $x$, while $-\log p(x)$ is the information content (or “surprise”) of that outcome. Rare events have high information content, and common events have low information content. Entropy is the probability-weighted average of these information contents.

The joint entropy measures uncertainty about multiple variables. $$H(X,Y)=-\sum _{x,y}p(x,y)\log p(x,y)$$ The conditional entropy is the average information we get from sampling $X$ when we already know the value of $Y$. $$H(X\mid Y)=H(X,Y)-H(Y)$$

Mutual Information

The mutual information between two random variables measures how much information they share, or equivalently, how much knowing one tells you about the other.

We give two equivalent forms:

$$I(X;Y)=H(X)+H(Y)-H(X,Y)=H(X)-H(X\mid Y)$$

The first form shows mutual information as “total information minus joint information”—the extent to which $X$ and $Y$ are redundant. The second form shows it as “reduction in uncertainty”—how much knowing $Y$ reduces our uncertainty about $X$.

Conditional mutual information measures the information $X$ and $Y$ share, given that we already know $Z$:

$$I(X;Y\mid Z)=H(X\mid Z)+H(Y\mid Z)-H(X,Y\mid Z)=H(X\mid Z)-H(X\mid Y,Z)$$

Again, the first form captures redundancy given $Z$, while the second form captures how much $Y$ reduces our uncertainty about $X$ beyond what $Z$ already tells us.

Chain rule for mutual information decomposes mutual information about multiple variables into successive contributions:

$$I(X;Y,Z\mid W)=I(X;Y\mid W)+I(X;Z\mid Y,W)$$

This says: the information $X$ shares with both $Y$ and $Z$ (given $W$) equals the information $X$ shares with $Y$ (given $W$), plus the additional information $X$ shares with $Z$ beyond what $Y$ already provided (given both $W$ and $Y$). We use this identity in our proof of the Mediator Determines Redund theorem.

Mutual information is bounded by entropy of either argument. Since entropy is nonnegative, $H(X\mid Y,Z)\ge 0$, so from the second form we have:

$$I(X;Y\mid Z)\le H(X\mid Z)$$

This states that knowing $Y$ cannot tell you more about $X$ than the total uncertainty you had about $X$ in the first place (given $Z$). This bound appears in our proof of Mediator Determines Redund.

Total Correlation

Total correlation measures the total dependence among a set of variables—how much we learn by observing them together versus independently.

We give two equivalent forms for both the unconditional and conditional cases.

Unconditional total correlation:

$$\mathrm{TC}(X_1,\dots ,X_n)=\sum _{i=1}^{n}H(X_i)-H(X_1,\dots ,X_n)=\sum _{i=1}^{n}I(X_i;X_{<i})$$

where $X_{<i}=\{X_1,\dots ,X_{i-1}\}$.

The first form is a direct generalization of the two-variable mutual information: “sum of individual entropies minus joint entropy,” measuring how much information is double-counted when we sum the individual entropies. The second form expresses this as a sum of successive dependencies: how much each variable $X_i$ shares with all preceding variables.

Conditional total correlation:

$$\mathrm{TC}(X_1,\dots ,X_n\mid Z)=\sum _{i=1}^{n}H(X_i\mid Z)-H(X_1,\dots ,X_n\mid Z)=\sum _{i=1}^{n}I(X_i;X_{<i}\mid Z)$$

The conditional case measures the same redundancy, but with everything measured relative to already knowing $Z$.