This work was done at the Iliad fellowship under the mentorship of @Daniel C. Thanks to Daniel C and Sam Eisenstat for comments on the draft of this post.

In Sam Eisenstat’s recent post “Research directions in condensation: varieties of objectivity”, he wrote:

In Eisenstat (2025) and in Demski (2025), condensation is expressed in terms of some functions, the simple score and the conditioned score, which relate it to compression. Almost perfect condensation and related ideas are helpful for understanding why the objectivity theorem gives good bounds in certain situations, but the hypotheses of almost perfect condensation don’t have a clear relation to compression. It would be interesting if these two perspectives could be unified.

I have attempted to unify the two perspectives here.

To summarize:

• A low conditioned score is stronger than the almost perfect condensation conditions.
• A low conditioned-score gap gives bounds for the reconstruction and Markov conditions.
• A small Markov condition bound together with a strengthened reconstruction condition gives a low conditioned score gap.

1. Setup

Let $X=(X_1,\dots ,X_n)$ be finite-alphabet random variables and let $L=(Y_A{)}_{A\in \mathcal{J}}$ be a condensation, where $\mathcal{J}\subseteq {\mathcal{P}}^{+}I$ indexes the latent variables. All slots outside $\mathcal{J}$ are taken to be constant.

Write

$$Y_{\supseteq A}:=(Y_K:K\in \mathcal{J},K\supseteq A),Y_{⊋A}:=(Y_K:K\in \mathcal{J},K⊋A).$$

Assume that the latents reconstruct the observables:

$$X_i⪯Y_{\ni i}:=(Y_A:A\in \mathcal{J},i\in A),i\in I,$$

where $U⪯V$ means $H(U\mid V)=0$.

Conditioned score

For nonempty $B\subseteq I$, let

$${\mathcal{J}}_B:=\{A\in \mathcal{J}:A\cap B\ne \varnothing \},Y_{{\mathcal{J}}_B}:=(Y_A:A\in {\mathcal{J}}_B).$$

I will often refer to $Y_{{\mathcal{J}}_B}$ as the “active latents”.

Define the conditioned score^[1]

$$\chi (B):=\sum _{A\in {\mathcal{J}}_B}H(Y_A\mid Y_{⊋A}).$$

Its lower bound is $H(X_B)$, so I will call $\chi (B)-H(X_B)$ the conditioned-score gap.

Almost perfect condensation

Call a family $\mathcal{F}\subseteq \mathcal{J}$ upward closed in $\mathcal{J}$ if $A\in \mathcal{F}$, $K\in \mathcal{J}$, $K\supseteq A$ imply $K\in \mathcal{F}$. For any $\mathcal{F}\subseteq \mathcal{J}$ write $Y_{\mathcal{F}}:=(Y_A:A\in \mathcal{F})$.

Define the worst conditional dependence between two upward-closed families given their common part:

$$\eta :=\underset{\mathcal{F},\mathcal{G}}{\max }I(Y_{\mathcal{F}};Y_{\mathcal{G}}\mid Y_{\mathcal{F}\cap \mathcal{G}}),$$

the maximum running over all pairs $\mathcal{F},\mathcal{G}\subseteq \mathcal{J}$ that are each upward closed in $\mathcal{J}$.

An $(m,r,\epsilon )$-almost perfect condensation has (size condition) $|\mathcal{J}|\le m$; (reconstruction condition) $H(Y_{\supseteq A}\mid X_B)\le \epsilon$ for all $A\in \mathcal{J},B\in {\mathcal{P}}^{+}I$ with $|A\cap B|>r$; and (Markov condition) $\eta \le \epsilon$.^[2]

2. Breaking Up the Conditioned Score

The conditioned-score gap can be decomposed into two nonnegative information-theoretic charges. This comes from the exact characterization of perfect condensation in Sam’s Theorem 5.10.^[3]

$$\chi (B)-H(X_B)=D_{\mathrm{KL}}(P_B\parallel Q_B)+H(Y_{{\mathcal{J}}_B}\mid X_B).$$

The KL divergence from the reverse-inclusion factorization

Let $P_B$ be the joint distribution of the active latents $Y_{{\mathcal{J}}_B}$. Define a reference distribution $Q_B$ that retains only the dependence of each $Y_A$ on the latents $Y_K$ indexed by strict supersets $K⊋A$.

$$Q_B(Y_{{\mathcal{J}}_B}):=\prod _{A\in {\mathcal{J}}_B}{P}_B(Y_A\mid Y_{⊋A}).$$

The KL divergence between the true distribution and this factored reference is

$$D_{\mathrm{KL}}(P_B\parallel Q_B)=\sum _{A\in {\mathcal{J}}_B}H(Y_A\mid Y_{⊋A})-H(Y_{{\mathcal{J}}_B})$$^[4]

By Gibbs’ inequality this is nonnegative. It measures how much dependence is not explained by the reverse-inclusion hierarchy.

The reconstruction defect ($H(Y_{{\mathcal{J}}_B}\mid X_B)$)

The remaining difference from $H(X_B)$ is the reconstruction defect, which measures the randomness remaining in the active latents once the observables in $B$ are known.

By our assumption that the latents reconstruct the observables ($X_i⪯Y_{\ni i}$), the active latents $Y_{{\mathcal{J}}_B}$ completely determine $X_B$. Therefore, $H(X_B\mid Y_{{\mathcal{J}}_B})=0$, which gives:

$$H(Y_{{\mathcal{J}}_B})-H(X_B)=H(Y_{{\mathcal{J}}_B}\mid X_B)$$

Clearly $H(Y_{{\mathcal{J}}_B}\mid X_B)\ge 0$.

Now, by adding $0$ in the form of $H(Y_{{\mathcal{J}}_B})-H(Y_{{\mathcal{J}}_B})$ to the conditioned score gap, we can decompose the gap into two clean nonnegative pieces:

$$\chi (B)-H(X_B)=\sum _{A\in {\mathcal{J}}_B}H(Y_A\mid Y_{⊋A})-H(Y_{{\mathcal{J}}_B})+H(Y_{{\mathcal{J}}_B})-H(X_B)$$

Substituting our pieces in we get what I will refer to as the score gap anatomy:

$$\chi (B)-H(X_B)=\underset{\text{Markov defect}}{\underbrace{D_{\mathrm{KL}}(P_B\parallel Q_B)}}+\underset{\text{reconstruction defect}}{\underbrace{H(Y_{{\mathcal{J}}_B}\mid X_B)}}.$$

These defects are related to but distinct from the corresponding conditions of almost perfect condensation.

3. Low scores imply the reconstruction and Markov conditions

Statement

For every $A\in \mathcal{J}$ and $B\in {\mathcal{P}}^{+}I$ with $A\cap B\ne \varnothing$,

$$H(Y_{\supseteq A}\mid X_B)\le \chi (B)-H(X_B)$$

For every pair of upward-closed families $\mathcal{F},\mathcal{G}\subseteq \mathcal{J}$,

$$I(Y_{\mathcal{F}};Y_{\mathcal{G}}\mid Y_{\mathcal{F}\cap \mathcal{G}})\le \chi (I)-H(X_I)$$^[5]

Proof.^[6] If $A\cap B\ne \varnothing$, every slot containing $A$ meets $B$. So $Y_{\supseteq A}$ is a subtuple of $Y_{{\mathcal{J}}_B}$, and the score gap anatomy gives

$$H(Y_{\supseteq A}\mid X_B)\le H(Y_{{\mathcal{J}}_B}\mid X_B)\le \chi (B)-H(X_B).$$

For the Markov bound, partition $\mathcal{J}$ into quadrants based on intersections with $\mathcal{F},\mathcal{G}\subseteq \mathcal{J}$:

$$C=\mathcal{F}\cap \mathcal{G},P=\mathcal{F}\setminus \mathcal{G},Q=\mathcal{G}\setminus \mathcal{F},R=\mathcal{J}\setminus (\mathcal{F}\cup \mathcal{G}).$$

Because $\mathcal{F}$ and $\mathcal{G}$ are upward closed, the block order $C,P,Q,R,$ internally ordered by reverse inclusion, is a valid reverse-inclusion order. Conditioning each score term on the preceding blocks and applying the chain rule:

$$\chi (I)\ge H(Y_C)+H(Y_P\mid Y_C)+H(Y_Q\mid Y_C)+H(Y_R\mid Y_C,Y_P,Y_Q).$$

The right-hand side equals

$$H(Y_{\mathcal{J}})+I(Y_P;Y_Q\mid Y_C)=H(Y_{\mathcal{J}})+I(Y_{\mathcal{F}};Y_{\mathcal{G}}\mid Y_{\mathcal{F}\cap \mathcal{G}}).$$

Therefore the Markov defect is at most $D_{\mathrm{KL}}(P_I\parallel Q_I)=\chi (I)-H(Y_{\mathcal{J}})$, which is at most $\chi (I)-H(X_I)$ by the score gap anatomy. $\square$

If the relevant conditioned score gaps $\chi (B)-H(X_B)$ are at most $\delta$, then the reconstruction condition holds with error $\delta$ wherever $|A\cap B|>r$, and the Markov condition holds with error $\chi (I)-H(X_I)\le \delta$. Therefore low conditioned-score gaps imply the almost-perfect condensation conditions (other than the size condition).

4. The reconstruction and Markov conditions imply low conditioned scores

For a nonempty $B\subseteq I$, recall the definitions

$$\eta :=\underset{\mathcal{F},\mathcal{G}}{\max }I(Y_{\mathcal{F}};Y_{\mathcal{G}}\mid Y_{\mathcal{F}\cap \mathcal{G}})$$

and

$${\mathcal{J}}_B:=\{A\in \mathcal{J}:A\cap B\ne \varnothing \}.$$

Define

$${\mathcal{M}}_B:=\left\{C\in {\mathcal{J}}_B:\text{there is no }D\in {\mathcal{J}}_B\text{ with }D⊊C\right\}.$$

Thus ${\mathcal{M}}_B$ is the set of active indices that are minimal under inclusion. Every $A\in {\mathcal{J}}_B$ contains some $C\in {\mathcal{M}}_B$. Therefore, every active latent $Y_A$ occurs in at least one of the joint variables $Y_{\supseteq C}$ with $C\in {\mathcal{M}}_B$.

Statement

For every nonempty $B\subseteq I$ with ${\mathcal{J}}_B\ne \varnothing$,

$$\chi (B)-H(X_B)\le (|{\mathcal{J}}_B|-1)\eta +\sum _{C\in {\mathcal{M}}_B}H(Y_{\supseteq C}\mid X_B).$$

Consequently, if $|{\mathcal{J}}_B|\le m,\eta \le \epsilon$, and

$$H(Y_{\supseteq C}\mid X_B)\le \epsilon \text{for every }C\in {\mathcal{M}}_B,$$

then

$$\chi (B)-H(X_B)\le (|{\mathcal{J}}_B|-1+|{\mathcal{M}}_B|)\epsilon \le (2m-1)\epsilon ,(m\ge 1).$$

In particular, the displayed reconstruction bounds follow from the reconstruction condition of almost-perfect condensation whenever

$$|C\cap B|>r\text{for every }C\in {\mathcal{M}}_B$$

Proof.^[7] Let $N:=|{\mathcal{J}}_B|$, and choose an ordering $A_1,\dots ,A_N$ of ${\mathcal{J}}_B$ such that strict supersets occur before their subsets: if $A_s⊋A_t$, then $s<t$.

Recall that $P_B$ is the joint distribution of $Y_{{\mathcal{J}}_B}$ and that

$$Q_B(Y_{{\mathcal{J}}_B}):=\prod _{A\in {\mathcal{J}}_B}{P}_B(Y_A\mid Y_{⊋A})$$

is its reverse-inclusion factorization. Expanding the KL divergence gives

$$D_{\mathrm{KL}}(P_B\parallel Q_B)=\sum _{t=1}^{N}H(Y_{A_t}\mid Y_{⊋A_t})-H(Y_{{\mathcal{J}}_B}).$$

The chain rule applied in the above ordering gives

$$H(Y_{{\mathcal{J}}_B})=\sum _{t=1}^{N}H(Y_{A_t}\mid Y_{A_1},\dots ,Y_{A_{t-1}}).$$

Therefore,

$$D_{\mathrm{KL}}(P_B\parallel Q_B)=\sum _{t=1}^N\left[H(Y_{A_t}\mid Y_{⊋A_t})-H(Y_{A_t}\mid Y_{A_1},\dots ,Y_{A_{t-1}})\right].$$

The summand for $t=1$ is zero, as $A_1$ has no strict superset in ${\mathcal{J}}_B$, since any such superset would have to occur before $A_1$. Hence

$$D_{\mathrm{KL}}(P_B\parallel Q_B)=\sum _{t=2}^N\left[H(Y_{A_t}\mid Y_{⊋A_t})-H(Y_{A_t}\mid Y_{A_1},\dots ,Y_{A_{t-1}})\right].$$

Thus the KL divergence is a sum of $N-1$ potentially nonzero terms.

For each $t\ge 2$, define

$${\mathcal{F}}_t:=\{K\in \mathcal{J}:K\supseteq A_t\},{\mathcal{G}}_t:=\{A_1,\dots ,A_{t-1}\}.$$

The families ${\mathcal{F}}_t$ and ${\mathcal{G}}_t$ are both upward closed by definition.

Their intersection is ${\mathcal{F}}_t\cap {\mathcal{G}}_t=\{K\in \mathcal{J}:K⊋A_t\}={\mathcal{F}}_t\setminus \{A_t\}.$

Therefore, $Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t}=Y_{⊋A_t}$ and $Y_{{\mathcal{F}}_t}=(Y_{A_t},Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})$.

Using $I(U;V\mid W)=H(U\mid W)-H(U\mid V,W)$, we have

$$I(Y_{{\mathcal{F}}_t};Y_{{\mathcal{G}}_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})=H(Y_{{\mathcal{F}}_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})-H(Y_{{\mathcal{F}}_t}\mid Y_{{\mathcal{G}}_t},Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})=H(Y_{A_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})-H(Y_{A_t}\mid Y_{{\mathcal{G}}_t}).$$

The second equality holds because the variables in $Y_{{\mathcal{F}}_t}$ other than $Y_{A_t}$ are being conditioned on.

Therefore, $I(Y_{{\mathcal{F}}_t};Y_{{\mathcal{G}}_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})=H(Y_{A_t}\mid Y_{⊋A_t})-H(Y_{A_t}\mid Y_{A_1},\dots ,Y_{A_{t-1}})$.

As ${\mathcal{F}}_t$ and ${\mathcal{G}}_t$ are upward closed, the Markov condition gives

$$I(Y_{{\mathcal{F}}_t};Y_{{\mathcal{G}}_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})\le \eta .$$

Applying this bound to each of the $N-1=|{\mathcal{J}}_B|-1$ summands gives

$$D_{\mathrm{KL}}(P_B\parallel Q_B)=\sum _{t=2}^{N}I(Y_{{\mathcal{F}}_t};Y_{{\mathcal{G}}_t}\mid Y_{{\mathcal{F}}_t\cap {\mathcal{G}}_t})\le (|{\mathcal{J}}_B|-1)\eta .$$

It remains to bound the other term in the conditional score gap decomposition. Recall: every $A\in {\mathcal{J}}_B$ contains some $C\in {\mathcal{M}}_B$, and therefore $Y_A$ is a component of $Y_{\supseteq C}$ for at least one $C\in {\mathcal{M}}_B$. The joint variables $(Y_{\supseteq C}{)}_{C\in {\mathcal{M}}_B}$ consequently contain all the active latents, and subadditivity gives

$$H(Y_{{\mathcal{J}}_B}\mid X_B)\le \sum _{C\in {\mathcal{M}}_B}H(Y_{\supseteq C}\mid X_B).$$

Combining this with

$$\chi (B)-H(X_B)=D_{\mathrm{KL}}(P_B\parallel Q_B)+H(Y_{{\mathcal{J}}_B}\mid X_B)$$

gives

$$\chi (B)-H(X_B)\le (|{\mathcal{J}}_B|-1)\eta +\sum _{C\in {\mathcal{M}}_B}H(Y_{\supseteq C}\mid X_B).$$

If $\eta \le \epsilon$ and

$H(Y_{\supseteq C}\mid X_B)\le \epsilon \text{for every }C\in {\mathcal{M}}_B$,

then

$$\chi (B)-H(X_B)\le (|{\mathcal{J}}_B|-1+|{\mathcal{M}}_B|)\epsilon .$$

Finally, $|{\mathcal{J}}_B|,|{\mathcal{M}}_B|\le |\mathcal{J}|\le m$, so

$$|{\mathcal{J}}_B|-1+|{\mathcal{M}}_B|\le 2m-1.$$

This proves the stated bound. $\square$

On the strengthened reconstruction condition

Note that the hypothesis of the conditions to score direction requires

$$H(Y_{\supseteq C}\mid X_B)\le \epsilon \text{for every }C\in {\mathcal{M}}_B.$$

This follows from the reconstruction condition of an $(m,r,\epsilon )$-almost perfect condensation only when

$$|C\cap B|>r\text{for every }C\in {\mathcal{M}}_B.$$

Suppose a condensation is $(m,r,\epsilon )$-almost perfect. The Markov condition gives

$$\eta \le \epsilon .$$

The reconstruction condition gives

$$H(Y_{\supseteq C}\mid X_B)\le \epsilon$$

for every $C\in {\mathcal{M}}_B$ satisfying $|C\cap B|>r$. Therefore, if

$$|C\cap B|>r\text{for every }C\in {\mathcal{M}}_B,$$

then

$$\chi (B)-H(X_B)\le (|{\mathcal{J}}_B|-1+|{\mathcal{M}}_B|)\epsilon \le (2m-1)\epsilon .$$

The value of $r$ determines which sets $B$ receive this bound. When $r=0$, it always holds because every $C\in {\mathcal{M}}_B\subseteq {\mathcal{J}}_B$ intersects $B$. When $r>0$, the bound applies to those $B$ that intersect every minimal active index in more than $r$ elements.

We can modify the score to make it more equivalent to the almost perfect condensation conditions. I will provide only a sketch here; you can follow the exact arguments above to see that it works. Recall the score gap anatomy

$$\chi (B)-H(X_B)=\underset{\text{Markov defect}}{\underbrace{D_{\mathrm{KL}}(P_B\parallel Q_B)}}+\underset{\text{reconstruction defect}}{\underbrace{H(Y_{{\mathcal{J}}_B}\mid X_B)}}$$

and define the family of indexes where the almost perfect condensation requirement holds;

$${\mathcal{K}}_r(B):=\{A\in {\mathcal{J}}_B:|A\cap B|>r\}$$

Then we can decompose the reconstruction defect into parts:

$$H(Y_{{\mathcal{J}}_B}\mid X_B)=H(Y_{{\mathcal{K}}_r(B)}\mid X_B)+H(Y_{{\mathcal{J}}_B\setminus {\mathcal{K}}_r(B)}\mid X_B,Y_{{\mathcal{K}}_r(B)})$$

So if we want a score which is bounded by the almost perfect condensations without the added assumption that $H(Y_{\supseteq C}\mid X_B)\le \epsilon$ for every $C\in {\mathcal{M}}_B$, we can simply subtract away that second piece of the reconstruction defect.

Footnotes

1. As defined in Definition 4.3 of Sam Eisenstat, Condensation: a theory of concepts. Proposition 5.2 proves (\chi(B)\ge H(X_B)). ↩

2. Defined in 1.1 of Sam Eisenstat, “Research directions in condensation: varieties of objectivity”. ↩

3. Cf. Theorem 5.10 of Eisenstat, which characterizes the perfect condensation case by exact reconstruction from each contributing observable together with the ordered Markov condition. ↩

4. Expanding the definition of KL divergence under $P_B$ gives

   $$D_{\mathrm{KL}}(P_B\parallel Q_B)={\mathbb{E}}_{P_B}\left[\log \frac{P_B(Y_{{\mathcal{J}}_B})}{{\prod }_{A\in {\mathcal{J}}_B}{P}_B(Y_A\mid Y_{⊋A})}\right]$$

   $$={\mathbb{E}}_{P_B}\left[\log P_B(Y_{{\mathcal{J}}_B})\right]-\sum _{A\in {\mathcal{J}}_B}{\mathbb{E}}_{P_B}\left[\log P_B(Y_A\mid Y_{⊋A})\right]$$

   $$=-H(Y_{{\mathcal{J}}_B})+\sum _{A\in {\mathcal{J}}_B}H(Y_A\mid Y_{⊋A})$$ ↩

5. In this line $I$ is overloaded; it means first the mutual information and then the index set. ↩

6. For the reconstruction direction, cf. the chain-rule argument in footnote 7 of Gillen and Daniel C; we’re using the same argument here. For the Markov direction, cf. Proposition 5.9 and the proof of Theorem 5.10, $(B1)\Rightarrow (B2)$, in Eisenstat. ↩

7. Cf. the proof of Theorem 5.10, $(B2)\Rightarrow (B1)$, in Eisenstat. ↩