Key Concepts of Mutual Information
Key concepts of mutual information (between random variables \(X\) and \(Y\)): Definition as KL divergence (dependence vs. independence): Mutual information \(I(X;Y)\) is the KL divergence between the joint distributi...
Key concepts of mutual information (between random variables \(X\) and \(Y\)): Definition as KL divergence (dependence vs. independence): Mutual information \(I(X;Y)\) is the KL divergence between the joint distribution \(P {XY}\) and the product of the marginal distributions \(P X P Y\) (i.e., the distribution if \(X\) and \(Y\) were independent). [:cite[1]{ln=4}], [:cite[2]{ln=4}] Entropy reduction interpretation: \(I(X;Y)=H(Y) H(Y\mid X)\), i.e., it equals the amount of entropy removed (on average) in \(X\) by observing \(Y\) (equivalently, reduced uncertainty). [:cite[3]{ln=2}], [:cite[4]{ln=2}] “Similarity” / dependence measure: Mutual information can be viewed as measuring the difference between the joint distribution of \((X,Y)\) and the distributions obtained when treating \(X\) and \(Y\) as independent. [:cite[3]{ln=3}], [:cite[4]{ln=3}] Nonnegativity (conditioning reduces entropy): Conditioning reduces entropy: \(H(X\mid Y)\le H(X)\) because \(I(X;Y)=H(X) H(X\mid Y)\ge 0\). [:cite[5]{ln=3}], [:cite[6]{ln=3}] Chain rule / additivity over sequences: Mutual information obeys chain rules; for example, it can be decomposed over sequences (e.g., \(I(X;Y 1^n)=H(Y 1^n) H(Y 1^n\mid X)=\sum i H(Y i\mid Y^{i 1})\)). [:cite[8]{ln=1}], [:cite[7]{ln=1}] Data processing inequality (information can only decrease through a channel): For a Markov chain \(X\to Y\to Z\), mutual information satisfies \(I(X;Z)\le I(X;Y)\). [:cite[10]{ln=4}], [:cite[9]{ln=4}]