各种熵

熵

$$H(X) = -\sum_i p(x_i)logp(x_i)$$

条件熵

$$H(X|Y) = -\sum_i p(x_i,y_i)logp(x_i|y_i) = H(X,Y)-H(Y)$$

互信息

$$I(X,Y) = H(X) - H(X|Y)$$

交叉熵

$$H(p,q) = -\sum p(x)logq(x)$$

相互熵

$$D_{KL}(p||q) = -\sum p(x)log\frac{q(x)}{p(x)} =H(p,q)-H(p)$$