Multistate, Multinomial

KL-distance from p₁ to p₂

E.g. Consider these two distributions over {A, C, G, T}:

	A	C	G	T
p1	1/4	1/4	1/4	1/4	pr
	2	2	2	2	-log pr, bits
p2	1/2	1/4	1/8	1/8	pr
	1	2	3	3	-log pr, bits

               1    1    1    1    1
KL(p1, p2) = - -1 + -0 + -1 + -1 = - bits
               4    4    4    4    4

               1    1    1    1    1
KL(p2, p1) =   -1 + -0 - -1 - -1 = - bits
               2    4    8    8    4

So KL(p₁, p₂) = KL(p₂, p₁) in this case -- but not in general.

 

 

Now you calculate the two distances for these two distributions:

	A	C	G	T
p1	1/2	1/4	1/8	1/8
p2	1/8	1/2	1/4	1/8

KL(p1, p2) = ...

KL(p2, p1) = ...