KL-distance from N_μ1,σ1 to N_μ2,σ2

The general form is

 

∫_x { pdf₁(x).{ log(pdf₁(x)) - log(pdf₂(x)) }}

 

we have two normals so pdf₁(x) is N_μ1,σ1(x), etc..

 

= ∫_x N_μ1,σ1(x).{ log(N_μ1,σ1(x)) - log(N_μ2,σ2(x)) }

 

= ∫_x N_μ1,σ1(x).{ (1/2)( - ((x-μ₁)/σ₁)² + ((x-μ₂)/σ₂)² ) + ln(σ₂/σ₁) }

 

can replace x with x+μ₁. The expected value of x² is σ₁². Terms that are odd in x, and otherwise symmetric about zero, cancel out over [-∞,∞] leaving the ...x² and ...constant terms.

 

= (1/2){ - (σ₁/σ₁)² + (σ₁/σ₂)² + ((μ₁-μ₂)/σ₂)² } + ln(σ₂/σ₁)

 

= { (μ₁ - μ₂)² + σ₁² - σ₂² } / (2.σ₂²) + ln(σ₂/σ₁)

 

This is zero if μ₁=μ₂ and σ₁=σ₂. It obviously increases with |μ₁-μ₂| and has rather complex behaviour with σ₁ and σ₂  (and is consistent P&R, and with J&S where σ₁=σ₂).

KL(N(μ_q,σ_q) || N(μ_p,σ_p)), p.18 of Penny & Roberts, PARG-00-12, 2000.

KL(N(μ₁,σ), N(μ₂,σ)) = (μ₁-μ₂)²/(2σ²), Johnson & Sinanovic, NB. a common σ [...] .

---

 

Note that the distance is convenient to integrate over, say, a range of μ₁ & σ₁:

∫ μ1max ∫ σ1max        μ1min σ1min	(μ1 - μ2)2 2σ22	+ ln σ2 - 1/2	+	σ12 2σ22	- ln σ1
	NB. no σ1 here ...			... & no μ1

 

let

f(μ1) =

(μ1 - μ2)3 6σ22

+ μ1 . (ln σ2 - 1/2)

and	g(σ1) =	σ13 6σ22	- σ1 . (ln σ1 - 1)

 

= (f(μ_1max) - f(μ_1min)) . (σ_1max - σ_1min) + (μ_1max - μ_1min) . (g(σ_1max) - g(σ_1min))