Suppose is the true density of a random sample while is the assumed model. The Kullback-Leibler distance is defined as
As we will show below the Kullback-Leibler information has a very useful property.
We know that , . Hence
notice that the rhs of the inequality can be rewritten as which (since a density integrates to one) is equal to
is the Hellinger metric.
We have thus just proved that
Hence the convergence of the Kullback-Leibler information always yields consistency in the Hellinger metric.