Kullback-Leibler (KL) divergence
KL divergence, also referred to as relative entropy, calculates the difference between two probability distributions.
It is also known as relative entropy or information divergence (I-divergence). It is not symmetric in the two distributions, meaning that the KL divergence between distribution P and distribution Q is generally not equal to the KL divergence between distribution Q and distribution P.
When comparing two distributions, KL divergence is helpful if one has a high variance or a limited sample size relative to the other.
Which is equivalent to
Additionally, KL divergence is asymmetric. In contrast to PSI, this indicates that the results will vary if the reference and production(compared) distributions are switched. Therefore, KL(P || Q) != KL(Q || P). This makes it handy for applications utilising Baye's theorem or when you have a large number of training (reference) samples but only a limited set of samples, resulting in increased variance in the comparison distribution.
A KL score can range from 0 to infinity, with 0 denoting equality between the two distributions. The output will be in "nats" if the natural log (base-e) is used, and "bits" if the KL formulae are taken to log base 2.
KL divergence can be used for both numerical and categorical features. The default KL score threshold for drift in AryaXAI is '0.05'.
Liked the content? you'll love our emails!
Is Explainability critical for your 'AI' solutions?
Schedule a demo with our team to understand how AryaXAI can make your mission-critical 'AI' acceptable and aligned with all your stakeholders.