Kalman filter to minimize the entropy value of random error with non-Gaussian distribution

Introduction

On Habr, the mathematical description of the work of the Kalman filter and the features of its application were considered in the following publications [1 ÷ 10]. In the publication [2], the algorithm of the Kalman filter (FC) in the state space model is considered in a simple and intelligible form. It should be noted that the study of control systems and control in the time domain using state variables has been widely used recently due to the simplicity of the analysis [eleven].

Publication [8] is of considerable interest just for learning. The author’s methodical technique is very effective, he began his article by considering the Gauss random error distribution, considered the FC algorithm and finished with a simple iteration formula for selecting the FC gain coefficient. The author confined himself to considering the Gauss distribution, motivating this by the fact that for sufficiently large $$$n$ (multiple measurements) the distribution law of the sum of random variables tends to the Gauss distribution.

Theoretically, such an assertion is certainly true, but in practice the number of measurements at each point of the range cannot be very large. RE Kalman himself obtained results on the minimum covariance of the filter based on orthogonal projections, without assuming that the measurement errors are Gaussian [12].

The purpose of this publication is to investigate the capabilities of the Kalman filter to minimize the entropy value of the random error with a non-Gaussian distribution.
To assess the effectiveness of the Kalman filter in identifying the distribution law or by superposing laws based on experimental data, we use the information theory of measurements based on the information theory of C. Shannon, according to which information, like a physical quantity, can be measured and evaluated.

The main advantage of the information approach to the description of measurements is that the size of the entropy uncertainty interval can be found strictly mathematically for any distribution law. This eliminates the historically established arbitrariness, which is unavoidable with the volitional assignment of different values ​​of the confidence probability. This is especially important in the educational process, when a teacher can observe a decrease in measurement uncertainty when applying Kalman filtering on a given numerical sample [13,14].

In addition, the set of probabilistic and informational characteristics of the sample can more accurately determine the nature of the distribution of random error. This is explained by the extensive base of the numerical values ​​of such parameters as the entropy coefficient and the counter-process for the various distribution laws and their superposition.

Evaluation of the superposition of the laws of the distribution of a random variable according to the entropy coefficient and the counter-process (obtained from experimental data)

The density of the probability distribution for each bar of the histogram [14] width $$$d$ equals $$$p_ {i} (x_ {i}) = n_ {i} / (n \ cdot d)$ , hence the estimate of the entropy probabilities is defined as $$$H \ left (x \ right) = \ int _ {- \ infty} ^ {+ \ infty} p \ left (x \ right) \ ln p \ left (x \ right) dx$ when finding the entropy on the histogram of $$$m$ columns we get the ratio:

$$$\ displaystyle H \ left (x \ right) = - \ sum_ {i = 1} ^ {m} \ int _ {\ tilde {x_ {i}} - \ frac {d} {2}} ^ {\ tilde { x_ {i}} + \ frac {d} {2}} \ frac {n_ {i}} {nd} \ ln \ frac {n_ {i}} {nd} = \ sum_ {i = 1} ^ {m } \ frac {n_ {i}} {n} \ ln \ frac {n} {n_ {i}} + \ ln d$

Imagine an expression for the estimation of entropy in the form:

$$$H \ left (x \ right) = \ ln \ left [d \ prod_ {i = 1} ^ {m} \ left (\ frac {n} {n_ {i}} \ right) ^ {\ frac {n_ {i}} {n}} \ right]$

We obtain an expression for estimating the entropy value of a random variable:

$$$\ Delta _ {e} = \ frac {1} {2} e ^ {H \ left (x \ right)} = \ frac {dn} {2} 10 ^ {- \ frac {1} {n} \ sum_ {1} ^ {m} n_ {i} \ lg n_ {i}}$

The classification of the distribution laws is carried out on the plane in the coordinates of the entropy coefficient $$$k = \ frac {\ Delta _ {e}} {\ sigma}$ and counter-procession $$$\ psi = \ frac {\ sigma ^ {2}} {\ sqrt {\ mu _ {4}}}$ Where $$$\ mu _ {4} = \ frac {1} {n} \ sum_ {1} ^ {n} \ left (x_ {i} - \ bar {X} \ right) ^ {4}$ .

For all possible distribution laws, \ psi varies from 0 to 1, and k from 0 to 2.066, so each law can be characterized by a certain point. Let's show it using the following program:




On a plane in coordinates $$$k, \ psi$ removed from the remaining distributions, the Pareto and Cauchy distributions, although they belong to different fields of application, are the first in physics and the second in economics. For comparison, one should choose the normal Gauss distribution at the top of the classification. All the comparisons given below are performed on a limited sample and are in the nature of demonstrating the possibilities of FC by the example of the numerical determination of the entropy error.

Selection of the Kalman filter algorithm

At each selected point of the measurement range, multiple measurements are taken and their result is compared with the measure that the FC does not know. Therefore, you should choose FC, for example Kalman Filter to Estimate a Constant [16] However, I prefer Python and settled on option [16] with extensive documentation. I will give a description of the algorithm:

Since the constant is always one model of the system can be represented as:

$$$x_ {k} = x_ {k-1} + w_ {k}$ , (one)

For the model, the transition matrix degenerates into one, and the control matrix to zero. The measurement model takes the form:

$$$y_ {k} = y_ {k-1} + v_ {k}$ , (2)

For model (2), the measurement matrix is ​​transformed into one, and the covariance matrices $$$P, Q, R$ turn into dispersions. At the next $$$k$ step, before the arrival of the measurement results, the Kalman scalar filter tries using the formula (1) to evaluate the new state of the system:

$$$\ hat {x} _ {k / (k-1)} = \ hat {x} _ {(k-1) / (k-1)}$ , (3)

Equation (3) shows that the a priori estimate in the next step is equal to the a posteriori estimate made in the previous step. A priori error variance estimate:

$$$P_ {k / (k-1)} = P _ {(k-1) / (k-1)} + Q_ {k}$ , (four)

By a priori assessment of the state $$$\ hat {x} _ {k / (k-1)}$ You can calculate the measurement forecast:

$$$\ hat {y} _ {k} = \ hat {x} _ {k / (k-1)}$ , (five)

After getting the next measurement $$$y_ {k}$ , the filter calculates its forecast error $$$k$ th measurement:

$$$e_ {k} = y_ {k} - \ hat {y} _ {k} = y_ {k} - \ hat {x} _ {k / (k-1)}$ , (6)

The filter adjusts its assessment of the state of the system, choosing a point somewhere between the initial assessment $$$\ hat {x} _ {k / (k-1)}$ and the point corresponding to the new dimension $$$y_ {k}$ :

$$$e_ {k} = y_ {k} - \ hat {y} _ {k} = y_ {k} - \ hat {x} _ {k / (k-1)}$ , (7)

Where $$$G _ {k}$ - filter gain.

The error variance estimate is also corrected:

$$$P_ {k / (k)} = (1-G_ {k}) \ cdot P_ {k / (k-1)}$ , (eight)

It can be proved that the variance of the error $$$e_ {k}$ equals:

$$$S_ {k} = P_ {k / (k-1)} + R_ {k}$ , (9)

The gain of the filter at which the minimum error of the system state is attained is determined from the relationship:

$$$G_ {k} = P_ {k / (k-1)} / S_ {k}$ , (ten)

Minimization of entropy error of FC for noise with Cauchy, Pareto and Gauss distribution

1. In probability theory, the Cauchy distribution density is determined from the relation:

$$$f (x) = \ frac {b} {\ pi \ cdot (1-x ^ {2})}$

For this distribution it is impossible to estimate the error by the methods of probability theory ( $$$\ sigma = \ infty$ ) but the information theory allows you to do this:




The view of the graph may change both when the program is restarted (new generation of the distribution sample) and depending on the number of measurements and distribution parameters, but one thing remains the same; the FK minimizes the value of the entropy error calculated on the basis of the information theory of measurements. For the above graph, FC reduces the entropy error by 3.9 times.

2. In probability theory, the Pareto distribution density with parameters $$$x_ {m}$ and $$$k$ determined from the relationship:

$$

$$f_ {X} (x) = \ left \ {\ begin {matrix} \ frac {kx_ {k} ^ {m}} {x ^ {k + 1}}, & x \ geq x_ {m} \\ 0, & x <x_ {m} \ end {matrix} \ right.$$

It should be noted that the Pareto distribution is found not only in the economy. You can give the following example of the distribution of the file size in Internet traffic by TCP protocol.

3. In probability theory, the density of the normal distribution (Gauss) with the expectation $$$\ mu$ and standard deviation $$$\ sigma$ determined from the relationship:

$$$f (x) = \ frac {1} {\ sigma \ sqrt {2 \ pi}} \ cdot e ^ {- \ frac {(x- \ mu) ^ {2}} {2 \ sigma ^ {2} }}$

The definition of the FK minimization of the entropy error from noise with the Gaussian distribution is given for comparison with the non-Gaussian Cauchy and Pareto distributions.




The Gaussian distribution provides a higher stability of the result with 50 measurements and for the given graph the entropy error decreases 2.2 times.

Minimization of the FC of entropy error in a sample of experimental data with an unknown noise distribution law




When analyzing a sample of experimental data, we obtain stable results on minimizing the PK of entropy error. For this sample, the FC reduces the entropy error by 4.85 times.

Conclusion

All comparisons given in this article were carried out on limited data samples, therefore, we should refrain from fundamental conclusions, however, the use of entropy error allows us to quantify the effectiveness of the Kalman filter in the above implementation, therefore the training task of this article can be considered complete.