Fisher Information Properties

Abstract

A set of Fisher information properties are presented in order to draw a parallel with similar properties of Shannon differential entropy. Already known properties are presented together with new ones, which include: (i) a generalization of mutual information for Fisher information; (ii) a new proof that Fisher information increases under conditioning; (iii) showing that Fisher information decreases in Markov chains; and (iv) bound estimation error using Fisher information. This last result is especially important, because it completes Fano’s inequality, i.e., a lower bound for estimation error, showing that Fisher information can be used to define an upper bound for this error. In this way, it is shown that Shannon’s differential entropy, which quantifies the behavior of the random variable, and the Fisher information, which quantifies the internal structure of the density function that defines the random variable, can be used to characterize the estimation error.

1. Introduction

The birth of information theory was signaled by the publication of Claude Shannon’s work [1], which is based on studying the behavior of systems described by density functions. However, much before that work was published, Ronald Fisher had already published the definition of a quantity called Fisher information [2], a hard bound on the capacity to estimate the parameters that define a system [3,4]. Hence, this quantity regulates how well it is possible to determine the internal structure of a system and provides another point of view that can be used to study systems: how they are composed, what they are made of. This work springs from the belief that the combination of these approaches is what completely defines systems: their behavior (Shannon) and their architecture (Fisher). In the following, a series of published results is summarized, together with new results, in order to present a coherent set of Fisher information properties that will hopefully be useful for those that work with this quantity.

1.1. Fisher Information and Other Fields

One connection between Fisher information and the Shannon differential entropy was stated by Kullback [5] (p. 26), who proved that the second derivatives of the Kullback–Leibler divergence with respect to the density functions parameters produce the Fisher information matrix terms. Related results were presented by Blahut [6] (p. 300), and Frieden [7] (p. 37). Another important result that also relates these two frameworks is Bruijn’s identity ([8,9] and [10] (p. 672)), which establishes a relation between the derivative of Shannon differential entropy and Fisher information when the underlying random variable is the subject of Gaussian perturbations. This result was recently generalized to non-Gaussian perturbations [11,12]. A consequence of these results is the convolution inequality for Fisher information ([8,9,13–16]; [10] (p. 674)).

Others have been studying the relation between Fisher information and physics. Here, it is important to point out the extreme physical information principle derived by Frieden and others in order to establish a general framework that explains physics [7,17–20]. Of special interest has been the role of Fisher information to generate thermodynamical theory [7,17–22]. It is very common in these approaches to use a special case of Fisher information where the estimated parameter is a location parameter. In this work, the original and general Fisher information definitions, and not the later special case, are addressed only.

Even thought Shannon’s ideas have been part of the the machine learning tool set for a long time, Fisher information has not followed the same track. Even though Fisher information is intimately connected to estimation theory [23], its use in the development of learning systems has not been well developed yet. Nevertheless, Amari discovered that natural gradient descent, i.e., common gradient descent corrected with the Fisher information matrix terms, takes into account the topology in a more precise manner, allowing for more efficient training procedures [24,25]. The use of Fisher information has also been taken into account in order to design objective functions to lead the estimation procedure. One of them is mixing maximum entropy with minimum Fisher information [26,27]. On the other hand, mixing Shannon’s differential entropy, Fisher information and the central limit theorem has allowed proving that in the presence of large datasets, it is natural to search for minimum Kullback–Leibler, or equivalent, solutions [28].

1.2. Contribution of This Work

This work is focused on presenting already known properties of Fisher information [3,4,7,8,10,29–32] and introducing new ones, such that the reader can have a better grasp of Fisher information and its usefulness. The main results presented in this work are: (i) the generalization of the mutual information concept using Fisher information expressions; (ii) a new proof that conditioning under certain assumptions increases Fisher information; (iii) proving that in Markov chains, the Fisher information increases as the random variables become further away from the estimated parameter; and (iv) an upper bound on estimation error, which is regulated by the Fisher information.

This work is structured roughly in the same way in which is organized the first chapter of the well-known book of Cover and Thomas [30], in order to help the reader to draw a parallel between Shannon and Fisher information.

2. Notation

In the following sections, vectors and matrices are denoted with a bold font [7,31]. Furthermore, density functions are denoted by f_X;θ ≡ f_X;θ(x), where the f is reserved for density functions, the lowercase X corresponds to the name of the random variable, θ represents the parameters that define the density function and the symbol within the (·) stands for the instance of the random variable that is used to evaluate the density function. In this way, as an example, a different random variable could be denoted by f_Y;θ ≡ f_Y;θ(y). A similar notation is used in [33].

3. Fisher Information

Let there be a random variable X and its associated density function f_X;θ ≡ f_X;θ(x), which has a support $\mathrm{S}$, and it depends on a set of parameters that is represented by the vector θ ∈ Θ. The value θ_k is the k-th component of θ. According to the original definition designed by Fisher to characterize maximum likelihood estimation [2]:

Definition 1 (Fisher Information). Given a random variable X and its associated density function f_X;θ(x), which depends on the parameter vector θ ∈ Θ, and θ_k is the k-th component of θ, then the Fisher information associated with θ_k is defined by:

$$i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \int f_{\mathrm{X};\theta }}\left(\mathrm{X}\right){\left(\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{X}\right)}{\partial {\theta }_k}\right)}^{2}dx$$

(1)

From the definition, it is clear that $i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}\ge 0$. Furthermore, if f_X does not depend on θ_k, then $i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}=0$.

Example 1. In a Gaussian case with mean µ and standard deviation η, the density function is given by:

$$f_{X;\mu ,\eta }\left(x\right)=\frac{1}{\sqrt{2\pi }\eta }\exp \left(-\frac{{\left(x-\mu \right)}^2}{{\eta }^2}\right)$$

(2)

In this case:

$$f_{X;\mu ,\eta }\left(x\right)=\ln \frac{1}{\sqrt{2\pi }\eta }-\frac{{\left(x-\mu \right)}^2}{2{\eta }^2}$$

(3)

$$\ln \frac{1}{\sqrt{2\pi }\eta }-\frac{x^2-2\mu x+{\mu }^2}{2{\eta }^2}$$

(4)

If the parameter to be estimated is the mean µ, the previous expression needs to be derived with respect to µ:

$$\frac{d\ln f_{\mathrm{X};\mu ,\eta }\left(x\right)}{d\mu }=\frac{2x}{2{\eta }^2}-\frac{2\mu }{2{\eta }^2}$$

(5)

$$=\frac{x-\mu }{{\eta }^2}$$

(6)

Replacing into the definition of Fisher information definition:

$$i_F{\left(f_{X;\mu ,\eta }\right)}_{\mu }={\displaystyle \int f_{X;\mu ,\eta }\left(x\right)}{\left(\frac{x-\mu }{{\eta }^2}\right)}^{2}dx$$

(7)

$$={\displaystyle \int f_{X;\mu ,\eta }\left(x\right)}\frac{x^2-2\mu x+{\mu }^2}{{\eta }^4}dx$$

(8)

$$=\frac{1}{{\eta }^4}{\displaystyle \int f_{X;\mu ,\eta }\left(x\right)}{x}^{2}dx-\frac{2\mu }{{\eta }^4}\pm {\displaystyle \int f_{X;\mu ,\eta }\left(x\right)xdx}+\frac{{\mu }^2}{{\eta }^4}{\displaystyle \int f_{X;\mu ,\eta }\left(x\right)dx}$$

(9)

$$=\frac{1}{{\eta }^4}\left\{\left({\eta }^2+{\mu }^2\right)-2{\mu }^2+{\mu }^2\right\}$$

(10)

$$=\frac{1}{{\eta }^2}$$

(11)

This shows that for Gaussian functions, the variance of any estimator of the mean is directly proportional to the variance of the density function.

There is another expression that can be used to represent the Fisher information.

Theorem 1. Given a random variable X and its associated density function f_X;θ(x), which depends on the parameter vector θ ∈ Θ and complies with the boundary condition for θ_k (see Appendix A), where θ_k is the k-th component of θ, then the Fisher information associated with θ_k is equal to:

$$i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}=-{\displaystyle \int f_{\mathrm{X};\theta }(\mathrm{X})}\frac{{\partial }^2\ln f_{X;\theta }(\mathrm{X})}{\partial {\theta }_k^2}dx$$

(12)

A proof of this theorem can be found in [34] (p. 373).

Example 2. Continuing the Gaussian example, and using the alternative definition of the Fisher information, the required second derivative is first calculated:

$$\frac{d^2\ln f_{X;\mu ,\eta }\left(x\right)}{d{\mu }^2}=\frac{d}{d\mu }\left(\frac{x-\mu }{{\eta }^2}\right)=-\frac{1}{{\eta }^2}$$

(13)

Replacing into Equation (12), the same result is obtained:

$$i_F{\left(f_{X;\mu ,\eta }\right)}_{\mu }=-{\displaystyle \int f_{X;\mu ,\eta }\left(x\right)}\frac{d^2\ln f_{X;\mu ,\eta }\left(x\right)}{d{\mu }^2}dx$$

(14)

$$=-{\displaystyle \int f_{X;\mu ,\eta }\left(x\right)\left(-\frac{1}{{\eta }^2}\right)}dx$$

(15)

$$=\frac{1}{{\eta }^2}{\displaystyle \int dx{f}_{X;\mu ,\eta }\left(x\right)}$$

(16)

$$=\frac{1}{{\eta }^2}$$

(17)

The importance of the Fisher information quantity stems from the Cramer–Rao bound [3,4,23,35]:

Theorem 2 (Cramer–Rao Bound). Given a random variable X and its associated density function f_X;θ(x), which depends on the parameter vector θ ∈ Θ and complies with the boundary condition for θ_k (see Appendix A), where θ_k is the k-th component of θ, also given that there is an unbiased estimator${\widehat{\theta }}_k\left(\mathrm{x}\right)$of the scalar parameter θ_k, then:

$$\frac{1}{i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}}\le {\sigma }_{{\widehat{\theta }}_k}^2$$

(18)

where:

$${\sigma }_{{\widehat{\theta }}_k}^2\equiv {\displaystyle \int f_{\mathrm{X};\theta }}\left(\mathrm{x}\right){\left({\widehat{\theta }}_k\left(\mathrm{x}\right)-{\theta }_k\right)}^{2}dx$$

(19)

is the variance of the estimator. Proofs of this theorem can be found in [7] (p. 29) and [23] (p. 66).

The Cramer–Rao bound establishes that the reciprocal of the Fisher information is a lower bound of the variance of an estimator. Any estimator that reaches the bound imposed by the Cramer–Rao theorem is called efficient [34]. It is important to notice that the bound does not depend on the estimator itself; it only depends on $i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}$. In this work, the case of biased estimators will not be analyzed, nor when the parameters themselves are random variables.

The following theorem states that the topology of the Fisher information in the density function space is very simple:

Theorem 3. The Fisher information$i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}$is convex in f_X;θ. Proofs of this theorem can be found in [7] (p. 69) and [29].

4. Several Random Variables Depending on θ_k

4.1. Joint Fisher Information Definition

Definition 2. Given two random variables X and Y and the associated joint density function f_X,Y;θ(x, y), which depends on the parameter vector θ ∈ Θ, and θ_k is the k-th component of θ, then the joint Fisher information associated with θ_k is defined by:

$$i_F{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }}{\left(\frac{\partial \ln f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\partial {\theta }_k}\right)}^{2}dxdy$$

(20)

4.2. An Equivalent Joint Fisher Information Definition

Theorem 4. Given two random variables X and Y and the associated joint density function f_X,Y;θ(x, y), which depends on the parameter vector θ ∈ Θ and complies with the boundary condition for θ_k (see Appendix A), where θ_k is the k-th component of θ, then the joint Fisher information associated with θ_k is equal to:

$$i_F{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=-{\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}\frac{{\partial }^2\ln f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\partial {\theta }_k^2}dxdy$$

(21)

Proof. This follows trivially from the alternative definition of the Fisher information. □

4.3. Conditional Fisher Information Definition

Definition 3.

$$i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}\left(\frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}\right)dxdy$$

(22)

4.4. Chain Rule for Two Random Variables

The following result was first published by Zamir [32], who used it to produce an alternative proof of the Fisher information inequality. In the following lines, the same chain rule is proven using the results presented in the previous sections.

Theorem 5 (Chain Rule for Two Random Variables). Given a joint density function f_X,Y;θ(x, y), which depends on the parameter vector θ ∈ Θ, and given that the density functions comply with the boundary condition for θ_k (see Appendix A), where θ_k is the k-th component of θ, then:

$$i_{\mathrm{F}}{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}+i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}$$

(23)

$$=i_F{\left(f_{\mathrm{X}|\mathrm{Y};\theta }\right)}_{{\theta }_k}+i_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}$$

(24)

respectively.

Proof.

$$i_F{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\left(\frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\partial {\theta }_k}\right)}^{2}dxdy$$

(25)

$$={\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\left(\frac{\partial \ln \left(f_{\mathrm{Y}|\mathrm{X};\theta }|\left(\mathrm{x},\mathrm{y}\right)f_{\mathrm{X};\theta }\left(\mathrm{x}\right)\right)}{\partial {\theta }_k}\right)}^{2}dxdy$$

(26)

$$={\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}{\left(\frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }|\left(\mathrm{x},\mathrm{y}\right)}{\partial {\theta }_k}+\frac{\partial \ln f_{\mathrm{X};\theta }|\left(\mathrm{x}\right)}{\partial {\theta }_k}\right)}^{2}dxdy$$

(27)

$$=i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}+i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}+2{\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)}\frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\partial {\theta }_k}dxdy$$

(28)

but,

$${\displaystyle \iint \frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\partial {\theta }_k}dxdy={\displaystyle \iint \frac{\partial \ln f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\partial {\theta }_k}dxdy}}$$

(29)

$$={\displaystyle \int \left(\frac{\partial f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}{\displaystyle \int \frac{\partial f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}dy}\right)}dx$$

(30)

If $f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)$complies with the boundary condition with respect to θ_k (see Appendix A), then:

$${\displaystyle \int \frac{\partial f_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}{\partial {\theta }_k}dy}=\frac{\partial }{\partial {\theta }_k}{\displaystyle \int dy{f}_{\mathrm{Y}|\mathrm{X};\theta }\left(\mathrm{y}|\mathrm{x}\right)}=0$$

(31)

Therefore, the theorem is proven. The other result is proven analogously. □

When the chain rule is used to estimate the Fisher information associated with a parameter, it is important to take into account that all of the terms that come out after applying the chain rule contain derivatives with respect to the same parameter. Because some of these terms may be dependent on density functions that do not depend on the parameter, some of these terms may be equal to zero.

Example 3. Given the random variable Y = X + N, where X is a Gaussian density function with mean µ and standard deviation η and N another Gaussian density function with mean zero and standard deviation ν, if the joint density function is available, and the parameter to be estimated is µ, then:

$$i_F{\left(f_{\mathrm{Y},\mathrm{X};}{}_{\mu ,\eta ,\nu }\right)}_{\mu }=i_F{\left(f_Y{}_{|X;\mu ,\eta ,\nu }\right)}_{\mu }+i_F{\left(f_{\mathrm{X};\mu ,\eta }\right)}_{\mu }$$

(32)

$$=i_F{\left(f_{N;\nu }\right)}_{\mu }+i_F{\left({\mathrm{f}}_{X;\mu ,\eta }\right)}_{\mu }$$

(33)

$$=i_F{\left(f_{X;\mu ,\eta }\right)}_{\mu }$$

(34)

$$=\frac{1}{{\eta }^2}$$

(35)

The previous result implies that if the joint density function of the output Y and the input X is available, the noise does not affect the estimation process. This is not surprising, since Y is a corrupted version of X, and it cannot shed more information on µ than that contained in X. Because all of the information hidden in X is available through the joint density function, it makes sense to think that the Fisher information of the joint density function corresponds to that of the marginal distribution f_X;µ,η.

Given the density functions mentioned above, it is possible to prove that:

$$f_{Y;\mu ,\eta ,\nu }\left(y\right)=\frac{1}{\sqrt{2\pi \left({\eta }^2+{\nu }^2\right)}}\exp \left(-\frac{{\left(y-\mu \right)}^2}{2\left({\eta }^2+{\nu }^2\right)}\right)$$

(36)

with Fisher information associated with μ equal to:

$$i_F{\left(f_{Y;\mu ,\eta ,\nu }\right)}_{\mu }=\frac{1}{{\eta }^2+{\nu }^2}$$

(37)

Using the other expression for the chain rule:

$$i_F{\left(f_{\mathrm{Y},\mathrm{X};}{}_{\mu ,\eta ,\nu }\right)}_{\mu }=i_F{\left(f_X{}_{|Y;\mu ,\eta ,\nu }\right)}_{\mu }+i_F{\left(f_{\mathrm{Y};\mu ,\eta ,\nu }\right)}_{\mu }$$

(38)

$$=i_F{\left(f_X{}_{|Y;\mu ,\eta ,\nu }\right)}_{\mu }+\frac{1}{{\eta }^2+{\nu }^2}$$

(39)

Using the previous results:

$$\frac{1}{{\eta }^2}=i_F{\left(f_{X|Y;\mu ,\eta ,\nu }\right)}_{\mu }+\frac{1}{{\eta }^2+{\nu }^2}$$

(40)

which implies:

$$i_F{\left(f_{Y;\mu ,\eta ,\nu }\right)}_{\mu }=\frac{{\nu }^2}{{\eta }^2\left({\eta }^2+{\nu }^2\right)}$$

(41)

4.5. Chain Rule for Many Random Variables

In the case of more than two density functions:

Theorem 6 (Chain Rule for Many Random Variables). Given a set of n random variables X₁, X₂, …, X_n, all of them depending on θ_k, if the density functions comply with the boundary condition for θ_k (see Appendix A), then:

$$i_F{\left(f_{{\mathrm{X}}_1,{\mathrm{X}}_2,\dots ,{\mathrm{X}}_n;\theta }\right)}_{{\theta }_k}={\displaystyle \sum _{k=1}^n{i}_F{\left(f_{{\mathrm{X}}_k|{\mathrm{X}}_{k-1},\dots ,{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}}$$

(42)

Proof.

$$i_F{\left(f_{{\mathrm{X}}_1,{\mathrm{X}}_2,\dots ,{\mathrm{X}}_n;\theta }\right)}_{{\theta }_k}=i_F{\left(f_{{\mathrm{X}}_n,\dots ,{\mathrm{X}}_2|{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}+i_F{\left(f_{{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}$$

(43)

$$i_F{\left(f_{{\mathrm{X}}_1,{\mathrm{X}}_2,\dots ,{\mathrm{X}}_n;\theta }\right)}_{{\theta }_k}=i_F{\left(f_{{\mathrm{X}}_n,\dots ,{\mathrm{X}}_3{|\mathrm{X}}_2,{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}+i_F{\left(f_{{\mathrm{X}}_2|{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}+i_F{\left(f_{{\mathrm{X}}_1;\theta }\right)}_{{\theta }_k}$$

(44)

$$⋮$$

$$={\displaystyle \sum _{k=1}^n{i}_F{\left(f_{{\mathrm{x}}_k|{\mathrm{x}}_{k-1},\dots ,{\mathrm{x}}_{1;\theta }}\right)}_{{\theta }_k}}$$

(45)

□

If the n random variables in Theorem 6 are i.i.d., then $i_F{\left(f_{{\mathrm{X}}_1,{\mathrm{X}}_2,\dots ,{\mathrm{X}}_n;\theta }\right)}_{{\theta }_k}=n\cdot i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}$.

5. Relative Fisher Information Type I

In the following, the relative Fisher information is defined. As far as it was possible to determine, the first definition of the relative Fisher information was given by Otto and Villani [36], who defined it for the translationally-invariant case. Furthermore, this expression has been rediscovered or simply used in many applications thereafter in different problems and fields [22,37–44]. Furthermore, it seems that the first general analysis of the relative Fisher information was presented by the author in [45]. The following sections focus on this latter general case, where there is no assumption of translational invariance.

Analogously to the Kullback–Leibler divergence [46], also known as as relative entropy, which was designed to established how much two density functions differed, the relative Fisher information of Type I is obtained when the ratio of two intervening density functions is replaced into Equation (1), as is shown in the following definition.

Definition 4. The relative Fisher information Type I is defined by:

$$d_{F\left(I\right)}{\left(f_{\mathrm{X};\theta }\left|\right|f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \int f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\left(\frac{\partial }{\partial {\theta }_k}\left(\ln \left(\frac{f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{f_{\mathrm{Y};\theta }\left(\mathrm{x}\right)}\right)\right)\right)}^{2}dx$$

(46)

The same mechanism can be used to generate a second definition for the relative Fisher information. The same ratio can be replaced into Equation (12), producing an alternative and equally valid expression, which is designated as relative Fisher information Type II. This second expression is studied in the following sections.

6. Information Correlation

Definition 5. The information correlation with respect to θ_k is defined by:

$$i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }\left(\mathrm{x},\mathrm{y}\right)\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\partial {\theta }_k}}\frac{\partial \ln f_{\mathrm{Y};\theta }}{\partial {\theta }_k}dxdy$$

(47)

The name information correlation comes from the similarity between this definition and that of the classical correlation coefficient. It is important to keep in mind that it is different from the terms that fill the Fisher information matrix [23].

According to the definition $i_C{\left(f_{\mathrm{X},\mathrm{X};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}$, and. $i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=i_C{\left({\mathrm{f}}_{\mathrm{Y},\mathrm{X};\theta }\right)}_{{\theta }_k}$.

Example 4. Continuing with the example where Y = X + N, the information correlation between Y and X is given by:

$$i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }={\displaystyle \iint f_{Y,X;\mu ,\eta ,\nu }\left(y,x\right)}\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }\frac{d\ln f_{X;\mu ,\eta }\left(x\right)}{d\mu }dydx$$

(48)

$$={\displaystyle \iint f_{Y,X;\mu ,\eta ,\nu }\left(y|x\right)f_{X;\mu ,\eta }\left(x\right)}\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }\frac{d\ln f_{X;\mu ,\eta }\left(x\right)}{d\mu }dydx$$

(49)

$$={\displaystyle \iint f_{N;\nu }\left(y-x\right)f_{X;\mu ,\eta }\left(x\right)}\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }\frac{d\ln f_{X;\mu ,\eta }\left(x\right)}{d\mu }dydx$$

(50)

$$={\displaystyle \iint \left(\frac{1}{\sqrt{2\pi }\nu }\exp \left(-\frac{{\left(y-x\right)}^2}{2v^2}\right)\right)\left(\frac{1}{\sqrt{2\pi }\eta }\exp \left(-\frac{{\left(x-\mu \right)}^2}{2{\eta }^2}\right)\right)}\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }\frac{d\ln f_{X;\mu ,\eta }\left(x\right)}{d\mu }dydx$$

(51)

$$=\frac{1}{2\pi \nu \eta }{\displaystyle \iint \exp \left(-\frac{1}{2}\left(\frac{{\left(y-x\right)}^2}{v^2}+\frac{{\left(x-\mu \right)}^2}{{\eta }^2}\right)\right)}\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }\frac{d\ln f_{X;\mu ,\eta }\left(x\right)}{d\mu }dydx$$

(52)

where:

$$\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }=\frac{d}{d\mu }\left(\ln \frac{1}{\sqrt{2\pi }\sqrt{{\eta }^2+{\nu }^2}}-\frac{{\left(y-\mu \right)}^2}{2\left({\eta }^2+{\nu }^2\right)}\right)$$

(53)

$$=\frac{\left(y-\mu \right)}{{\eta }^2+{\nu }^2}$$

(54)

Analogously:

$$\frac{d\ln f_{Y;\mu ,\eta ,\nu }\left(y\right)}{d\mu }=\frac{\left(x-\mu \right)}{{\eta }^2}$$

(55)

Replacing these derivatives into the information correlation expression:

$$i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }=\frac{1}{2\pi \nu \eta }{\displaystyle \iint \exp }\left(-\frac{1}{2}\left(\frac{{\left(y-x\right)}^2}{v^2}+\frac{{\left(x-\mu \right)}^2}{{\eta }^2}\right)\right)\left(\frac{y-\mu }{{\eta }^2+{\nu }^2}\right)\left(\frac{x-\mu }{{\eta }^2}\right)dydx$$

(56)

$$=\frac{1}{2\pi \nu \eta \left({\eta }^2+{\nu }^2\right){\eta }^2}{\displaystyle \iint \left(y-\mu \right)\left(x-\mu \right)\exp }\left(-\frac{1}{2}\left(\frac{{\left(y-x\right)}^2}{v^2}+\frac{{\left(x-\mu \right)}^2}{{\eta }^2}\right)\right)dydx$$

(57)

$$=\frac{1}{2\pi \nu \eta \left({\eta }^2+{\nu }^2\right){\eta }^2}{\displaystyle \int \left(x-\mu \right)\exp }\left(-\frac{1}{2}\frac{{\left(x-\mu \right)}^2}{{\eta }^2}\right)\left({\displaystyle \int \left(y-x\right)\exp \left(-\frac{1}{2}\frac{{\left(y-x\right)}^2}{{\nu }^2}\right)dy}\right)dx$$

(58)

$$\frac{1}{{\eta }^2+{\nu }^2}$$

(59)

Theorem 7. The information correlation is bounded according to:

$${\left(i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}\right)}^2\le i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}{i}_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}$$

(60)

Proof.

$$0\le {\displaystyle \iint f_{\mathrm{X},\mathrm{Y};\theta }}\left(\mathrm{x},\mathrm{y}\right)\left(a\frac{\partial \ln f_{\mathrm{X};\theta }\left(\mathrm{x}\right)}{\partial {\theta }_k}+\frac{\partial \ln f_{\mathrm{Y};\theta }\left(\mathrm{y}\right)}{\partial {\theta }_k}\right)dxdy$$

(61)

which can be reexpressed as:

$$0\le a^2{i}_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}+2a\cdot i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}+i_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}$$

(62)

This is a second degree equation that is true for every possible a. Because this equation is always greater than zero, the discriminant of the equation has to comply with $4{\left(i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}\right)}^2-4i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}{i}_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}\le 0,$ which proves the theorem. □

Definition 6. The information correlation coefficient is defined by:

$${\rho }_F=\frac{i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}}{\sqrt{i_F{\left(f_{\mathrm{X};\theta }\right)}_{{\theta }_k}{i}_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}}}$$

(63)

Theorem 8. The information correlation coefficient is limited by:

$$-1\le {\rho }_F\le 1$$

(64)

Proof. This comes from the definition of the information correlation coefficient and Theorem 7.

Theorem 9. If at least one of the following conditions:

• f_X;θ and f_Y;θ are independent.
• Either f_X;θ or f_Y;θ does not depend on θ_k.

is true, then:

$$i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=0$$

(65)

Proof. Examination of the information correlation definition clearly shows that compliance with the first and second cases directly implies that this quantity is zero.

7. Mutual Fisher Information Type I

As happens in Shannon’s differential entropy handling, in this work, mutual Fisher information is also defined as relative Fisher information Type I, where the argument is the ratio between a joint density function and the product of its marginals.

7.1. Definition

Definition 7. The mutual Fisher information Type I is defined by:

$$m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\theta }}(\mathbf{x},\mathbf{y})}{\left(\frac{\partial }{\partial {\theta }_k}\ln \left(\frac{f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})f_{\mathbf{Y};\theta }(\mathbf{y})}\right)\right)}^{2}d\mathbf{x}d\mathbf{y}$$

(66)

From the definition, it is obvious that $m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}\ge 0$.

Theorem 10. If the boundary condition (see Appendix A) with respect to θ_k holds for f_X,Y;θ(x,y), the mutual Fisher information Type I can be reformulated as a function of the Fisher information as follows:

$$m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{X}|\mathbf{Y};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{X};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}$$

(67)

$$=i_F{\left(f_{\mathbf{Y}|\mathbf{X};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{Y};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}$$

(68)

Proof.

$${\left(\frac{\partial }{\partial {\theta }_k}\ln \left(\frac{f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})f_{\mathbf{Y};\theta }(\mathbf{y})}\right)\right)}^2={\left(\frac{\partial \ln f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{\partial {\theta }_k}-\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}-\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\right)}^2$$

(69)

$$\begin{array}{l}={\left(\frac{\partial \ln f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{\partial {\theta }_k}\right)}^2+{\left(\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\right)}^2+{\left(\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\right)}^2-2\frac{\partial \ln f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\\ -2\frac{\partial \ln f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}+2\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\end{array}$$

(70)

$$\begin{array}{l}={\left(\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\right)}^2+2\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}+{\left(\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\right)}^2\\ +{\left(\frac{\partial \ln f_{\mathbf{X},\theta }(\mathbf{x})}{\partial {\theta }_k}\right)}^2+{\left(\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\right)}^2\\ -2\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}-2\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\\ -2\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}-2\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\\ -2\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}\end{array}$$

(71)

Simplifying:

$$\begin{array}{c}{\left(\frac{\partial }{\partial {\theta }_k}\ln \left(\frac{f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})f_{\mathbf{Y};\theta }(\mathbf{y})}\right)\right)}^2=\left(\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\right)+{\left(\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\right)}^2\\ -2\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\end{array}$$

(72)

Now,

$$\begin{array}{l}2{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}d\mathbf{x}d\mathbf{y}}}\\ =2{\displaystyle \int {\displaystyle \int \frac{f_{\mathbf{Y};\theta }(\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})}\frac{\partial f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}d\mathbf{x}d\mathbf{y}}}\end{array}$$

(73)

$$=2{\displaystyle \int \left(\frac{1}{f_{\mathbf{X};\theta }(\mathbf{x})}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}{\displaystyle \int f_{\mathbf{Y};\theta }(\mathbf{y})\frac{\partial f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}d\mathbf{y}}\right)}d\mathbf{x}$$

(74)

Assuming that f_X,Y;θ complies with the boundary condition (see Appendix A) with respect to θ_k, then:

$$\frac{\partial f_{\mathrm{X};\theta }(\mathrm{x})}{\partial {\theta }_k}=\frac{\partial }{\partial {\theta }_k}{\displaystyle \int f_{\mathrm{X},\mathrm{Y};\theta }}(\mathbf{x},\mathbf{y})d\mathbf{y}$$

(75)

$$={\displaystyle \int \frac{\partial f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}}d\mathbf{y}$$

(76)

$$={\displaystyle \int f_{\mathbf{Y};\theta }}(\mathbf{y})\frac{\partial f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}d\mathbf{y}+{\displaystyle \int f_{\mathbf{X}|\mathbf{Y};\theta }}(\mathbf{x}|\mathbf{y})\frac{\partial f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}d\mathbf{y}$$

(77)

Hence,

$$={\displaystyle \int f_{\mathbf{Y};\theta }}(\mathbf{y})\frac{\partial f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}d\mathbf{y}=\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}-{\displaystyle \int f_{\mathbf{X}|\mathbf{Y};\theta }}(\mathbf{x}|\mathbf{y})\frac{\partial f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}d\mathbf{y}$$

(78)

Using the previous result, it is obtained:

$$\begin{array}{c}2{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\theta }}}(\mathbf{x},\mathbf{y})\frac{\partial \ln f_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}d\mathbf{x}d\mathbf{y}\\ =2{\displaystyle \int \frac{1}{f_{\mathbf{X};\theta }(\mathbf{x})}}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}d\mathbf{x}-2{\displaystyle \int \frac{1}{f_{\mathbf{X};\theta }(\mathbf{x})}}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}{\displaystyle \int d\mathbf{y}}{f}_{\mathbf{X}|\mathbf{Y};\theta }(\mathbf{x}|\mathbf{y})\frac{\partial f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}d\mathbf{x}\end{array}$$

(79)

$$=2i_F{\left(f_{\mathbf{X};\theta }\right)}_{{\theta }_k}-2{\displaystyle \int {\displaystyle \int \frac{f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})f_{\mathbf{Y};\theta }(\mathbf{y})}}\frac{\partial f_{\mathbf{X};\theta }(\mathbf{x})}{\partial {\theta }_k}}\frac{\partial f_{\mathbf{Y};\theta }(\mathbf{y})}{\partial {\theta }_k}d\mathbf{x}d\mathbf{y}$$

(80)

This implies:

$$m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{X}|\mathbf{Y};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{X};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}$$

(81)

The other result is obtained analogously. □

Example 5. Continuing with the example where Y = X + N, the mutual Fisher information Type I is given by:

$$m_{F(I)}{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }=i_F{\left(f_{Y|X}\right)}_{{\theta }_k}-i_F{\left(f_Y\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }$$

(82)

$$=i_F{\left(f_N\right)}_{{\theta }_k}-i_F{\left(f_Y\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }$$

(83)

$$=\frac{1}{{\nu }^2}-\frac{1}{{\eta }^2+{\nu }^2}+2\cdot i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }$$

(84)

$$=\frac{{\eta }^2}{{\nu }^2({\eta }^2+{\nu }^2)}+2\cdot i_C{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{\mu }$$

(85)

$$=\frac{{\eta }^2}{{\nu }^2({\eta }^2+{\nu }^2)}+\frac{2}{{\eta }^2+{\nu }^2}$$

(86)

$$=\frac{1}{{\eta }^2+{\nu }^2}+\frac{1}{{\nu }^2}$$

(87)

7.2. Conditional Mutual Fisher Information of Type I

Definition 8. The conditional information correlation with respect to θ_k of random variables X and Y given random variable Z is defined by:

$$i_C{\left(f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y},\mathbf{Z};\theta }}}(\mathbf{x},\mathbf{y},\mathbf{z})\frac{\partial \ln f_{\mathbf{X}|\mathbf{Z};\theta }(\mathbf{x}|\mathbf{z})}{\partial {\theta }_k}\frac{\partial \ln f_{\mathbf{Y}|\mathbf{Z};\theta }(\mathbf{y}|\mathbf{z})}{\partial {\theta }_k}d\mathbf{x}d\mathbf{y}d\mathbf{z}$$

(88)

Definition 9. The conditional mutual Fisher information of Type I of random variables X and Y given random variable Z is defined by:

$$m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}\equiv {\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y},\mathbf{Z};\theta }}}(\mathbf{x},\mathbf{y},\mathbf{z}){\left(\frac{\partial }{\partial {\theta }_k}\left(\ln \left(\frac{f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }(\mathbf{x},\mathbf{y}|\mathbf{z})}{f_{\mathbf{X}|\mathbf{Z};\theta }(\mathbf{x}|\mathbf{z})f_{\mathbf{Y}|\mathbf{Z};\theta }(\mathbf{y}|\mathbf{z})}\right)\right)\right)}^{2}d\mathbf{x}d\mathbf{y}d\mathbf{z}$$

(89)

Corollary 1. If the boundary condition (see Appendix A) with respect to 0_k holds for f_X,Y,Z;θ(x, y, z), the conditional mutual Fisher information of Type I of random variables X and Y given random variable Z can be reformulated as a function of the Fisher information as follows:

$$m_{F(I)}{\left(f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{X}|\mathbf{Y},\mathbf{Z};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{X}|\mathbf{Z};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}$$

(90)

$$=i_F{\left(f_{\mathbf{Y}|\mathbf{X},\mathbf{Z};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{X},\mathbf{Y}|\mathbf{Z};\theta }\right)}_{{\theta }_k}$$

(91)

Proof. This follows analogously to that of the simpler case.

8. Relative Fisher Information Type II

Given that there is an alternative expression for the Fisher information (check Equation (12)), there is another way of defining the relative Fisher information expression.

Definition 10. The relative Fisher information Type II is defined by:

$$d_{F(II)}{\left(f_{\mathbf{X};\theta }\Vert f_{\mathbf{Y};\theta }\right)}_{{\theta }_k}\equiv -{\displaystyle \int f_{\mathbf{X};\theta }}(\mathbf{x})\frac{{\partial }^2}{\partial {\theta }_k^2}\left(\ln \left(\frac{f_{\mathbf{X};\theta }(\mathbf{x})}{f_{\mathbf{Y};\theta }(\mathbf{x})}\right)\right)d\mathbf{x}$$

(92)

Even though both definitions for the relative Fisher information are derived from equivalent expressions, they are not equivalent. Why is this so? This is because the argument of the Fisher information definition is a density function, whereas the argument of the relative Fisher information expression is a ratio of density functions, not a density function, thus their difference.

9. Mutual Fisher Information Type II

Analogously to the definition of the mutual Fisher information Type I, but in this case using the relative Fisher information of Type II, the following definition is obtained:

Definition 11. The mutual Fisher information Type II is defined by:

$$m_{F(II)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}\equiv -{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\theta }}(\mathbf{x},\mathbf{y})\frac{{\partial }^2}{\partial {\theta }_k^2}\ln \left(\frac{f_{\mathbf{X},\mathbf{Y};\theta }(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\theta }(\mathbf{x})f_{\mathbf{Y};\theta }(\mathbf{y})}\right)d\mathbf{x}}d\mathbf{y}$$

(93)

Theorem 11. The mutual Fisher information Type II can be reformulated as a function of the Fisher information as follows:

$$m_{F(II)}{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{X},\mathbf{Y};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{X};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{Y};\theta }\right)}_{{\theta }_k}$$

(94)

Proof.

$$m_{F(II)}{\left(f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}\right)}_{{\theta }_k}=-{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}}}(\mathbf{x},\mathbf{y})\frac{{\partial }^2}{\partial {\theta }_k^2}\ln \left(\frac{f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}(\mathbf{x},\mathbf{y})}{f_{\mathbf{X};\mathbf{\theta }}(\mathbf{x})f_{\mathbf{Y};\mathbf{\theta }}(\mathbf{y})}\right)d\mathbf{x}d\mathbf{y}$$

(95)

$$\begin{array}{c}=-{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}}}(\mathbf{x},\mathbf{y})\frac{{\partial }^2\ln f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}(\mathbf{x},\mathbf{y})}{\partial {\theta }_k^2}d\mathbf{x}d\mathbf{y}\\ +{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}}}(\mathbf{x},\mathbf{y})\frac{{\partial }^2\ln f_{\mathbf{X};\mathbf{\theta }}(\mathbf{x})}{\partial {\theta }_k^2}d\mathbf{x}d\mathbf{y}\\ +{\displaystyle \int {\displaystyle \int f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}}}(\mathbf{x},\mathbf{y})\frac{{\partial }^2\ln f_{\mathbf{Y};\mathbf{\theta }}(\mathbf{y})}{\partial {\theta }_k^2}d\mathbf{x}d\mathbf{y}\end{array}$$

(96)

from which the theorem follows. □

Corollary 2.

$$m_{F(II)}{\left(f_{\mathbf{X},\mathbf{Y};\mathbf{\theta }}\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{X}|\mathbf{Y};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{X};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(97)

$$=i_F{\left(f_{\mathbf{Y}|\mathbf{X};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{Y};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(98)

Proof. This comes from combining Theorem 11 and the chain rule for Fisher information. □

Example 6. For the example where Y = X + N, the mutual Fisher information Type II is given by:

$$\begin{array}{c}{m}_{F(II)}{\left(f_{Y,X;\mu ,\eta ,\nu }\right)}_{{\theta }_k}=i_F{\left(f_{Y|X}\right)}_{{\theta }_k}-i_F{\left(f_Y\right)}_{{\theta }_k}\\ =i_F{\left(f_N\right)}_{{\theta }_k}-i_F{\left(f_Y\right)}_{{\theta }_k}\end{array}$$

(99)

$$\frac{1}{{\nu }^2}-\frac{1}{{\eta }^2+{\nu }^2}$$

(100)

$$=\frac{{\eta }^2}{{\nu }^2({\eta }^2+{\nu }^2)}$$

(101)

Corollary 3.

$$m_{F(I)}{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=m_{F(II)}{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}$$

(102)

Proof. This can be deduced from the mutual Fisher information theorems.

Given that m_F(I) is always greater than or equal to zero, the expression m_F(II) can be positive or negative according to the value of the information correlation.

10. Other Properties

10.1. Lower Bound for Fisher Information

Stam’s inequality [8,9,40,47–50] states a lower bound for Fisher information, which links Fisher information and Shannon’s entropy power. However, this expression is limited to the special case where the parameters in the Fisher information expression correspond to a location parameter.

A more general result was recently proven by Stein et al. [51], which says that given a multidimensional random variable with density function f_X;θ with:

$$\mathbf{\mu }(\mathbf{\theta })={\displaystyle {\int }_{\mathcal{S}}\mathbf{x}{f}_{\mathbf{x};\theta }}(\mathbf{x})d\mathbf{x}$$

(103)

$$\sum (\mathbf{\theta })={\displaystyle {\int }_{\mathcal{S}}(\mathbf{x}-\mathbf{\mu }(\mathbf{\theta }))}{\left(\mathbf{x}-\mathbf{\mu }(\mathbf{\theta })\right)}^{\mathrm{T}}{f}_{\mathbf{x};\mathbf{\theta }}(\mathbf{x})d\mathbf{x}$$

(104)

If the Fisher information matrix is defined by:

$$\mathbf{F}(f_{\mathrm{X};\theta })={\displaystyle {\int }_{\mathcal{S}}{f}_{\mathrm{X};\theta }}(\mathrm{x}){\left(\frac{\partial \ln f_{\mathrm{X};\theta }(\mathbf{x})}{\partial \mathbf{\theta }}\right)}^{\mathrm{T}}\left(\frac{\partial \ln f_{\mathrm{X};\theta }(\mathbf{x})}{\partial \mathbf{\theta }}\right)d\mathbf{x}$$

(105)

then:

$$\mathbf{F}(f_{\mathrm{X};\theta })\underset{¯}{\succ }{\left(\frac{\partial \mathbf{\mu }(\mathbf{\theta })}{\partial \mathbf{\theta }}\right)}^{\mathrm{T}}{\sum }^{-1}(\mathbf{\theta })\left(\frac{\partial \mathbf{\mu }(\mathbf{\theta })}{\partial \mathbf{\theta }}\right)$$

(106)

if $\frac{\partial \mu (\theta )}{\partial \theta }$ exists. The authors of [51] explain that this is the same as saying that:

$$0\le {\mathbf{x}}^{\mathrm{T}}\left(\mathbf{F}(f_{\mathrm{X};\theta })-{\left(\frac{\partial \mathbf{\mu }(\mathbf{\theta })}{\partial \mathbf{\theta }}\right)}^{\mathrm{T}}{\sum }^{-1}(\mathbf{\theta })\left(\frac{\partial \mathbf{\mu }(\mathbf{\theta })}{\partial \mathbf{\theta }}\right)\right)\mathbf{x}$$

(107)

The previous expression states that the difference of matrices between the large parenthesis is a positive semi-definite matrix. Thus, its diagonal elements are non-negative, and it can be stated:

Corollary 4. The following lower bound for Fisher information holds:

$${\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^m\frac{\partial {\mu }_i}{\partial {\theta }_k}{c}_{ij}^{-1}}}\frac{\partial {\mu }_j}{\partial {\theta }_k}\le i_F{(f_{\mathrm{X};\theta })}_{{\theta }_k}$$

(108)

and$c_{ij}^{-1}$ stands for the ij-th element of Σ⁻¹ (θ).

10.2. In Some Cases, Conditioning Increases the Fisher Information

The following result states that in some cases, conditioning a random variable with another variable may increase the Fisher information. This result is a generalization of another published previously by Zamir [32].

Theorem 12 (Conditioning Increases Information). If f_Y|X;θ depends on θ_k and f_X does not depend on it, then:

$$i_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}\le i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}$$

(109)

Proof. Thus, given that only f_Y|X;θ depends on θ_k, Theorem 9 guarantees that:

$$i_C{(f_{\mathrm{X},\mathrm{Y};\theta })}_{{\theta }_k}=0$$

(110)

Hence, from the previous mutual Fisher information expressions:

$$0\le m_{F(I)}{\left(f_{\mathrm{X},\mathrm{Y};\theta }\right)}_{{\theta }_k}=i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}-i_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}$$

(111)

Thus:

$$i_F{\left(f_{\mathrm{Y};\theta }\right)}_{{\theta }_k}\le i_F{\left(f_{\mathrm{Y}|\mathrm{X};\theta }\right)}_{{\theta }_k}$$

(112)

10.3. Data Processing Inequality

Following the same analysis done by Cover and Thomas to present the data processing theorem for Shannon entropy [30] and continuing with the work done by Zamir [32], the case where the joint density function of the random variables R, S and T can be expressed by f_R,S,T;θ = f_R;θ · f_S|R;θ · f_T|S;θ is considered. In this case, they form a short Markov chain that is represented by R → S → T. Because Markovicity implies conditional independence, then it is true that f_R,T|S;θ = f_R|S;θ · f_T|S;θ.

Theorem 13. Given a Markov chain R → S → T, where only f_T|S;θ depends on θ_k, then:

$$m_{F(I)}{\left(f_{\mathbf{R},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\le m_{F(I)}{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(113)

Proof. From the previous results:

$$m_{F(I)}{\left(f_{(\mathbf{R},\mathbf{S}),\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{R},\mathbf{S}|\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{R},\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{(\mathbf{R},\mathbf{S}),\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(114)

$$=i_F{\left(f_{\mathbf{R}|\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}+i_F{\left(f_{\mathbf{S}|\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{R}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{(\mathbf{R},\mathbf{S}),\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(115)

$$\begin{array}{l}=\left(i_F{\left(f_{\left|\mathbf{R}\right|\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{R}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{R},\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}\right)-2\cdot i_C{\left(f_{\mathbf{R},\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}\\ +\left(i_F{\left(f_{\mathbf{S}|\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\right)-2\cdot i_C{(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }})}_{{\theta }_k}+2\cdot i_C{\left(f_{(\mathbf{R},\mathbf{S}),\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\end{array}$$

(116)

$$=m_{F(I)}{\left(f_{\mathbf{R},\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}-m_{F(I)}{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}-2\cdot i_C{\left(f_{\mathbf{R},\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}-2\cdot i_C{(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }})}_{{\theta }_k}+2\cdot i_C{\left(f_{(\mathbf{R},\mathbf{S}),\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(117)

Analogously:

$$\begin{array}{c}{m}_{F(I)}{\left(f_{(\mathrm{R},\mathrm{S}),\mathrm{T};\theta }\right)}_{{\theta }_k}=m_{F(I)}{\left(f_{\mathrm{S},\mathrm{T}|\mathrm{R};\theta }\right)}_{{\theta }_k}+m_{F(I)}{\left(f_{\mathrm{R},\mathrm{T};\theta }\right)}_{{\theta }_k}\\ -2\cdot i_C{\left(f_{\mathrm{S},\mathrm{T}|\mathrm{R};\theta }\right)}_{{\theta }_k}-2\cdot i_C{\left(f_{\mathrm{R},\mathrm{T};\theta }\right)}_{{\theta }_k}+2\cdot i_C{\left(f_{(\mathrm{R},\mathrm{S}),\mathrm{T};\theta }\right)}_{{\theta }_k}\end{array}$$

(118)

Because only f_T|S;θ depends on θ_k and all of the information correlation terms have derivatives of density functions that do not depend on this parameter, then all of the information correlation terms are zero. Hence:

$$m_{F(I)}{\left(f_{\mathrm{R},\mathrm{T}|\mathrm{S};\theta }\right)}_{{\theta }_k}+m_{F(I)}{\left(f_{\mathrm{S},\mathrm{T};\theta }\right)}_{{\theta }_k}=m_{F(I)}{\left(f_{\mathrm{S},\mathrm{T}|\mathrm{R};\theta }\right)}_{{\theta }_k}+m_{F(I)}{\left(f_{\mathrm{R},\mathrm{T};\theta }\right)}_{{\theta }_k}$$

(119)

Given that $m_{F(I)}{\left(f_{\mathrm{R},\mathrm{T}|\mathrm{S};\theta }\right)}_{{\theta }_k}=0$ because R and T are independent given S, and $m_{F(I)}{\left(f_{\mathrm{S},\mathrm{T}|\mathrm{R};\theta }\right)}_{{\theta }_k}\ge 0$, then:

$$m_{F(I)}{\left(f_{\mathrm{R},\mathrm{T};\theta }\right)}_{{\theta }_k}\le m_{F(I)}{\left(f_{\mathrm{S},\mathrm{T};\theta }\right)}_{{\theta }_k}$$

(120)

□

Given that in the previous proof, all of the information correlation terms are zero, then $m_{F\left(II\right)}{\left(f_{\mathbf{R},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}=m_{F\left(I\right)}{\left(f_{\mathbf{R},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$, and $m_{F\left(II\right)}{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}=m_{F\left(I\right)}{\left(f_{\mathbf{R},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$. Thus, the following corollary is obtained:

Corollary 5. Given a Markov chain R → S → T, where only f_T|S;θ depends on θ_k, then:

$$m_{F\left(II\right)}{\left(f_{\mathbf{R},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\le m_{F\left(II\right)}{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(121)

Proof. The conditional independence provided by the Markovicity of the random variables follows directly from the mutual Fisher information Type II definition, and in this case, the values of mutual Fisher information Type I and mutual Fisher information Type II are identical.

Using the definition of mutual Fisher information Type II and the previous expression, it is readily obtained, in a simpler way, a result already proven by Plastino et al. [52]:

Corollary 6. From the previous results, it is obvious that:

$$i_F{\left(f_{\mathbf{T}|\mathbf{R};\mathbf{\theta }}\right)}_{{\theta }_k}\le i_F{\left(f_{\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(122)

Proof. From Equation (121):

$$m_{F\left(II\right)}{\left(f_{\mathbf{R}|\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\le m_{F\left(II\right)}{\left(f_{\mathbf{S},\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(123)

$$i_F{\left(f_{\mathbf{T}|\mathbf{R};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}\le i_F{\left(f_{\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}-i_F{\left(f_{\mathbf{T};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(124)

$$i_F{\left(f_{\mathbf{T}|\mathbf{R};\mathbf{\theta }}\right)}_{{\theta }_k}\le i_F{\left(f_{\mathbf{T}|\mathbf{S};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(125)

□

In other words, in any Markovian process, the further away that the random variables used by the estimator are, the larger is the variance of the estimated parameter.

10.4. Upper Bound on Estimation Error

A well-known result states that given a variance η, of all possible density functions, the one that maximizes the differential entropy is the Gaussian density function [30]. Hence, for an arbitrary density function f_X, some side information Y and an estimator $\widehat{X}$, it is possible to obtain an estimation version of the Fano inequality [10] (p. 255):

$$\frac{1}{2\pi e}{e}^{2h_S\left(f_{X|Y}\right)}\le {\mathbb{E}}_X\left\{{\left(X-\widehat{X}\left(Y\right)\right)}^2\right\}$$

(126)

In the context of Fisher information, the same question arises: is it possible to bound the estimation error using this quantity as well? Surprisingly, the answer is yes, but in the form of an upper bound. Thus, Shannon entropy can be used to set error lower bounds and Fisher information upper ones. In order to establish this bound, the following setup is defined, where a random variable R is given, and a related random variable Y is observed, which, in turn, is used to calculate a function $\widehat{\mathbf{R}}=\mathbf{g}\left(\mathbf{Y}\right)$. It is desired to bound the probability that ${\left(\mathbf{R}-\widehat{\mathbf{R}}\right)}^2>\in$. It is important to note that $\mathbf{R}\to \mathbf{Y}\to \widehat{\mathbf{R}}$ is a Markov chain and that $\widehat{\mathbf{R}}$ depends on θ.

Theorem 14. Given a random variable R and an estimator of it named$\widehat{\mathbf{R}}$, the estimation error is defined by:

$$E={\left(\mathbf{R}-\widehat{\mathbf{R}}\right)}^2$$

(127)

Then, the probability that the estimation error exceeds some ε value:

$$\mathrm{P}\left\{E>\epsilon \right\}\le \frac{i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}}{i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E=\xi ;\mathbf{\theta }}\right)}_{{\theta }_k}}$$

(128)

for some ξ ∈ [ε, ∞].

Proof. Using the chain rule for Fisher information:

$$i_F{\left(f_{\mathbf{R},E|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\right)}_{{\theta }_k}+i_F{\left(f_{E|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}=i_F{\left(f_{E|\mathbf{R},\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}+i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(129)

Using the fact that given R and $\widehat{\mathbf{R}}$, then E is no longer a random variable, then:

$$i_F{\left(f_{E|\mathbf{R},\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}=0$$

(130)

Hence,

$$i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\right)}_{{\theta }_k}+i_F{\left(f_{E|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}=i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(131)

Neglecting $i_F{\left(f_{E|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}$, because it is always greater or equal to zero, it is obtained:

$$i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\right)}_{{\theta }_k}\le i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}$$

(132)

Moreover, the term:

$$i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\right)}_{{\theta }_k}={\displaystyle {\int }_{e\in E}^{\infty }{\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}},e\right)}}}{\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}^{2}d\mathbf{r}d\widehat{\mathbf{r}}de$$

(133)

$$\begin{array}{l}={\displaystyle {\int }_{e\in E|e\le \epsilon }^{\epsilon }{\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}},e\right)}}\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}d\mathbf{r}d\widehat{\mathbf{r}}de\\ +={{\displaystyle {\int }_{e\in E|e>\epsilon }^{\infty }{\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}},e\right)}}\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}}^{2}d\mathbf{r}d\widehat{\mathbf{r}}de\end{array}$$

(134)

$$\ge {{\displaystyle {\int }_{e\in E|e>\epsilon }^{\infty }{\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}},e\right)}}\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}}^{2}d\mathbf{r}d\widehat{\mathbf{r}}de$$

(135)

$$={\displaystyle {\int }_{e\in E|e>\epsilon }^{\infty }{\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}}|e\right)}{f}_{E;\mathbf{\theta }}\left(e\right)}{\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}^2}d\mathbf{r}d\widehat{\mathbf{r}}de$$

(136)

$$={{\displaystyle {\int }_{e\in E|e\le \epsilon }^{\infty }{f}_{E;\mathbf{\theta }}\left(e\right){\displaystyle \int {\displaystyle {\int }_{\mathbf{r}\in \mathbf{R},\widehat{\mathbf{r}}\in \widehat{\mathbf{R}}|{\left(\mathbf{r}-\widehat{\mathbf{r}}\right)}^2=e}{f}_{\mathbf{R},\widehat{\mathbf{R}}|E;\mathbf{\theta }}\left(\mathbf{r},\widehat{\mathbf{r}},e\right)}}\left(\frac{\partial \ln f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\left(\mathbf{r}|\widehat{\mathbf{r}},e\right)}{\partial {\theta }_k}\right)}}^{2}d\mathbf{r}d\widehat{\mathbf{r}}de$$

(137)

$$={\displaystyle {\int }_{e\in E|e>\epsilon }^{\infty }{f}_{E;\mathbf{\theta }}\left(e\right)i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E=e;\mathbf{\theta }}\right)}_{\theta }{}_{{}_k}}de$$

(138)

Using the mean value theorem, for some ξ ∈ [ε, ∞]:

$$i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E=\xi ;\mathbf{\theta }}\right)}_{{\theta }_k}{{\displaystyle {\int }_{e\in E|e>\epsilon }^{\infty }{f}_{E;\mathbf{\theta }}\left(e\right)de=i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E=\xi ;\mathbf{\theta }}\right)}_{{\theta }_k}\cdot \mathrm{P}\left\{E>\epsilon \right\}\le i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E;\mathbf{\theta }}\right)}_{{\theta }_k}\le i_F\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}}_{{\theta }_k}$$

(139)

Hence:

$$\mathrm{P}\left\{E>\epsilon \right\}\le \frac{i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}};\mathbf{\theta }}\right)}_{{\theta }_k}}{i_F{\left(f_{\mathbf{R}|\widehat{\mathbf{R}},E=\xi ;\mathbf{\theta }}\right)}_{{\theta }_k}}$$

(140)

□

11. Discussion

The Fisher information, which sets a bound on how precise the estimation of an unknown parameter of a density function can be, has an associated set of properties that are equivalent to those of Shannon’s differential entropy. The properties presented in this work help to understand how to manipulate and use Fisher information in ways that so far have been exclusive to Shannon’s differential entropy These properties that are of special importance to the generalization of the mutual information concept for the Fisher information realm are a new version of the data processing theorem that shows that Fisher information decreases in a Markov chain and an upper bound of the estimation error of a random variable that is regulated by the Fisher information.