交叉熵在Cross-entropy error function and logistic regression

转载https://en.wikipedia.org/wiki/Cross_entropy

In information theory, the cross entropy between two probability
distributions {\displaystyle p}

 and {\displaystyle
q}

 over
the same underlying set of events measures the average number of bits needed to identify an event drawn from the
set, if a coding scheme is used that is optimized for an "unnatural" probability distribution {\displaystyle q}

,
rather than the "true" distribution {\displaystyle p}

.

The cross entropy for the distributions {\displaystyle p}

 and {\displaystyle
q}

 over
a given set is defined as follows:
{\displaystyle H(p,q)=\operatorname {E} _{p}[-\log
q]=H(p)+D_{\mathrm {KL} }(p\|q),\!}


where {\displaystyle H(p)}

 is
the entropy of {\displaystyle
p}

,
and {\displaystyle D_{\mathrm {KL} }(p||q)}

 is
the Kullback–Leibler divergence of {\displaystyle
q}

 from {\displaystyle
p}

 (also
known as the relative entropy of p with respect to q — note the reversal of emphasis).

For discrete {\displaystyle
p}

 and {\displaystyle
q}

 this
means
{\displaystyle H(p,q)=-\sum _{x}p(x)\,\log q(x).\!}


The situation for continuous distributions is analogous.
We have to assume that {\displaystyle p}

 and {\displaystyle
q}

 are absolutely
continuous with respect to some reference measure {\displaystyle
r}

 (usually {\displaystyle
r}

 is
a Lebesgue measure on a Borel σ-algebra).
Let {\displaystyle P}

 and {\displaystyle
Q}

 be
probability density functions of {\displaystyle p}

 and {\displaystyle
q}

 with
respect to {\displaystyle r}

.
Then
{\displaystyle -\int _{X}P(x)\,\log Q(x)\,dr(x)=\operatorname
{E} _{p}[-\log Q].\!}


NB: The notation {\displaystyle H(p,q)}

 is
also used for a different concept, the joint entropy of {\displaystyle
p}

 and {\displaystyle
q}

.

Motivation[edit]

In information theory, the Kraft–McMillan theorem establishes that any directly decodable
coding scheme for coding a message to identify one value {\displaystyle x_{i}}

 out
of a set of possibilities {\displaystyle X}

 can
be seen as representing an implicit probability distribution {\displaystyle q(x_{i})=2^{-l_{i}}}

 over {\displaystyle
X}

,
where {\displaystyle l_{i}}

 is
the length of the code for {\displaystyle x_{i}}

 in
bits. Therefore, cross entropy can be interpreted as the expected message-length per datum when a wrong distribution {\displaystyle
Q}

 is
assumed while the data actually follows a distribution {\displaystyle P}

.
That is why the expectation is taken over the probability distribution {\displaystyle P}

 and
not {\displaystyle Q}

.
{\displaystyle H(p,q)=\operatorname {E} _{p}[l_{i}]=\operatorname
{E} _{p}\left[\log {\frac {1}{q(x_{i})}}\right]}

{\displaystyle H(p,q)=\sum _{x_{i}}p(x_{i})\,\log
{\frac {1}{q(x_{i})}}\!}

{\displaystyle H(p,q)=-\sum _{x}p(x)\,\log q(x).\!}

Estimation[edit]

There are many situations where cross-entropy needs to be measured but the distribution of {\displaystyle p}

 is
unknown. An example is language modeling, where a model is created based on a training set {\displaystyle
T}

,
and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, {\displaystyle
p}

 is
the true distribution of words in any corpus, and {\displaystyle q}

 is
the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula:
{\displaystyle H(T,q)=-\sum _{i=1}^{N}{\frac
{1}{N}}\log _{2}q(x_{i})}


where {\displaystyle N}

 is
the size of the test set, and {\displaystyle q(x)}

 is
the probability of event {\displaystyle x}

 estimated
from the training set. The sum is calculated over {\displaystyle N}

.
This is a Monte Carlo estimate of the true cross entropy, where the training set is treated as samples from {\displaystyle
p(x)}

.

Cross-entropy minimization[edit]

Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the cross-entropy
method.

When comparing a distribution {\displaystyle q}

 against
a fixed reference distribution {\displaystyle p}

,
cross entropy and KL divergence are identical up to an additive
constant (since {\displaystyle p}

 is
fixed): both take on their minimal values when {\displaystyle p=q}

,
which is {\displaystyle 0}

 for
KL divergence, and {\displaystyle \mathrm {H} (p)}

 for
cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle
of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

However, as discussed in the article Kullback–Leibler divergence,
sometimes the distribution {\displaystyle q}

 is
the fixed prior reference distribution, and the distribution {\displaystyle p}

 is
optimised to be as close to {\displaystyle q}

 as
possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be {\displaystyle
D_{\mathrm {KL} }(p\|q)}

,
rather than {\displaystyle H(p,q)}

.

Cross-entropy error function and logistic regression[edit]

Cross entropy can be used to define the loss function in machine learning and optimization. The true probability {\displaystyle
p_{i}}

 is
the true label, and the given distribution {\displaystyle q_{i}}

 is
the predicted value of the current model.

More specifically, let us consider logistic regression, which (in its most basic form)
deals with classifying a given set of data points into two possible classes generically labelled {\displaystyle 0}

 and {\displaystyle
1}

.
The logistic regression model thus predicts an output {\displaystyle y\in \{0,1\}}

,
given an input vector {\displaystyle \mathbf {x} }

.
The probability is modeled using the logistic function {\displaystyle
g(z)=1/(1+e^{-z})}

.
Namely, the probability of finding the output {\displaystyle y=1}

 is
given by
{\displaystyle q_{y=1}\ =\ {\hat {y}}\ \equiv
\ g(\mathbf {w} \cdot \mathbf {x} )\,,}


where the vector of weights {\displaystyle \mathbf {w} }

 is
optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary
probability of finding the output {\displaystyle y=0}

 is
simply given by
{\displaystyle q_{y=0}\ =\ 1-{\hat {y}}}


The true (observed) probabilities can be expressed similarly as {\displaystyle p_{y=1}=y}

 and {\displaystyle
p_{y=0}=1-y}

.

Having set up our notation, {\displaystyle p\in \{y,1-y\}}

 and {\displaystyle
q\in \{{\hat {y}},1-{\hat {y}}\}}

,
we can use cross entropy to get a measure for similarity between {\displaystyle p}

 and {\displaystyle
q}

:
{\displaystyle H(p,q)\ =\ -\sum _{i}p_{i}\log
q_{i}\ =\ -y\log {\hat {y}}-(1-y)\log(1-{\hat {y}})}


The typical loss function that one uses in logistic regression is computed by taking the average of all cross-entropies in the sample. For example, suppose we have {\displaystyle
N}

 samples
with each sample labeled by {\displaystyle n=1,\dots ,N}

.
The loss function is then given by:
{\displaystyle {\begin{aligned}L(\mathbf {w}
)\ &=\ {\frac {1}{N}}\sum _{n=1}^{N}H(p_{n},q_{n})\ =\ -{\frac {1}{N}}\sum _{n=1}^{N}\ {\bigg [}y_{n}\log {\hat {y}}_{n}+(1-y_{n})\log(1-{\hat {y}}_{n}){\bigg ]}\,,\end{aligned}}}


where {\displaystyle {\hat {y}}_{n}\equiv g(\mathbf {w} \cdot \mathbf {x} _{n})}

,
with {\displaystyle g(z)}

 the
logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It's also known as log loss (In this case, the binary label is often denoted by {-1,+1}).[1]