Monday, April 21, 2014

ANOVA and the "Temperature" of a data set

Typically, with analysis of variance, one chooses to weight all data points -- or deviations from our model -- equally in the chi-squared statistic, assuming

\begin{eqnarray}
\chi^2 &=& \sum_i^N \frac{(x_i-f(x_i))^2}{2\sigma^2_i}\\
\sigma_i &=& \sigma \ \ \forall, i
\end{eqnarray}

Which yields a very simple posterior distribution:

\begin{eqnarray}
P(\mu, \sigma^2 \vert D) &=& \frac{\left(2\pi \sigma^2\right)^{-N/2}e^{-\chi^2/2}P(\mu,\sigma^2)}{Z}
\end{eqnarray}

Where Z is our awful, marginalized normalizing factor from before. We can write our posterior distribution much like the partition function of an Bose-Einstein ensemble, and associate our chi-squared statistic -- or deviation from our model f(x) -- with an 'Energy':
\begin{eqnarray}
P(\mu, \sigma^2 \vert D) &=& \frac{e^{-\beta E(\mu,\sigma^2)}}{Z} \\
Z &=& \int \int e^{-\beta E(\mu,\sigma^2)} d\mu d\sigma^2 \\
\beta E(\mu,\sigma^2) &=& \sum_i^N \frac{(x_i-f(x_i))^2}{2\sigma^2}
\end{eqnarray}

In this case the variance of each and every data point y_i -- the denominator in our chi-squared expression above -- acts much like the "temperature" of our distribution. We can make things even clearer by putting in the chemical potential, or the energy associated with adding/replacing a member of our dataset:

\begin{eqnarray}
\mu_c &=& \mathrm{chemical}\ \ \mathrm{potential}\\
P(\mu, \sigma^2 \vert D) &=& \frac{e^{N\beta\mu_c-\beta E)}}{Z} \\
\beta \mu_c &=& \log\left(\frac{1}{\sqrt{2\pi \sigma^2}} \right)
\end{eqnarray}

Which manifests itself as our extra normalizing factor for adding an extra data point. Exactly like our chemical potential mu's function in the bose einstein distribution! (Pardon the repetition of mu).

No comments:

Post a Comment