persist_js_state$25f41118-d294-11ef-13a8-3fa6587c1bf3running§runtimeAdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f41118-d294-11ef-13a8-3fa6587c1bf3depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Zhas_pluto_hook_features¤body
Let us recapitulate the differences between the generative and discriminative approaches to classification in a table:
Generative Discriminative (ML)
1 Like density estimation , model joint prob.
\[p(\mathcal{C}_k) p(x|\mathcal{C}_k) = \pi_k \mathcal{N}(\mu_k,\Sigma)\]
Like (linear) regression , model conditional
\[p(\mathcal{C}_k|x,\theta)\]
2 Leads to softmax posterior class probability
\[ p(\mathcal{C}_k|x,\theta ) = e^{\theta_k^T x}/Z\]
with structured \(\theta\)
Choose also softmax posterior class probability
\[ p(\mathcal{C}_k|x,\theta ) = e^{\theta_k^T x}/Z\]
but now with ‘free’ \(\theta\)
3
For Gaussian \(p(x|\mathcal{C}_k)\) and multinomial priors,
\[\hat \theta_k = \left[ {\begin{array}{c}
{ - \frac{1}{2} \mu_k^T \sigma^{-1} \mu_k + \log \pi_k} \\
{\sigma^{-1} \mu_k } \\
\end{array}} \right]\]
in one shot .
Find \(\hat\theta_k\) through gradient-based adaptation
\[\nabla_{\theta_k}\mathrm{L}(\theta) = \sum_n \Big( y_{nk} - \frac{e^{\theta_k^T x_n}}{\sum_{k^\prime} e^{\theta_{k^\prime}^T x_n}} \Big)\, x_n\]
persist_js_state$9704ee6a-e233-49f1-8c66-d81543927342running§runtime cdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$9704ee6a-e233-49f1-8c66-d81543927342depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&v`has_pluto_hook_features¤body
🎯 Key concept
It is easy to write down the Bayesian equations for the weight posterior and the predictive distribution of $y_\bullet$ , but in practice these do not admit closed-form solutions. A traditional workaround is the Laplace approximation , which has long been used to approximate the marginalization integrals in Bayesian inference.
persist_js_state$38b4854f-be02-4696-802f-2106481e3aearunning§runtime 8depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$38b4854f-be02-4696-802f-2106481e3aeadepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&*33has_pluto_hook_features¤body
Bayesian Processing of Uncertainties
We now make an important observation: According to Eq. B-4.143, the posterior covariance matrix $S_N$ of the weight vector depends on both the prior variance $S_0$ and the distribution of the training data $\{(x_n, y_n)\}_{n=1}^N$ . In regions with limited training data and/or an uninformative prior (i.e., large $S_0$ ), the posterior uncertainty about the weights remains high. This increased uncertainty raises $\sigma_a^2$ in Eq. B-4.150, causing the posterior class probability in Eq. B-4.152 to approach $0.5$ (since $\Phi(0) = 0.5$ , see Gaussian CDF image ), thereby reflecting greater uncertainty in the prediction.
In other words, if you draw a new feature $x_\bullet$ from a region with little training data, then the predictive class probability $p(y_\bullet | x_\bullet, D)$ naturally tends toward $0.5$ , a built-in expression of uncertainty, courtesy of the Bayesian framework.
In contrast, if you eliminate uncertainty by representing the weights as fixed-point estimates (i.e., as mere numbers), then the model becomes overconfident. It will still produce sharp predictions, even in regions where it has seen no data, exactly when it should be most uncertain.
persist_js_state$25f0adde-d294-11ef-353e-4b4773df9ff5running§runtime
Ÿdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f0adde-d294-11ef-353e-4b4773df9ff5depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body'
Data-generating distribution
We will take inspiration from the generative classification approach, where we derived the class posterior
$$p(y_{nk} = 1\,|\,x_n,\beta_k,\gamma_k) = \sigma(\beta_k^T x_n + \gamma_k)$$
as a softmax function of a linear map of the input.
Here, for the 2-class problem, we choose the 2-class softmax function (which is called the logistic function
$$p(y_n =1 \,|\, x_n, w) = \sigma(w^T x_n) \,.$$
where
$$\sigma(a) = \frac{1}{1+e^{-a}}$$
is the logistic function.
persist_js_state$25ef6ece-d294-11ef-270a-999c8d457b24running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25ef6ece-d294-11ef-270a-999c8d457b24depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&shas_pluto_hook_features¤bodyj
Framework
A data set is given by $D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}$ with $x_n \in \mathbb{R}^M$ and $y_n \in \mathcal{C}_k$ , with $k=1,\ldots,K$ .
persist_js_state$8b0bb225-bdc1-45ec-bd34-68d674d6f08drunning§runtime 9 udepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$8b0bb225-bdc1-45ec-bd34-68d674d6f08ddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&&hhas_pluto_hook_features¤body
The Laplace Approximation approximates a function by a Gaussian-shaped function. In this case, we will approximate the weight posterior $p(w|D)$ (Eq. B-4.142) by a Gaussian distribution
$$q(w) = \mathcal{N}\left(w\,|\, m_{N}, S_N\right) \tag{B-4.144}$$
with
$$\begin{align}
m_N &= \arg\max_w \log p(w|D) \\
S_N^{-1} &= S_0^{-1} + \sum_n \sigma_n (1-\sigma_n) x_n x_n^T \tag{B-4.143}
\end{align}$$
where we used short-hand $\sigma_n = \sigma\left((2y_n-1) m_N^T x_n\right)$ .
If we substitute the Gaussian approximation from Eq. B-4.143 into the expression for the predictive class distribution (Eq. B-4.145), we obtain (after some additional approximations):
$$\begin{align*}
p(y_\bullet = 1 \mid x_\bullet, D) &= \int p(y_\bullet = 1 \,|\, x_\bullet, w) \cdot p(w\,|\, D) \,\mathrm{d}w \\
&\approx \int p(y_\bullet = 1 \,|\, x_\bullet, w) \cdot q(w) \,\mathrm{d}w \\
&= \int \sigma(w^T x_\bullet) \cdot \mathcal{N}\left(w \,|\, m_N, S_N\right) \,\mathrm{d}w \tag{B-4.145} \\
&\approx \Phi\left( \frac{\mu_a}{\sqrt(\lambda^{-2} +\sigma_a^2)}\right) \tag{B-4.152}
\end{align*}$$
where
$$\begin{align}
\lambda^2 &= \pi / 8 \\
\mu_a &= m^T_{N} x_\bullet \tag{B-4.149} \\
\sigma_a^2 &= x^T_\bullet S_N x_\bullet \tag{B-4.150}
\end{align}$$
and $\Phi(x)= \frac{1}{\sqrt(2\pi)}\int_{-\infty}^{x}e^{-t^2/2}\mathrm{d}t$ is the Standard Normal cumulative distribution function (CDF).
Although the intermediate equations may look intimidating, the final result for the predictive distribution Eq. B-4.152 has a simple closed-form expression.
persist_js_state$33b859f2-9ea8-4f8b-b0f8-08a19c6a96fcrunning§runtimetdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$33b859f2-9ea8-4f8b-b0f8-08a19c6a96fcdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&vhas_pluto_hook_features¤body
persist_js_state$fe66a986-2f55-4417-a71d-b3b99f6369ccrunning§runtime1~depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$fe66a986-2f55-4417-a71d-b3b99f6369ccdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&
4Dhas_pluto_hook_features¤body2
Challenge: difficult class-conditional data distributionspersist_js_state$e4cc517b-d3b5-4517-a28b-efb8aba24496running§runtime yƸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$e4cc517b-d3b5-4517-a28b-efb8aba24496depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&)Ƚhas_pluto_hook_features¤body
The numerical issues associated with the Laplace approximation and the evaluation of the predictive class distribution are discussed in detail in the following mini-lecture.
persist_js_state$25f1c2a0-d294-11ef-009c-69b64e87e5fbrunning§runtime 4depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f1c2a0-d294-11ef-009c-69b64e87e5fbdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&"has_pluto_hook_features¤body
Many methods have been developed to approximate these types of Bayesian integrals. Here, we present the Laplace approximation , which is one of the simplest methods with broad applicability to Bayesian calculations.
persist_js_state$173e545a-9125-4dc4-8df4-c3aa84a4b88arunning§runtime &Jdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$173e545a-9125-4dc4-8df4-c3aa84a4b88adepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&1Ohas_pluto_hook_features¤body:
Summary
persist_js_state$d1bbdc6a-e5ff-4cd6-9175-860b5ec04f3crunning§runtime Xdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$d1bbdc6a-e5ff-4cd6-9175-860b5ec04f3cdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&槷has_pluto_hook_features¤bodyd
Bayesian Logistic Regression
persist_js_state$69706576-0333-43bf-8523-e7838f373529running§runtime 1cŸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$69706576-0333-43bf-8523-e7838f373529depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&*has_pluto_hook_features¤body
Note that we get a full predictive posterior distribution over the assignment of class labels for every datapoint, so instead of assigning a point to a class, we get a measure of uncertainty over our class assignment!
persist_js_state$fce5d561-ea76-4bd8-9cce-6f707f72fc60running§runtimeY=depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$fce5d561-ea76-4bd8-9cce-6f707f72fc60depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤bodypersist_js_state$66351c02-1921-44dc-b461-84a536c40fd5running§runtime 6ɸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$66351c02-1921-44dc-b461-84a536c40fd5depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&'!!has_pluto_hook_features¤body7
The logistic function $\sigma(a) = 1/(1+e^{-a})$ (red), together with the scaled Standard Normal cumulative distribution function (CDF) $\Phi(\lambda a) = \frac{1}{\sqrt(2\pi)}\int_{-\infty}^{a}e^{-t^2/2}\mathrm{d}t$ , for $\lambda^2=\pi/8$ (in blue). We will use this approximation later in the Laplace approximation . Based on Bishop fig.4.9.
persist_js_state$ad196ae6-c65e-4aaa-b0cc-bd72daa41952running§runtimedepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$ad196ae6-c65e-4aaa-b0cc-bd72daa41952depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&dзhas_pluto_hook_features¤bodypersist_js_state$93083660-6a49-4147-b00c-d62a4453f222running§runtime
depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$93083660-6a49-4147-b00c-d62a4453f222depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&e
has_pluto_hook_features¤bodyy
That's not a good fit for the $y=1$ class!
Now, we could continue down this road and try to fit a more complex distribution to the $y = 1$ class to make it fit. But let's explore a different approach , modeling the discriminative boundary directly.
persist_js_state$b94644f8-725d-49bf-9641-3dad8b647f45running§runtime 돸depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$b94644f8-725d-49bf-9641-3dad8b647f45depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&2Bhas_pluto_hook_features¤body
Discrimination boundaries (*)
Show that for logistic regression with $p(y_n =1 \,|\, x_n) = \sigma(w^T x_n)$ , the discrimination boundary, which can be computed by
$$\frac{p(y_n =1|x_n)}{p(y_n =0|x_n)} \overset{!}{=} 1$$
is a straight line.
persist_js_state$4ceede48-a4d5-446b-bb34-26cec4af357arunning§runtime1ʸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$4ceede48-a4d5-446b-bb34-26cec4af357adepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&0,has_pluto_hook_features¤body w
N = 120 persist_js_state$a65ca01a-0e9a-42cb-b1d7-648102a77eb5running§runtimedepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$a65ca01a-0e9a-42cb-b1d7-648102a77eb5depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&qJǷhas_pluto_hook_features¤body-plot_dataset (generic function with 1 method)persist_js_state$25efa2fe-d294-11ef-172f-9bb09277f59erunning§runtime edepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25efa2fe-d294-11ef-172f-9bb09277f59edepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&=has_pluto_hook_features¤body
Here's an IDEA ! Let's model the posterior
$$p(y_n\in\mathcal{C}_k|x_n)$$
directly , without any assumptions on the prior class densities $p(x)$ . This approach is called Discriminative Classification .
persist_js_state$6eee35ee-fd55-498f-9441-f18c2508de19running§runtime ?depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$6eee35ee-fd55-498f-9441-f18c2508de19depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&2has_pluto_hook_features¤body4
Code
persist_js_state$6b56ec96-4b9d-4281-bd63-061df324867frunning§runtimewrdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$6b56ec96-4b9d-4281-bd63-061df324867fdepends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneeX_c1last_run_timestampAڹ&\ַhas_pluto_hook_features¤bodyE72×2 adjoint(::Matrix{Float64}) with eltype Float64:
2.37585 3.34038
0.859544 4.75254
0.931862 4.07941
1.3216 5.8365
0.018782 3.679
0.0323811 2.75244
0.985239 2.89621
⋮
1.12348 3.87913
0.342695 2.65799
0.666544 2.99033
0.943866 3.06345
1.86646 3.73893
0.301837 3.20804persist_js_state$b8790891-1546-48e0-9f96-e09eade31c12running§runtime 姸depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$b8790891-1546-48e0-9f96-e09eade31c12depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&\has_pluto_hook_features¤body&logσ (generic function with 1 method)persist_js_state$4ab4bcca-da6e-4137-858d-257c04388277running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$4ab4bcca-da6e-4137-858d-257c04388277depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&vܷhas_pluto_hook_features¤body
🎯 Key concept
The Laplace approximation renders a closed-form and interpretable (approximate) solution for the predictive class distribution,
$$p(y_\bullet = 1 | x_\bullet, D) \approx \Phi\left( \frac{m^T_{N} x_\bullet}{\sqrt{(8/\pi) +x^T_\bullet S_N x_\bullet)}}\right)\,.$$
persist_js_state$25f02ac6-d294-11ef-26c4-f142b8ac4b5frunning§runtime fdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f02ac6-d294-11ef-26c4-f142b8ac4b5fdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body٦
What model should we use for the posterior distribution $p(y_n \in \mathcal{C}_k|x_n)$ ?
persist_js_state$3b24b142-2239-4951-9177-ff87b5da4b68running§runtimedepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3b24b142-2239-4951-9177-ff87b5da4b68depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Ɯkhas_pluto_hook_features¤body
Click to see proof
The Log-likelihood is
$$\mathrm{L}(w) = \log \prod_n \prod_k {\underbrace{p(y_{nk}=1|x_n,w)}_{p_{nk}}}^{y_{nk}} = \sum_{n,k} y_{nk} \log p_{nk}$$
Use the fact that the softmax $\phi_k \equiv e^{a_k} / {\sum_j e^{a_j}}$ has analytical derivative,
$$ \begin{align*}
\frac{\partial \phi_k}{\partial a_j} &= \frac{(\sum_j e^{a_j})e^{a_k}\delta_{kj}-e^{a_j}e^{a_k}}{(\sum_j e^{a_j})^2} = \frac{e^{a_k}}{\sum_j e^{a_j}}\delta_{kj} - \frac{e^{a_j}}{\sum_j e^{a_j}} \frac{e^{a_k}}{\sum_j e^{a_j}}\\
&= \phi_k \cdot(\delta_{kj}-\phi_j) \,.
\end{align*}$$
Take the derivative of $\mathrm{L}(w)$ (or: how to spend an hour ...)
$$\begin{align*}
\nabla_{w_j} \mathrm{L}(w) &= \sum_{n,k} \frac{\partial \mathrm{L}_{nk}}{\partial p_{nk}} \cdot\frac{\partial p_{nk}}{\partial a_{nj}}\cdot\frac{\partial a_{nj}}{\partial w_j} \\
&= \sum_{n,k} \frac{y_{nk}}{p_{nk}} \cdot p_{nk} (\delta_{kj}-p_{nj}) \cdot x_n \\
&= \sum_n \Big( y_{nj} (1-p_{nj}) -\sum_{k\neq j} y_{nk} p_{nj} \Big) \cdot x_n \\
&= \sum_n \left( y_{nj} - p_{nj} \right)\cdot x_n \\
&= \sum_n \Big( \underbrace{y_{nj}}_{\text{target}} - \underbrace{\frac{e^{w_j^T x_n}}{\sum_{j^\prime} e^{w_{j^\prime}^T x_n}}}_{\text{prediction}} \Big)\cdot x_n
\end{align*}$$
persist_js_state$25f12528-d294-11ef-0c65-97c61935e9c2running§runtime wdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f12528-d294-11ef-0c65-97c61935e9c2depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& ʷhas_pluto_hook_features¤body5
Since $y_n \in \{0,1\}$ , it follows that $p(y_n =0 \,|\, x_n, w) = 1- \sigma(w^T x_n)$ .
In summary, the posterior class distribution $p(y_n \,|\, x_n, w)$ in logistic regression is specified as
$$\begin{align*}
p(y_n \,|\, x_n, w) &= \mathrm{Bernoulli}\left(y_n \,|\, \sigma(w^T x_n) \right) \\
&= \sigma(w^T x_n)^{y_n} \left(1 - \sigma(w^T x_n)\right)^{(1-y_n)} \tag{B-4.89} \\
&= \sigma\left( (2y_n-1) w^T x_n\right)
\end{align*}$$
Note that for the 3rd equality, we have made use of the fact that $\sigma(-a) = 1-\sigma(a)$ .
Each of these three models in B-4.89 are equivalent . We mention all three notational options since they all appear in the literature.
persist_js_state$3422dd29-6da9-4e0f-a4ab-646f223c2244running§runtime Idepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3422dd29-6da9-4e0f-a4ab-646f223c2244depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&% has_pluto_hook_features¤bodyي
Working out Numerics with Laplace Approximation
persist_js_state$5d91a2ae-975c-4913-b8ff-3df043cb1f03running§runtime Ƹdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$5d91a2ae-975c-4913-b8ff-3df043cb1f03depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&.㬷has_pluto_hook_features¤bodyT
Recap Classification
persist_js_state$0e713a7c-f4fa-4b86-abb3-26520e24d989running§runtime-depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$0e713a7c-f4fa-4b86-abb3-26520e24d989depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ʷhas_pluto_hook_features¤body
persist_js_state$25ef12bc-d294-11ef-1557-d98ba829a804running§runtime mƸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25ef12bc-d294-11ef-1557-d98ba829a804depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&'has_pluto_hook_features¤body
Preliminaries
Goal
Materials
persist_js_state$25f19230-d294-11ef-2dfd-6d4927e86f57running§runtime Zdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f19230-d294-11ef-2dfd-6d4927e86f57depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&1has_pluto_hook_features¤body
Discriminative Training or Discriminative Models?
In this lecture series, we presented two approaches to classification, namely the generative and the discriminative approach.
While the discriminative approach is intuitive and effective for many tasks, it sits somewhat uncomfortably with the Bayesian modeling approach outlined in the Bayesian Machine Learning lecture. Specifically, the discriminative approach does not define a full joint model over all variables in the system. Instead, it focuses only on modeling the conditional distribution $p(y | x)$ , effectively ignoring the input distribution $p(x)$ that would normally be part of a fully generative Bayesian model.
In a short paper by T. Minka (2005) , the model assumptions underlying discriminative classification are reinterpreted as arising from a special case of a generative model. This effectively restores discriminative approaches as fully compatible with the Bayesian modeling framework. (Note: the Minka paper is not required reading for the exam.)
persist_js_state$a00c545c-2274-4086-94ca-319d1436fa26running§runtimeødepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$a00c545c-2274-4086-94ca-319d1436fa26depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&"has_pluto_hook_features¤body
Exercises persist_js_state$22121f20-6b90-4782-8bed-25486cc23ae7running§runtime Qdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$22121f20-6b90-4782-8bed-25486cc23ae7depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&3rhas_pluto_hook_features¤body
persist_js_state$e7c45ff8-9fa2-4ea3-a06f-5769d877540erunning§runtime *depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$e7c45ff8-9fa2-4ea3-a06f-5769d877540edepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Yhas_pluto_hook_features¤bodyP
persist_js_state$25f1b404-d294-11ef-1c3a-a5a8142bb202running§runtime Hdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f1b404-d294-11ef-1c3a-a5a8142bb202depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&"߷has_pluto_hook_features¤body;
While Eq. B-4.145 gives the expression for the Bayesian predictive class distribution, the integral becomes analytically intractable when we substitute the posterior distribution over weights, $p(w | D)$ (from Eq. B-4.142), into it :(
persist_js_state$7932fff4-0568-49de-b34c-711e51487ae3running§runtimedepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$7932fff4-0568-49de-b34c-711e51487ae3depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&yQhas_pluto_hook_features¤body$
Challenge Revisited: Bayesian Logistic Regressionpersist_js_state$1f2bfcf4-fef4-4612-8683-d5c86a326eefrunning§runtime ldepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$1f2bfcf4-fef4-4612-8683-d5c86a326eefdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&.fAhas_pluto_hook_features¤bodyL
Closing Thoughts
persist_js_state$9554ed0b-69dd-443c-9538-03a4117eeb78running§runtime ?depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$9554ed0b-69dd-443c-9538-03a4117eeb78depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Rhas_pluto_hook_features¤body3
Click to see proof
$$\begin{align}
\frac{ p(y_n =1 |x_n) }{ p(y_n =0|x_n) } &= \frac{ \sigma(w^T x_n)}{1 - \sigma(w^T x_n)} \\
&= \frac{ \frac{1}{1+\exp(-w^T x_n)} }{ 1 -\frac{1}{1+\exp(-w^T x_n)} } \\
&= \frac{1}{1+\exp(-w^T x_n) - 1} \\
&= \exp(w^T x)
\end{align}$$
Setting $\exp(w^T x) \overset{!}{=} 1$ , leads to
$$ w^T x = 0$$
for the discrimination boundary, which is a line.
persist_js_state$aeee1072-5173-4eae-8027-3fbf2e338d95running§runtime e֘depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$aeee1072-5173-4eae-8027-3fbf2e338d95depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤bodyo
Why Not Generative Classification?
Like in the previous lecture , let's try to fit a generative distribution to the data. This is what we get when fitting two Gaussians to the two classes:
persist_js_state$22f4972e-0f6d-49a1-bd1f-10569b77a852running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$22f4972e-0f6d-49a1-bd1f-10569b77a852depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Hhas_pluto_hook_features¤body
🎯 Key concept
Discriminative classification follows the approach we used for Regression, namely, a direct (conditional) model
$$\begin{align}
p(y_n \,|\, x_n, w) &= \mathrm{Bernoulli}\left(y_n \,|\, \sigma(w^T x_n) \right) \\
p(w) &= \mathcal{N}(w \,|\, m_0, S_0)
\end{align}$$
for outputs $y_n$ , given inputs $x_n$ .
persist_js_state$25f3bef2-d294-11ef-1438-e9f7e469336frunning§runtime |=depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f3bef2-d294-11ef-1438-e9f7e469336fdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&*QHhas_pluto_hook_features¤body2
Let us perform Bayesian inference to estimate the posterior distribution of $w$ given the data set from the introduction. To allow an offset in the discrimination boundary, we add a constant 1 to the feature vector $x$ .
persist_js_state$cf829697-6283-4d2f-b0dd-bbfbd689a145running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$cf829697-6283-4d2f-b0dd-bbfbd689a145depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&2has_pluto_hook_features¤bodyJ
Data Generation
persist_js_state$7e7cab21-09ab-4d06-9716-ab7864b229abrunning§runtime %depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$7e7cab21-09ab-4d06-9716-ab7864b229abdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Rhas_pluto_hook_features¤body\
persist_js_state$75076ca1-7226-4229-8e70-06cc586507c3running§runtime&`depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$75076ca1-7226-4229-8e70-06cc586507c3depends_on_disabled_cells¦queued¤logsoutputmimeimage/svg+xmlrootassigneelast_run_timestampAڹ&= has_pluto_hook_features¤body
persist_js_state$25f14f82-d294-11ef-02fb-2dc632b8f118running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f14f82-d294-11ef-02fb-2dc632b8f118depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& ?has_pluto_hook_features¤body
Note that, in the discriminative approach, the regression parameters $w$ are not structured into $\{\mu,\Sigma,\pi \}$ . In principle, they are "free" parameters for which we can choose any value that seems appropriate. This provides discriminative approach with more flexibility than the generative approach.
persist_js_state$25eefb10-d294-11ef-0734-2daf18636e8erunning§runtimeͨdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25eefb10-d294-11ef-0734-2daf18636e8edepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ѐ%has_pluto_hook_features¤body
Discriminative Classificationpersist_js_state$5e4bb719-ea9b-4a30-8800-5d753f405fd1running§runtimë́Ƹdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$5e4bb719-ea9b-4a30-8800-5d753f405fd1depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&0)has_pluto_hook_features¤bodypersist_js_state$25f19ed8-d294-11ef-3298-efa16dda1dderunning§runtime kdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f19ed8-d294-11ef-3298-efa16dda1ddedepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& ҷhas_pluto_hook_features¤bodyParameter Inference
Note that for generative classification, for the sake of simplicity, we used maximum likelihood estimation for the model parameters. We could have used Bayesian parameter estimation for the generative classification model but the math is not suited for an introductory lesson.
In this lesson on discriminative classification, we specify both a prior and likelihood function for the parameters $w$ , which allows us to compute a Bayesian posterior for the weights.
As before, once the model is specified, everything else follows directly from the rules of probability theory.
Note that the scaled standard normal CDF closely approximates the logistic sigmoid function, with
$$\Phi\big(\sqrt{\pi/8} a\big) \approx \sigma(a)\,.$$
Our task will be the same as in the preceding class on (generative) classification. But this time, the class-conditional data distributions look very non-Gaussian, yet the linear discriminative boundary looks easy enough.
The posterior for the weights follows by Bayes rule,
$$\begin{align*}
\underbrace{p(w \,|\, D)}_{\text{posterior}} &= \frac{p(w) p(D|w)}{\int p(w) p(D|w) \mathrm{d}w} \\ &= \frac{\overbrace{\mathcal{N}(w \,|\, m_0, S_0)}^{\text{prior}} \cdot \overbrace{\prod_{n=1}^N \sigma\left( (2y_n-1) w^T x_n\right)}^{\text{likelihood}}}{\underbrace{\int \mathcal{N}(w \,|\, m_0, S_0) \prod_{n=1}^N \sigma\left( (2y_n-1) w^T x_n\right) \mathrm{d}w}_{\text{evidence}}} \tag{B-4.142}
\end{align*}$$
In principle, Bayesian learning of the parameters is done now!
Unfortunately, the posterior $p(w | D)$ is not Gaussian, and the evidence $p(D)$ is also not analytically computable. (We will deal with this later).
persist_js_state$fcec3c3a-8b0b-4dfd-b010-66abbf330069running§runtime 6@depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$fcec3c3a-8b0b-4dfd-b010-66abbf330069depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body1generate_dataset (generic function with 1 method)persist_js_state$25f14226-d294-11ef-369f-e545d5fe2700running§runtime Cdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f14226-d294-11ef-369f-e545d5fe2700depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& ohas_pluto_hook_features¤body\This choice for the class posterior is called logistic regression , in analogy to linear regression :
$$\begin{align}
p(y_n|x_n,w) &= \mathcal{N}(y_n|w^T x_n,\beta^{-1}) \tag{for linear regression} \\
p(y_n|x_n,w) &= \mathrm{Bernoulli}\left(y_n \,|\, \sigma(w^T x_n) \right) \tag{for logistic regression}
\end{align}$$
Sometimes, the precise assumptions of the (Gaussian-Categorical) generative model
$$p(x_n,y_n\in\mathcal{C}_k|\theta) = \pi_k \cdot \mathcal{N}(x_n|\mu_k,\Sigma)$$
clearly do not match the data distribution.
predictive_posterior
Computes the predictive posterior eq. B-4.152 using the given approximation to the sigmoid function.
persist_js_state$25f365e2-d294-11ef-300e-9914333b1233running§runtime 3ødepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f365e2-d294-11ef-300e-9914333b1233depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&-6has_pluto_hook_features¤bodyXRather than the computationally involved Laplace approximation, in practice, discriminative classification is often executed through maximum likelihood estimation.
With the usual 1-of-K encoding scheme for classes,
$$y_{nk} = \begin{cases} 1 & \text{if } x_n \in \mathcal{C}_k \\
0 & \text{otherwise} \,,\end{cases}$$
the log-likelihood for a $K$ -dimensional discriminative classifier evaluates to
$$\begin{align*}
\mathrm{L}(w) &= \log \prod_n \prod_k {p(\mathcal{C}_k|x_n,w)}^{y_{nk}} \\
&= \sum_n \sum_k y_{kn} \log \bigg( \underbrace{\frac{e^{w_k^T x_n}}{ \sum_j e^{w_j^T x_n}}}_{=\text{softmax}(w_k^T x_n)} \bigg)
\end{align*}$$
The gradient $\nabla_{w_k} \mathrm{L}(w)$ to the weight $w_k$ can be worked out to
$$\nabla_{w_k} \mathrm{L}(w) = \sum_n \underbrace{\big( \underbrace{y_{nk}}_{\text{target}} - \underbrace{\frac{e^{w_k^T x_n}}{ \sum_j e^{w_j^T x_n}}}_{\text{prediction}} \big)}_{\text{prediction error}}\cdot x_n .$$
For the data set $D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}$ , the likelihood function for the parameters $w$ is given by
$$p(D|w) = \prod_{n=1}^N p(y_n|x_n,w) = \prod_{n=1}^N \sigma\left( (2y_n-1) w^T x_n\right)$$
bayesian_discrimination_boundary
This function computes the posterior distribution over regression weights using the Laplace Approximation. We use logσ as a numerically stable alternative to logistic, and we avoid matrix inversions by computing the precision matrix of the posterior distribution instead of the covariance.
The math in this function corresponds to eq. B-4.143
persist_js_state$98ef7093-f8ed-4a44-a153-0a64ab483f65running§runtime ¸depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$98ef7093-f8ed-4a44-a153-0a64ab483f65depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&*Rhas_pluto_hook_features¤bodyH
Implementation
persist_js_state$f37ac438-cd56-4a09-bc8e-a75469955a2frunning§runtime,RjJdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$f37ac438-cd56-4a09-bc8e-a75469955a2fdepends_on_disabled_cells¦queued¤logsoutputmimeimage/svg+xmlrootassigneelast_run_timestampAڹ&Qhas_pluto_hook_features¤body
persist_js_state$25f3ff84-d294-11ef-0031-63b23d23324drunning§runtime X8depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$25f3ff84-d294-11ef-0031-63b23d23324ddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&.ȗhas_pluto_hook_features¤bodyYWhy be Bayesian?
Why should you embrace the Bayesian approach to logistic regression? After all, Maximum Likelihood for logistic regression seems simpler.
Still, consider the following:
Bayesian logistic regression with the Laplace approximation ultimately leads to simple closed-form classification rules . Moreover, modern probabilistic programming languages and packages can automate the inference derivations above. (We just do them here to gain insight into a difficult inference process.)
Bayesian logistic regression allows for the computation of model evidence, enabling principled comparison of model performance across alternative models.
Perhaps most importantly, Bayesian logistic regression processes uncertainties; e.g., in places where almost no data is observed, the posterior class probability will pull back to the prior class probability rather than predicting some arbitrary probability.
N = 120 Model Specification
We will work this idea out for a 2-class problem. Assume a data set is given by
$$D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}$$
with $x_n \in \mathbb{R}^M$ and $y_n \in \{0,1\}$ .
persist_js_state$0045e569-dc3c-4998-86da-9d96f599c599running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$0045e569-dc3c-4998-86da-9d96f599c599depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&+yhas_pluto_hook_features¤bodyf
Maximum Likelihood Estimation
persist_js_state$ff31d8c1-db35-4c85-a609-67fc40e9e78drunning§runtime rdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$ff31d8c1-db35-4c85-a609-67fc40e9e78ddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&.has_pluto_hook_features¤bodyBThe parameter vector $w$ for logistic regression can then be estimated through iterative gradient-based adaptation. For instance, start with a random weight $\hat{w} = w_0$ , and iterate through
$$\hat{w}^{(i+1)} = \hat{w}^{(i)} + \eta \cdot \left. \nabla_w \mathrm{L}(w) \right|_{w = \hat{w}^{(i)}}$$
until convergence.
Generative Discriminative (ML)
1 Like density estimation , model joint prob.
```math
p(\\mathcal{C}_k) p(x|\\mathcal{C}_k) = \\pi_k \\mathcal{N}(\\mu_k,\\Sigma)
```
Like (linear) regression , model conditional
```math
p(\\mathcal{C}_k|x,\\theta)
```
2 Leads to softmax posterior class probability
```math
p(\\mathcal{C}_k|x,\\theta ) = e^{\\theta_k^T x}/Z
```
with structured ``\\theta`` Choose also softmax posterior class probability
```math
p(\\mathcal{C}_k|x,\\theta ) = e^{\\theta_k^T x}/Z
```
but now with 'free' ``\\theta``
3
For Gaussian ``p(x|\\mathcal{C}_k)`` and multinomial priors,
```math
\\hat \\theta_k = \\left[ {\\begin{array}{c}
{ - \\frac{1}{2} \\mu_k^T \\sigma^{-1} \\mu_k + \\log \\pi_k} \\\\
{\\sigma^{-1} \\mu_k } \\\\
\\end{array}} \\right]
```
in one shot . Find ``\\hat\\theta_k`` through gradient-based adaptation
```math
\\nabla_{\\theta_k}\\mathrm{L}(\\theta) = \\sum_n \\Big( y_{nk} - \\frac{e^{\\theta_k^T x_n}}{\\sum_{k^\\prime} e^{\\theta_{k^\\prime}^T x_n}} \\Big)\\, x_n
```
"""$9704ee6a-e233-49f1-8c66-d81543927342metadatadisabled©show_logsîskip_as_script§cell_id$9704ee6a-e233-49f1-8c66-d81543927342code_foldedäcodefkeyconcept("",
md"""
It is easy to write down the Bayesian equations for the weight posterior and the predictive distribution of ``y_\bullet``, but in practice these do not admit closed-form solutions. A traditional workaround is the **Laplace approximation**, which has long been used to approximate the marginalization integrals in Bayesian inference.
""")$38b4854f-be02-4696-802f-2106481e3aeametadatadisabled©show_logsîskip_as_script§cell_id$38b4854f-be02-4696-802f-2106481e3aeacode_foldedäcode@md"""
## Bayesian Processing of Uncertainties
We now make an important observation: According to Eq. B-4.143, the posterior covariance matrix ``S_N`` of the weight vector depends on both the prior variance ``S_0`` and the distribution of the training data ``\{(x_n, y_n)\}_{n=1}^N``. In regions with limited training data and/or an uninformative prior (i.e., large ``S_0``), the posterior uncertainty about the weights remains high. This increased uncertainty raises ``\sigma_a^2`` in Eq. B-4.150, **causing the posterior class probability in Eq. B-4.152 to approach ``0.5``** (since ``\Phi(0) = 0.5``, see [Gaussian CDF image](https://en.wikipedia.org/wiki/Normal_distribution#/media/File:Normal_Distribution_CDF.svg)), thereby reflecting greater uncertainty in the prediction.
In other words, if you draw a new feature ``x_\bullet`` from a region with little training data, then the predictive class probability ``p(y_\bullet | x_\bullet, D)`` naturally tends toward ``0.5``, a built-in expression of uncertainty, courtesy of the Bayesian framework.
In contrast, if you eliminate uncertainty by representing the weights as fixed-point estimates (i.e., as mere numbers), then the model becomes overconfident. It will still produce sharp predictions, even in regions where it has seen no data, exactly when it should be most uncertain.
"""$25f0adde-d294-11ef-353e-4b4773df9ff5metadatadisabled©show_logsîskip_as_script§cell_id$25f0adde-d294-11ef-353e-4b4773df9ff5code_foldedäcodemd"""
#### Data-generating distribution
We will take inspiration from the [generative classification](https://bmlip.github.io/course/lectures/Generative%20Classification.html#softmax) approach, where we derived the class posterior
```math
p(y_{nk} = 1\,|\,x_n,\beta_k,\gamma_k) = \sigma(\beta_k^T x_n + \gamma_k)
```
as a **softmax** function of a linear map of the input.
Here, for the 2-class problem, we *choose* the 2-class softmax function (which is called the [**logistic** function](https://en.wikipedia.org/wiki/Logistic_function)) with linear discrimination bounderies for the posterior class probability:
```math
p(y_n =1 \,|\, x_n, w) = \sigma(w^T x_n) \,.
```
where
```math
\sigma(a) = \frac{1}{1+e^{-a}}
```
is the *logistic* function.
"""$25ef6ece-d294-11ef-270a-999c8d457b24metadatadisabled©show_logsîskip_as_script§cell_id$25ef6ece-d294-11ef-270a-999c8d457b24code_foldedäcode٫md"""
## Framework
A data set is given by ``D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}`` with ``x_n \in \mathbb{R}^M`` and ``y_n \in \mathcal{C}_k``, with ``k=1,\ldots,K``.
"""$8b0bb225-bdc1-45ec-bd34-68d674d6f08dmetadatadisabled©show_logsîskip_as_script§cell_id$8b0bb225-bdc1-45ec-bd34-68d674d6f08dcode_foldedäcodemd"""
The [Laplace Approximation](https://en.wikipedia.org/wiki/Laplace%27s_approximation) approximates a function by a Gaussian-shaped function. In this case, we will approximate the weight posterior ``p(w|D)`` (Eq. B-4.142) by a Gaussian distribution
```math
q(w) = \mathcal{N}\left(w\,|\, m_{N}, S_N\right) \tag{B-4.144}
```
with
```math
\begin{align}
m_N &= \arg\max_w \log p(w|D) \\
S_N^{-1} &= S_0^{-1} + \sum_n \sigma_n (1-\sigma_n) x_n x_n^T \tag{B-4.143}
\end{align}
```
where we used short-hand ``\sigma_n = \sigma\left((2y_n-1) m_N^T x_n\right)``.
If we substitute the Gaussian approximation from Eq. B-4.143 into the expression for the predictive class distribution (Eq. B-4.145), we obtain (after some additional approximations):
```math
\begin{align*}
p(y_\bullet = 1 \mid x_\bullet, D) &= \int p(y_\bullet = 1 \,|\, x_\bullet, w) \cdot p(w\,|\, D) \,\mathrm{d}w \\
&\approx \int p(y_\bullet = 1 \,|\, x_\bullet, w) \cdot q(w) \,\mathrm{d}w \\
&= \int \sigma(w^T x_\bullet) \cdot \mathcal{N}\left(w \,|\, m_N, S_N\right) \,\mathrm{d}w \tag{B-4.145} \\
&\approx \Phi\left( \frac{\mu_a}{\sqrt(\lambda^{-2} +\sigma_a^2)}\right) \tag{B-4.152}
\end{align*}
```
where
```math
\begin{align}
\lambda^2 &= \pi / 8 \\
\mu_a &= m^T_{N} x_\bullet \tag{B-4.149} \\
\sigma_a^2 &= x^T_\bullet S_N x_\bullet \tag{B-4.150}
\end{align}
```
and ``\Phi(x)= \frac{1}{\sqrt(2\pi)}\int_{-\infty}^{x}e^{-t^2/2}\mathrm{d}t`` is the Standard Normal [cumulative distribution function](https://en.wikipedia.org/wiki/Cumulative_distribution_function) (CDF).
Although the intermediate equations may look intimidating, the final result for the predictive distribution Eq. B-4.152 has a simple closed-form expression.
"""$33b859f2-9ea8-4f8b-b0f8-08a19c6a96fcmetadatadisabled©show_logsîskip_as_script§cell_id$33b859f2-9ea8-4f8b-b0f8-08a19c6a96fccode_foldedäcodeQNotebookCard("https://bmlip.github.io/course/minis/Laplace%20Approximation.html")$fe66a986-2f55-4417-a71d-b3b99f6369ccmetadatadisabled©show_logsîskip_as_script§cell_id$fe66a986-2f55-4417-a71d-b3b99f6369cccode_foldedäcodeechallenge_statement("difficult class-conditional data distributions" , header_level=1, color= "red" )$e4cc517b-d3b5-4517-a28b-efb8aba24496metadatadisabled©show_logsîskip_as_script§cell_id$e4cc517b-d3b5-4517-a28b-efb8aba24496code_foldedäcodeٷmd"""
The numerical issues associated with the Laplace approximation and the evaluation of the predictive class distribution are discussed in detail in the following mini-lecture.
"""$25f1c2a0-d294-11ef-009c-69b64e87e5fbmetadatadisabled©show_logsîskip_as_script§cell_id$25f1c2a0-d294-11ef-009c-69b64e87e5fbcode_foldedäcodemd"""
Many methods have been developed to approximate these types of Bayesian integrals. Here, we present the **Laplace approximation**, which is one of the simplest methods with broad applicability to Bayesian calculations.
"""$173e545a-9125-4dc4-8df4-c3aa84a4b88ametadatadisabled©show_logsîskip_as_script§cell_id$173e545a-9125-4dc4-8df4-c3aa84a4b88acode_foldedäcodemd"""
# Summary
"""$d1bbdc6a-e5ff-4cd6-9175-860b5ec04f3cmetadatadisabled©show_logsîskip_as_script§cell_id$d1bbdc6a-e5ff-4cd6-9175-860b5ec04f3ccode_foldedäcode(md"""
# Bayesian Logistic Regression
"""$69706576-0333-43bf-8523-e7838f373529metadatadisabled©show_logsîskip_as_script§cell_id$69706576-0333-43bf-8523-e7838f373529code_foldedäcodemd"""
Note that we get a full predictive posterior distribution over the assignment of class labels for every datapoint, so instead of assigning a point to a class, we get a measure of uncertainty over our class assignment!
"""$fce5d561-ea76-4bd8-9cce-6f707f72fc60metadatadisabled©show_logsîskip_as_script§cell_id$fce5d561-ea76-4bd8-9cce-6f707f72fc60code_folded¤codebayesian_plot = let
X_ext = vcat(X, ones(1, length(y)))
# Define a prior distribution over parameters, play with this to see the result change!
prior = MvNormalMeanCovariance(zeros(3), 100 .* diagm(ones(3)))
posterior = bayesian_discrimination_boundary(prior, X_ext, y)
# Plot 50% boundary
θ = mean(posterior)
disc_boundary(x1) = -1 / θ[2] * (θ[1]*x1 + θ[3])
plot_dataset()
plot!([-2., 10.], disc_boundary; label="Discr. boundary", linewidth=2)
# Plot heatmap
xrange = range(-1.6, 9; length=50)
yrange = range(-2, 7; length=30)
heatmap!(
xrange, yrange, (x,y) -> predictive_posterior([x, y, 1], posterior);
alpha=0.5, color=:redblue,
)
end;$66351c02-1921-44dc-b461-84a536c40fd5metadatadisabled©show_logsîskip_as_script§cell_id$66351c02-1921-44dc-b461-84a536c40fd5code_foldedäcodemd"""
_The logistic function_ ``\sigma(a) = 1/(1+e^{-a})`` _(red), together with the scaled Standard Normal_ [cumulative distribution function](https://en.wikipedia.org/wiki/Cumulative_distribution_function) _(CDF)_ ``\Phi(\lambda a) = \frac{1}{\sqrt(2\pi)}\int_{-\infty}^{a}e^{-t^2/2}\mathrm{d}t``, _for_ ``\lambda^2=\pi/8`` _(in blue). We will use this approximation later in the_ [Laplace approximation](https://bmlip.github.io/course/minis/Laplace%20Approximation.html#gaussian-cdf). _Based on Bishop fig.4.9._
"""$ad196ae6-c65e-4aaa-b0cc-bd72daa41952metadatadisabled©show_logsîskip_as_script§cell_id$ad196ae6-c65e-4aaa-b0cc-bd72daa41952code_folded¤codeusing MarkdownLiteral: @mdx$93083660-6a49-4147-b00c-d62a4453f222metadatadisabled©show_logsîskip_as_script§cell_id$93083660-6a49-4147-b00c-d62a4453f222code_foldedäcodemd"""
That's not a good fit for the $y=1$ class!
Now, we could continue down this road and try to fit a more complex distribution to the ``y = 1`` class to make it fit. **But let's explore a different approach**, modeling the discriminative boundary directly.
"""$b94644f8-725d-49bf-9641-3dad8b647f45metadatadisabled©show_logsîskip_as_script§cell_id$b94644f8-725d-49bf-9641-3dad8b647f45code_foldedäcode
md"""
#### Discrimination boundaries (*)
Show that for logistic regression with ``p(y_n =1 \,|\, x_n) = \sigma(w^T x_n)``, the discrimination boundary, which can be computed by
```math
\frac{p(y_n =1|x_n)}{p(y_n =0|x_n)} \overset{!}{=} 1
```
is a straight line.
"""$4ceede48-a4d5-446b-bb34-26cec4af357ametadatadisabled©show_logsîskip_as_script§cell_id$4ceede48-a4d5-446b-bb34-26cec4af357acode_foldedäcodeKbegin
N_bond = @bindname N Slider(8:200; default=120, show_value=true)
end$a65ca01a-0e9a-42cb-b1d7-648102a77eb5metadatadisabled©show_logsîskip_as_script§cell_id$a65ca01a-0e9a-42cb-b1d7-648102a77eb5code_foldedäcode_function plot_dataset()
result = scatter(X_c1[:,1], X_c1[:,2],markersize=4, label=L"y=0", xlabel=L"x_1", ylabel=L"x_2", xlims=(-1.6, 9), ylims=(-2, 7))
scatter!(X_c2[:,1], X_c2[:,2],markersize=4, label=L"y=1")
scatter!([X_test[1]], [X_test[2]], markersize=7, marker=:star, label=L"y=?")
plot!(legend=:bottomright)
return result
end$25efa2fe-d294-11ef-172f-9bb09277f59emetadatadisabled©show_logsîskip_as_script§cell_id$25efa2fe-d294-11ef-172f-9bb09277f59ecode_foldedäcodemd"""
Here's an **IDEA**! Let's model the posterior
```math
p(y_n\in\mathcal{C}_k|x_n)
```
*directly*, without any assumptions on the prior class densities ``p(x)``. This approach is called [Discriminative Classification](https://en.wikipedia.org/wiki/Discriminative_model).
"""$6eee35ee-fd55-498f-9441-f18c2508de19metadatadisabled©show_logsîskip_as_script§cell_id$6eee35ee-fd55-498f-9441-f18c2508de19code_foldedäcodemd"""
# Code
"""$6b56ec96-4b9d-4281-bd63-061df324867fmetadatadisabled©show_logsîskip_as_script§cell_id$6b56ec96-4b9d-4281-bd63-061df324867fcode_folded¤code8X_c1 = X[:,findall(.!y)]' # Split X based on class label$b8790891-1546-48e0-9f96-e09eade31c12metadatadisabled©show_logsîskip_as_script§cell_id$b8790891-1546-48e0-9f96-e09eade31c12code_foldedäcodelogσ(x) = -softplus(x)$4ab4bcca-da6e-4137-858d-257c04388277metadatadisabled©show_logsîskip_as_script§cell_id$4ab4bcca-da6e-4137-858d-257c04388277code_foldedäcode)keyconcept("",
md"""
The Laplace approximation renders a closed-form and interpretable (approximate) solution for the predictive class distribution,
```math
p(y_\bullet = 1 | x_\bullet, D) \approx \Phi\left( \frac{m^T_{N} x_\bullet}{\sqrt{(8/\pi) +x^T_\bullet S_N x_\bullet)}}\right)\,.
```
""")$25f02ac6-d294-11ef-26c4-f142b8ac4b5fmetadatadisabled©show_logsîskip_as_script§cell_id$25f02ac6-d294-11ef-26c4-f142b8ac4b5fcode_foldedäcodedmd"""
What model should we use for the posterior distribution ``p(y_n \in \mathcal{C}_k|x_n)``?
"""$3b24b142-2239-4951-9177-ff87b5da4b68metadatadisabled©show_logsîskip_as_script§cell_id$3b24b142-2239-4951-9177-ff87b5da4b68code_foldedäcode hide_proof(
md"""
The Log-likelihood is
```math
\mathrm{L}(w) = \log \prod_n \prod_k {\underbrace{p(y_{nk}=1|x_n,w)}_{p_{nk}}}^{y_{nk}} = \sum_{n,k} y_{nk} \log p_{nk}
```
Use the fact that the softmax ``\phi_k \equiv e^{a_k} / {\sum_j e^{a_j}}`` has analytical derivative,
```math
\begin{align*}
\frac{\partial \phi_k}{\partial a_j} &= \frac{(\sum_j e^{a_j})e^{a_k}\delta_{kj}-e^{a_j}e^{a_k}}{(\sum_j e^{a_j})^2} = \frac{e^{a_k}}{\sum_j e^{a_j}}\delta_{kj} - \frac{e^{a_j}}{\sum_j e^{a_j}} \frac{e^{a_k}}{\sum_j e^{a_j}}\\
&= \phi_k \cdot(\delta_{kj}-\phi_j) \,.
\end{align*}
```
Take the derivative of ``\mathrm{L}(w)`` (or: how to spend an hour ...)
```math
\begin{align*}
\nabla_{w_j} \mathrm{L}(w) &= \sum_{n,k} \frac{\partial \mathrm{L}_{nk}}{\partial p_{nk}} \cdot\frac{\partial p_{nk}}{\partial a_{nj}}\cdot\frac{\partial a_{nj}}{\partial w_j} \\
&= \sum_{n,k} \frac{y_{nk}}{p_{nk}} \cdot p_{nk} (\delta_{kj}-p_{nj}) \cdot x_n \\
&= \sum_n \Big( y_{nj} (1-p_{nj}) -\sum_{k\neq j} y_{nk} p_{nj} \Big) \cdot x_n \\
&= \sum_n \left( y_{nj} - p_{nj} \right)\cdot x_n \\
&= \sum_n \Big( \underbrace{y_{nj}}_{\text{target}} - \underbrace{\frac{e^{w_j^T x_n}}{\sum_{j^\prime} e^{w_{j^\prime}^T x_n}}}_{\text{prediction}} \Big)\cdot x_n
\end{align*}
```
""")$25f12528-d294-11ef-0c65-97c61935e9c2metadatadisabled©show_logsîskip_as_script§cell_id$25f12528-d294-11ef-0c65-97c61935e9c2code_foldedäcodemd"""
Since ``y_n \in \{0,1\}``, it follows that ``p(y_n =0 \,|\, x_n, w) = 1- \sigma(w^T x_n)``.
In summary, the posterior class distribution ``p(y_n \,|\, x_n, w)`` in logistic regression is specified as
```math
\begin{align*}
p(y_n \,|\, x_n, w) &= \mathrm{Bernoulli}\left(y_n \,|\, \sigma(w^T x_n) \right) \\
&= \sigma(w^T x_n)^{y_n} \left(1 - \sigma(w^T x_n)\right)^{(1-y_n)} \tag{B-4.89} \\
&= \sigma\left( (2y_n-1) w^T x_n\right)
\end{align*}
```
Note that for the 3rd equality, we have made use of the fact that ``\sigma(-a) = 1-\sigma(a)``.
Each of these three models in B-4.89 are **equivalent**. We mention all three notational options since they all appear in the literature.
"""$3422dd29-6da9-4e0f-a4ab-646f223c2244metadatadisabled©show_logsîskip_as_script§cell_id$3422dd29-6da9-4e0f-a4ab-646f223c2244code_foldedäcode0.6)) # p(y=true) = 0.6
# Sample class 1 conditional distribution
if rand(rng)<0.5
X[:,n] = [6.0; 0.5] .* rand(rng, 2) .+ [3.0; 6.0]
else
X[:,n] = sqrt(0.5) * randn(rng, 2) .+ [5.5, 0.0]
end
else
# Sample class 2 conditional distribution
X[:,n] = randn(rng, 2) .+ [1., 4.]
end
end
return (X, y)
end$25f14226-d294-11ef-369f-e545d5fe2700metadatadisabled©show_logsîskip_as_script§cell_id$25f14226-d294-11ef-369f-e545d5fe2700code_foldedäcodemd"""
This choice for the class posterior is called **logistic regression**, in analogy to [linear regression](https://bmlip.github.io/course/lectures/Regression.html#likelihood-function):
```math
\begin{align}
p(y_n|x_n,w) &= \mathcal{N}(y_n|w^T x_n,\beta^{-1}) \tag{for linear regression} \\
p(y_n|x_n,w) &= \mathrm{Bernoulli}\left(y_n \,|\, \sigma(w^T x_n) \right) \tag{for logistic regression}
\end{align}
```
"""$a759653c-0da4-40b7-9e9e-1e3d2e4df4eametadatadisabled©show_logsîskip_as_script§cell_id$a759653c-0da4-40b7-9e9e-1e3d2e4df4eacode_folded¤code!using Random, Plots, LaTeXStrings$25ef7f54-d294-11ef-3f05-0d85fe6e7a17metadatadisabled©show_logsîskip_as_script§cell_id$25ef7f54-d294-11ef-3f05-0d85fe6e7a17code_foldedäcodemd"""
Sometimes, the precise assumptions of the (Gaussian-Categorical) generative model
```math
p(x_n,y_n\in\mathcal{C}_k|\theta) = \pi_k \cdot \mathcal{N}(x_n|\mu_k,\Sigma)
```
clearly do not match the data distribution.
"""$fd908bf5-71a1-4ae8-8416-cc1fdf084dcbmetadatadisabled©show_logsîskip_as_script§cell_id$fd908bf5-71a1-4ae8-8416-cc1fdf084dcbcode_foldedäcodeQ"""
Computes the predictive posterior eq. B-4.152 using the given approximation to the sigmoid function.
"""
function predictive_posterior(x, weight_posterior)
λsq = π / 8
wN = mean(weight_posterior)
μ = wN' * x
σ = x' * cov(weight_posterior) * x
query_point = μ / (sqrt(inv(λsq) + σ ))
return normcdf(0, 1, query_point)
end$25f365e2-d294-11ef-300e-9914333b1233metadatadisabled©show_logsîskip_as_script§cell_id$25f365e2-d294-11ef-300e-9914333b1233code_foldedäcodemd"""
Rather than the computationally involved Laplace approximation, in practice, discriminative classification is often executed through maximum likelihood estimation.
With the usual 1-of-K encoding scheme for classes,
```math
y_{nk} = \begin{cases} 1 & \text{if } x_n \in \mathcal{C}_k \\
0 & \text{otherwise} \,,\end{cases}
```
the log-likelihood for a ``K``-dimensional discriminative classifier evaluates to
```math
\begin{align*}
\mathrm{L}(w) &= \log \prod_n \prod_k {p(\mathcal{C}_k|x_n,w)}^{y_{nk}} \\
&= \sum_n \sum_k y_{kn} \log \bigg( \underbrace{\frac{e^{w_k^T x_n}}{ \sum_j e^{w_j^T x_n}}}_{=\text{softmax}(w_k^T x_n)} \bigg)
\end{align*}
```
The gradient ``\nabla_{w_k} \mathrm{L}(w)`` to the weight ``w_k`` can be worked out to
```math
\nabla_{w_k} \mathrm{L}(w) = \sum_n \underbrace{\big( \underbrace{y_{nk}}_{\text{target}} - \underbrace{\frac{e^{w_k^T x_n}}{ \sum_j e^{w_j^T x_n}}}_{\text{prediction}} \big)}_{\text{prediction error}}\cdot x_n .
```
"""$25f1390c-d294-11ef-364d-17e4c93b9a57metadatadisabled©show_logsîskip_as_script§cell_id$25f1390c-d294-11ef-364d-17e4c93b9a57code_foldedäcodemd"""
For the data set ``D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}``, the **likelihood function** for the parameters ``w`` is given by
```math
p(D|w) = \prod_{n=1}^N p(y_n|x_n,w) = \prod_{n=1}^N \sigma\left( (2y_n-1) w^T x_n\right)
```
"""$f8bb4fcd-9b20-44e5-8e22-e792e74b69dfmetadatadisabled©show_logsîskip_as_script§cell_id$f8bb4fcd-9b20-44e5-8e22-e792e74b69dfcode_folded¤codeX_c2 = X[:,findall(y)]'$b48f8800-473d-48e4-ab78-eb07653db7a5metadatadisabled©show_logsîskip_as_script§cell_id$b48f8800-473d-48e4-ab78-eb07653db7a5code_foldedäcodeRfunction log_likelihood(w, X, y)
return sum(logσ.((2*y .- 1) .* (X' * w)))
end
$1bfac9c5-e5cf-4a70-b077-11bb00cb1482metadatadisabled©show_logsîskip_as_script§cell_id$1bfac9c5-e5cf-4a70-b077-11bb00cb1482code_foldedäcoded"""
This function computes the posterior distribution over regression weights using the Laplace Approximation. We use `logσ` as a numerically stable alternative to `logistic`, and we avoid matrix inversions by computing the precision matrix of the posterior distribution instead of the covariance.
The math in this function corresponds to eq. B-4.143
"""
function bayesian_discrimination_boundary(prior_w, X::Matrix, y::Vector{Bool})
m_0 = mean(prior_w)
p_0 = precision(prior_w)
negative_unnormalized_posterior = w -> -log_likelihood(w, X, y) - logpdf(prior_w, w)
MAP_w = Optim.minimizer(optimize(negative_unnormalized_posterior, zeros(3)))
σ_n = logistic.((2y .- 1) .* (X' * MAP_w))
inv_Σ = p_0
for i in 1:length(y)
slice = view(X, :, i)
inv_Σ .+= σ_n[i] * (1.0 - σ_n[i]) .* (slice * slice')
end
return MvNormalMeanPrecision(MAP_w, inv_Σ)
end$98ef7093-f8ed-4a44-a153-0a64ab483f65metadatadisabled©show_logsîskip_as_script§cell_id$98ef7093-f8ed-4a44-a153-0a64ab483f65code_foldedäcodemd"""
## Implementation
"""$f37ac438-cd56-4a09-bc8e-a75469955a2fmetadatadisabled©show_logsîskip_as_script§cell_id$f37ac438-cd56-4a09-bc8e-a75469955a2fcode_foldedäcodeglet
d1 = fit_mle(MvNormal, X_c1')
d2 = fit_mle(MvNormal, X_c2')
plot_dataset()
xrange = range(-1.6, 9; length=20)
yrange = range(-2, 7; length=15)
contour!(
xrange, yrange,
(x,y) -> pdf(d1, [x,y]);
opacity=.4,
color=:blues,
)
contour!(
xrange, yrange,
(x,y) -> pdf(d2, [x,y]);
opacity=.4,
color=:red,
colorbar=nothing,
)
end$25f3ff84-d294-11ef-0031-63b23d23324dmetadatadisabled©show_logsîskip_as_script§cell_id$25f3ff84-d294-11ef-0031-63b23d23324dcode_foldedäcodemd"""
## Why be Bayesian?
Why should you embrace the Bayesian approach to logistic regression? After all, Maximum Likelihood for logistic regression seems simpler.
Still, consider the following:
* Bayesian logistic regression with the Laplace approximation ultimately leads to [simple closed-form classification rules](#Working-out-Numerics-with-Laplace-Approximation). Moreover, modern probabilistic programming languages and packages can automate the inference derivations above. (We just do them here to gain insight into a difficult inference process.)
* Bayesian logistic regression allows for the computation of model evidence, enabling principled comparison of model performance across alternative models.
* Perhaps most importantly, Bayesian logistic regression processes uncertainties; e.g., in places where almost no data is observed, the posterior class probability will pull back to the prior class probability rather than predicting some arbitrary probability.
"""$aaf764da-cf1b-4bc7-83ea-6d25a80ca3abmetadatadisabled©show_logsîskip_as_script§cell_id$aaf764da-cf1b-4bc7-83ea-6d25a80ca3abcode_foldedäcodeN_bond$25efd6b6-d294-11ef-3b21-6363ef531eb5metadatadisabled©show_logsîskip_as_script§cell_id$25efd6b6-d294-11ef-3b21-6363ef531eb5code_foldedäcodemd"""
## Model Specification
We will work this idea out for a 2-class problem. Assume a data set is given by
```math
D = \{(x_1,y_1),\dotsc,(x_N,y_N)\}
```
with ``x_n \in \mathbb{R}^M`` and ``y_n \in \{0,1\}`` .
"""$d29ccc9e-d4a6-46ae-b907-2bc68c8d99bcmetadatadisabled©show_logsîskip_as_script§cell_id$d29ccc9e-d4a6-46ae-b907-2bc68c8d99bccode_foldedäcodeplot_dataset()$0045e569-dc3c-4998-86da-9d96f599c599metadatadisabled©show_logsîskip_as_script§cell_id$0045e569-dc3c-4998-86da-9d96f599c599code_foldedäcode*md"""
# Maximum Likelihood Estimation
"""$ff31d8c1-db35-4c85-a609-67fc40e9e78dmetadatadisabled©show_logsîskip_as_script§cell_id$ff31d8c1-db35-4c85-a609-67fc40e9e78dcode_foldedäcode]md"""
The parameter vector ``w`` for logistic regression can then be estimated through iterative gradient-based adaptation. For instance, start with a random weight ``\hat{w} = w_0``, and iterate through
```math
\hat{w}^{(i+1)} = \hat{w}^{(i)} + \eta \cdot \left. \nabla_w \mathrm{L}(w) \right|_{w = \hat{w}^{(i)}}
```
until convergence.
"""$616e84d7-063d-4d9d-99e4-56aecf3c7ee4metadatadisabled©show_logsîskip_as_script§cell_id$616e84d7-063d-4d9d-99e4-56aecf3c7ee4code_folded¤codeousing Distributions, ExponentialFamily, LinearAlgebra, LogExpFunctions, StatsFuns, BayesBase, Optim, StableRNGslast_hot_reload_time shortpath Discriminative Classification.jlnbpkgbusy_packages,waiting_for_permission_but_probably_disabled§enabledðterminal_outputsLaTeXStrings&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mExponentialFamily&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mRandom&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mnbpkg_sync&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mStatsFuns&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mMarkdownLiteral&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mBmlipTeachingTools&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mDistributions&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mPlots&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mStableRNGs&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mLinearAlgebra&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mOptim&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mLogExpFunctions&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mBayesBase&
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_vhhcelnwnx/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39mwaiting_for_permission·restart_recommended_msgrestart_required_msginstalled_versionsLaTeXStrings1.4.1!__internal_julia_manifest_version1.13.3ExponentialFamily2.9.0RandomstdlibStatsFuns2.2.1MarkdownLiteral0.1.5BmlipTeachingTools1.4.1Distributions0.25.132Plots1.41.7StableRNGs1.1.0LinearAlgebrastdlib__internal_julia_version1.13.3Optim2.3.2LogExpFunctions1.0.3BayesBase1.5.12install_time_ns*#Tinstantiatedípluto_versionv1.1.0 
