Move the sliders below to create a 3-component vector $x = (x_1,x_2,x_3)$, where $x_i \in [-10:.1:10]$.
The softmax function (wikipedia), also called the normalized exponential function, is a mapping
$$\sigma : \mathbb{R}^M \;\rightarrow\; (0,1)^M, \quad M > 1,$$
defined component-wise as
$$\sigma(x)_i \;=\; \frac{\exp(x_i)}{\sum_{j=1}^M \exp(x_j)} , \quad i = 1, \dots, M.$$
The output of the softmax function defines the parameter vector of a Categorical distribution, regardless of the values of $x_i$, since all components $\sigma(x)_i$ are strictly positive and sum to one (see also the Multinomial Distributions lecture).
🎯 Key concept
The output of the logistic function $\sigma'(a)$ defines the parameter vector of a Bernoulli distribution, regardless of the values of $a$, since $\sigma'(a)$ is strictly positive and less than one.
🎯 Key concept
The output of the softmax function defines the parameter vector of a Categorical distribution, regardless of the values of $x_i$, since all components $\sigma(x)_i$ are strictly positive and sum to one.
Moreover, the softmax function preserves the relative order between entries: if $x_k > x_j$, then $\sigma(x)_k > \sigma(x)_j$. In particular,
$$\arg\max(x) = \arg\max(\sigma(x)) ,$$
which motivates the name softmax: it produces a “soft” probability distribution that emphasizes the maximum entry while still assigning nonzero weight to all others.
and display $y = \sigma(x)$
The logistic (or sigmoid) function (wikipedia) can be seen as a special case of the softmax function for $M=2$.
Let's work this out. The softmax for $M=2$ takes a vector $x = (x_1, x_2) \in \mathbb{R}^2$ and maps it to probabilities:
$$\sigma(x)_1 = \frac{e^{x_1}}{e^{x_1} + e^{x_2}}, \quad \sigma(x)_2 = \frac{e^{x_2}}{e^{x_1} + e^{x_2}}\, .$$
If we now look only at $\sigma(x)_1$, we can rewrite it as
$$\sigma(x)_1 = \frac{1}{1 + e^{-(x_1 - x_2)}} ,$$
which is the logistic (sigmoid) function
$$\sigma'(a) \triangleq \frac{1}{1 + e^{-a}}$$
applied to the difference $x_1 - x_2 \in \mathbb{R}$.
Choose any value in the range $[-10:.1:10]$
As we also see in Eq. B-4.89 in the Discriminative Classification lecture, the logistic function produces the parameters for a Bernoulli distribution.
And each entry lies between 0 and 1. Here are the lowest and highest values:
Check that the output of the softmax function sums to $1$.