Logistic 回归 损失函数推导及参数更新

Logistic 回归 损失函数推导及参数更新

Logistic 回归属于广义的线性模型，联系函数为sigmoid函数

广义的线性模型为 $y=g(w^{T}x+b)$y=g(wTx+b),函数 $g\left(.\right)$g(.)起到了将线性回归模型预测值和真实标记联系起来的作用，称为“联系函数”（link function）。使得 $g^{-1}\left(y\right)$g−1(y)与 $w^{T}x+b$wTx+b形成线性关系。

模型：
$z=w^{T}x+by=\frac{1}{1+e^{-z}}$z=wTx+by=1+e−z1​

损失函数推导：
采用极大似然估计来推导损失函数。
先写出正负样本的概率预测值：
$\begin{array}{cc}{\displaystyle p(y^{\ast }=1\mid x;w,b)}& {\displaystyle =y}\\ {\displaystyle p(y^{\ast }=0\mid x;w,b)}& {\displaystyle =1-y}\end{array}$p(y∗=1∣x;w,b)p(y∗=0∣x;w,b)​=y=1−y​
统一两个式子得到：
$p(y^{\ast }\mid x;w,b)=y^{y^{\ast }}(1-y{)}^{1-y^{\ast }}$p(y∗∣x;w,b)=yy∗(1−y)1−y∗
假设样本独立且同分布，要让对每个样本的概率值更接近其所属分类，列出极大似然函数：
$L(w;b)=\prod _{i=1}^{m}p(y_i^{\ast }\mid x_i,w,b)=\prod _{i=1}^m{y}_i^{y_i^{\ast }}(1-y_i{)}^{1-y_i^{\ast }}$L(w;b)=i=1∏m​p(yi∗​∣xi​,w,b)=i=1∏m​yiyi∗​​(1−yi​)1−yi∗​
取对数：
$l(w,b)=\sum _{i=1}^m\left[y_i^{\ast }log\right(y_i)+(1-y^{\ast }\left)log\right(1-y_i\left)\right])$l(w,b)=i=1∑m​[yi∗​log(yi​)+(1−y∗)log(1−yi​)])
损失函数一般求最小值，因此
$J(w,b)=-\frac{1}{m}\sum _{i=1}^m\left[y_i^{\ast }log\right(y_i)+(1-y^{\ast }\left)log\right(1-y_i\left)\right]$J(w,b)=−m1​i=1∑m​[yi∗​log(yi​)+(1−y∗)log(1−yi​)]
正好是交叉熵损失函数

梯度推导

计算图

$\frac{\partial J}{\partial w_j}=\sum _{i=1}^m\frac{\partial J}{\partial y_i}\frac{\partial y_i}{\partial z_i}\frac{\partial z_i}{\partial w_j}$∂wj​∂J​=i=1∑m​∂yi​∂J​∂zi​∂yi​​∂wj​∂zi​​

依次计算每一项

$J(w,b)=-\frac{1}{m}\sum _{i=1}^m\left[y_i^{\ast }log\right(y_i)+(1-y^{\ast }\left)log\right(1-y_i\left)\right]$J(w,b)=−m1​i=1∑m​[yi∗​log(yi​)+(1−y∗)log(1−yi​)]

$\begin{array}{cc}{\displaystyle \frac{\partial J}{\partial y_i}}& {\displaystyle =-\frac{1}{m}(\frac{y_i^{\ast }}{y_i}-\frac{1-y_i^{\ast }}{1-y_i})}\\ {\displaystyle }& {\displaystyle =-\frac{1}{m}\left(\frac{y_i^{\ast }-y_i}{y_i(1-y_i)}\right)}\end{array}$∂yi​∂J​​=−m1​(yi​yi∗​​−1−yi​1−yi∗​​)=−m1​(yi​(1−yi​)yi∗​−yi​​)​

$y=\frac{1}{1+e^{-z}}$y=1+e−z1​

$\begin{array}{cc}{\displaystyle \frac{\partial y_i}{\partial z_i}}& {\displaystyle =\frac{e^z(1+e^z)-e^z{e}^z}{(1+e^z{)}^2}}\\ {\displaystyle }& {\displaystyle =y_i(1-y_i)}\end{array}$∂zi​∂yi​​​=(1+ez)2ez(1+ez)−ezez​=yi​(1−yi​)​

$z=w^{T}x+b$z=wTx+b

$\frac{\partial z_i}{\partial w_j}=x_j$∂wj​∂zi​​=xj​
将三个导数计算结果代入上式，得到
$\begin{array}{cc}{\displaystyle \frac{\partial J}{\partial w_j}}& {\displaystyle =\sum _{i=1}^m-\frac{1}{m}\left(\frac{y_i^{\ast }-y_i}{y_i(1-y_i)}\right)\cdot y_i(1-y_i)\cdot x_j}\\ {\displaystyle }& {\displaystyle =\frac{1}{m}\sum _{i=1}^m(y_i-y_i^{\ast })x_j}\end{array}$∂wj​∂J​​=i=1∑m​−m1​(yi​(1−yi​)yi∗​−yi​​)⋅yi​(1−yi​)⋅xj​=m1​i=1∑m​(yi​−yi∗​)xj​​

参数更新公式

$\begin{array}{cc}{\displaystyle w_j}& {\displaystyle =w_j-\alpha \frac{\partial J}{\partial w_j}}\\ {\displaystyle }& {\displaystyle =w_j-\alpha \frac{1}{m}\sum _{i=1}^m(y_i-y_i^{\ast })x_j}\end{array}$wj​​=wj​−α∂wj​∂J​=wj​−αm1​i=1∑m​(yi​−yi∗​)xj​​