{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上一节我们看到可以使用逻辑回归来求解分类问题：\n",
    "\n",
    "$$\n",
    "ln \\frac{y}{1-y} = wx+b\n",
    "$$\n",
    "\n",
    "但是这里的 w 和 b 要怎么计算呢？前面在求解线性回归时，我们首先确定模型的损失函数，然后通过计算损失函数的最小值从而得到模型的各个参数。那么逻辑回归的损失函数该如何确定呢？\n",
    "\n",
    "如果按照最小二乘的求解思路，这里的 $(x, ln \\frac{y}{1-y})$ 呈线性关系，可以定义出损失函数：\n",
    "\n",
    "$$\n",
    "loss = \\sum_{i=1}^{n}(ln \\frac{y_i}{1-y_i} - (wx_i + b))^2\n",
    "$$\n",
    "\n",
    "但是很显然，$y_i$ 的取值不是 0 就是 1，代入上面的损失函数要么分母是 0 要么分子是 0，根本没法求解。回忆前面一节 S 型函数的拟合曲线，实际上，它的值域范围是 $(0, 1)$，并不是等于 0 或 1，只不过取值非常接近 0 或 1，我们不妨把这个值理解为可能性。\n",
    "\n",
    "若将 y 视为样本 x 作为正例的可能性，则 1-y 则是 x 作为反例的可能性，两者的比值 $\\frac{y}{1-y}$ 称为 **几率**（odds），反映了 x 作为正例的相对可能性。对几率取对数 $ln \\frac{y}{1-y}$ 就是 **对数几率**（log odds，亦称 logit），这也是对数几率函数名称的由来，而使用对数几率函数拟合分类数据的这种回归方法被称为 **对数几率回归**（logistic regression），可能是中文比较拗口，很多地方喜欢把它称为 **逻辑回归**（logistic regression， 或者 **对数回归**）。不过要注意的是，虽然名字中有回归两个字，但它解决的是分类问题，它不仅能够预测“类别”，还能给出近似的概率预测。\n",
    "\n",
    "这样我们就可以把 $y$ 改写成 $p(y = 1 | x)$，这是一个条件概率，表示当 $x$ 发生时 $y = 1$ 的概率，这个概率也被称为 **后验概率**，同理，我们把 $1-y$ 改写成 $p(y = 0 | x)$ 表示当 $x$ 发生时 $y = 0$ 的概率。所以有：\n",
    "\n",
    "$$\n",
    "ln \\frac{p(y = 1 | x)}{p(y = 0 | x)} = wx+b\n",
    "$$\n",
    "\n",
    "并把分类问题改写成下面的条件概率形式：\n",
    "\n",
    "$$\n",
    "p(y = 1 | x) = \\frac{e^{wx+b}}{1+e^{wx+b}}\n",
    "$$\n",
    "\n",
    "$$\n",
    "p(y = 0 | x) = \\frac{1}{1+e^{wx+b}}\n",
    "$$\n",
    "\n",
    "这就是逻辑回归模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 极大似然法解逻辑回归\n",
    "\n",
    "上面的逻辑回归模型和线性回归模型很不一样，没办法再用最小二乘来进行参数估计了。在最小二乘中，我们有每个样本的实际值和它对应的预测值，我们可以计算出两者的平方损失，然后累加起来，最小二乘的目标是使得累加起来的损失最小。而在这里，我们有每个样本的实际值和它对应的预测值的概率，如果我们的模型能让每个样本都正确分类，这样的模型损失是最小的，要实现这一点，也就是说我们要找一个模型让每个样本属于其真实分类的概率越大越好，这种方法被称为 **极大似然法**（maximum likelihood method）。\n",
    "\n",
    "$$\n",
    "\\mathop{\\arg \\max}_{w,b} \\prod_{i=1}^n p(y_i | x_i; w,b)\n",
    "$$\n",
    "\n",
    "但是这里的连乘结果很可能导致下溢，通常使用**对数似然**（log-likelihood）来代替，这样就可以将连乘转换成连加：\n",
    "\n",
    "$$\n",
    "\\mathop{\\arg \\max}_{w,b} \\sum_{i=1}^n \\ln p(y_i | x_i; w,b)\n",
    "$$\n",
    "\n",
    "其中 $p(y | x)$ 包含了 $y = 0$ 和 $y = 1$ 两种情况，这里有一个小技巧可以把两种情况写成一个表达式：\n",
    "\n",
    "$$\n",
    "p(y|x) = p(y=1|x)^y p(y=0|x)^{(1-y)}\n",
    "$$\n",
    "\n",
    "对其求对数有：\n",
    "\n",
    "$$\n",
    "\\ln p(y|x) = y \\ln p(y=1|x) + (1-y) \\ln p(y=0|x)\n",
    "$$\n",
    "\n",
    "令：$p(y=1|x) = \\pi(x)$，那么有 $p(y=0|x) = 1 - \\pi(x)$，所以有：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\ln p(y|x) &= y \\ln \\pi(x) + (1-y) \\ln (1-\\pi(x)) \\\\\n",
    "&= y \\ln \\pi(x) + \\ln (1-\\pi(x)) - y\\ln(1-\\pi(x)) \\\\\n",
    "&= y \\ln \\frac{\\pi(x)}{1-\\pi(x)} + \\ln (1-\\pi(x)) \\\\\n",
    "&= y \\ln e^{wx+b} + \\ln \\frac{1}{1+e^{wx+b}} \\\\\n",
    "&= y(wx+b) - \\ln (1+e^{wx+b})\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "于是我们得到了逻辑回归的损失函数：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "loss \n",
    "&= \\sum_{i=1}^n \\ln p(y_i|x_i) \\\\\n",
    "&= \\sum_{i=1}^n y_i \\ln \\pi(x_i) + (1-y_i) \\ln (1-\\pi(x_i)) \\\\\n",
    "&= \\sum_{i=1}^n [ y_i(wx_i+b) - \\ln (1+e^{wx_i+b}) ] \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "我们要求的就是上式最大值时的 w 和 b 参数，一般情况下，我们都是求损失函数的最小值，所以我们在上式中加一个负号。\n",
    "\n",
    "$$\n",
    "\\mathop{\\arg \\min}_{w,b} \\sum_{i=1}^n [ -y_i(wx_i+b) + \\ln (1+e^{wx_i+b}) ]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 梯度下降法\n",
    "\n",
    "逻辑回归的损失函数不能像线性回归那样求解析解，一般我们通过梯度下降法或牛顿法等优化算法来求解。这里我们使用梯度下降法，首先我们需要求出上面的损失函数的梯度，也就是分别对 w 和 b 求偏导。\n",
    "\n",
    "这里要运用到微分法则中很重要的一条：**链式法则**，即：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial y}{\\partial x} = \\frac{\\partial y}{\\partial z} \\frac{\\partial z}{\\partial x}\n",
    "$$\n",
    "\n",
    "我们令：\n",
    "\n",
    "$$\n",
    "z = \\ln p = \\ln (1+e^{wx_i+b})\n",
    "$$\n",
    "\n",
    "则：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial w} = \\frac{\\partial z}{\\partial p} \\frac{\\partial p}{\\partial w} = \\frac{1}{p} e^{wx_i+b} x_i = \\frac{e^{wx_i+b}}{1+e^{wx_i+b}}x_i\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial b} = \\frac{\\partial z}{\\partial p} \\frac{\\partial p}{\\partial b} = \\frac{1}{p} e^{wx_i+b} = \\frac{e^{wx_i+b}}{1+e^{wx_i+b}}\n",
    "$$\n",
    "\n",
    "再令：\n",
    "\n",
    "$$\n",
    "z = -y_i(wx_i+b)\n",
    "$$\n",
    "\n",
    "则：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial w} = -y_ix_i\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial z}{\\partial b} = -y_i\n",
    "$$\n",
    "\n",
    "于是有：\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial loss}{\\partial w} &= \\sum_{i=1}^n (\\frac{e^{wx_i+b}}{1+e^{wx_i+b}}x_i -y_ix_i) \\\\\n",
    "&= \\sum_{i=1}^n (\\frac{e^{wx_i+b}}{1+e^{wx_i+b}} -y_i)x_i \\\\\n",
    "&= \\sum_{i=1}^n (\\pi(x_i) -y_i)x_i\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial loss}{\\partial b} &= \\sum_{i=1}^n (\\frac{e^{wx_i+b}}{1+e^{wx_i+b}} -y_i) \\\\\n",
    "&= \\sum_{i=1}^n (\\pi(x_i) -y_i)\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "最后得到参数 w 和 b 的更新公式：\n",
    "\n",
    "$$\n",
    "w := w - \\eta \\frac{\\partial loss}{\\partial w} = w + \\eta \\sum_{i=1}^n (y_i - \\pi(x_i))x_i\n",
    "$$\n",
    "\n",
    "$$\n",
    "b := b - \\eta \\frac{\\partial loss}{\\partial b} = b + \\eta \\sum_{i=1}^n (y_i - \\pi(x_i))\n",
    "$$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}