{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在前面的学习中，我们知道逻辑回归问题可以转换为求解似然函数的极大值：\n",
    "\n",
    "$$\n",
    "\\mathop{\\arg \\max}_{\\theta} \\prod_{i=1}^n p(y_i | x_i; \\theta)\n",
    "$$\n",
    "\n",
    "再通过一系列的转换，最终得到逻辑回归的损失函数：\n",
    "\n",
    "$$\n",
    "loss = \\sum_{i=1}^n [ -y_i(\\theta^Tx_i) + \\ln (1+e^{\\theta^Tx_i}) ]\n",
    "$$\n",
    "\n",
    "为了让损失函数最小，然后使用了梯度下降算法对参数 $\\theta$ 不断迭代，最终得出参数 $\\theta$。\n",
    "\n",
    "它的梯度如下：\n",
    "\n",
    "$$\n",
    "\\frac{\\partial loss}{\\partial \\theta} = \\sum_{i=1}^n (\\pi(x_i) -y_i)x_i\n",
    "$$\n",
    "\n",
    "其中，\n",
    "\n",
    "$$\n",
    "\\pi(x) = p(y=1|x) = \\frac{e^{\\theta^Tx}}{1+e^{\\theta^Tx}}\n",
    "$$\n",
    "\n",
    "最后得出梯度下降的迭代公式如下：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j + \\eta \\sum_{i=1}^n (y^{(i)} - \\pi(x^{(i)}))x_j^{(i)}\n",
    "$$\n",
    "\n",
    "梯度下降还可以进一步衍生出随机梯度下降法，都可以很好的求解逻辑回归问题，这一节将介绍一种新的方法，**牛顿法**（Newton method），它可以利用曲线本身的信息，比梯度下降法更容易收敛，迭代次数更少。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 牛顿法解一元函数极值\n",
    "\n",
    "## 牛顿法解多元函数极值\n",
    "\n",
    "## 牛顿法解逻辑回归\n",
    "\n",
    "牛顿法迭代公式为：\n",
    "\n",
    "$$\n",
    "\\theta := \\theta - \\frac{loss'(\\theta)}{loss''(\\theta)}\n",
    "$$\n",
    "\n",
    "其中，$loss'(\\theta)$ 为损失函数关于 $\\theta$ 的一阶导数：\n",
    "\n",
    "$$\n",
    "loss'(\\theta) = \\frac{\\partial loss}{\\partial \\theta} = \\sum_{i=1}^n (\\pi(x_i) -y_i)x_i\n",
    "$$\n",
    "\n",
    "$loss''(\\theta)$ 为损失函数关于 $\\theta$ 的二阶导数：\n",
    "\n",
    "$$\n",
    "loss''(\\theta) = \\frac{\\partial^2 loss}{\\partial \\theta \\partial \\theta^T} = \\sum_{i=1}^n x_ix_i^T \\pi(x_i)(1-\\pi(x_i))\n",
    "$$\n",
    "\n",
    "https://blog.csdn.net/baimafujinji/article/details/51179381"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}