{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### sklearn 中的梯度下降法\n",
    "\n",
    "上面的求解过程中，我们每次的迭代都用到了所有样本，这样做可以让收敛速度最快，但是如果样本数非常多，计算性能就会变低，上面的例子中一共也就3个样本，所以不明显。像这种每次迭代都使用全部样本的方法，叫做 **批量梯度下降**（Batch gradient descend，BGD）。为了提高计算性能，可以在每次迭代中随机选取一个样本（或部分样本）用于计算，这样也可以保证收敛，虽然收敛的速度慢了点，但计算快很多，而且可以有效的避免陷入局部极小值情况。这种方法叫做 **随机梯度下降**（Stochastic gradient descend，SGD）。\n",
    "\n",
    "sklearn 中的 `SGDRegressor` 实现的就是随机梯度下降算法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a = 1.5356587127294614, b = 1.0052652195613447\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcjfX///HHyzJEiT6WZAmtSkqNalSfJlSUpT6t2msW+1qRVEppI1IhY2lP9UU1Y2cYUhMhkr0Uxr7LOtv794epm5/ENOecuc7yvP9jznUu53qem5mna65zXdfLnHOIiEjoK+J1ABER8Q8VuohImFChi4iECRW6iEiYUKGLiIQJFbqISJhQoYuIhAkVuohImFChi4iEiWKFubHy5cu7GjVqFOYmRURC3oIFC7Y75yqcbL1CLfQaNWowf/78wtykiEjIM7O1+VlPh1xERMKECl1EJEyo0EVEwoQKXUQkTKjQRUTChApdRCRMqNBFRMKECl1EJIB2HNhB50md2XNoT8C3pUIXEQkA5xyf/fwZtQfXZsj8IcxeOzvg2zxpoZvZKDPbamY/H7O8o5mtNLOlZvZ64CKKiISWjL0ZtPisBa3GtuLssmezIHEBzS9oHvDt5ufS//eBd4AP/1xgZjcALYG6zrnDZlYxMPFEREJHrstl2Pxh9Jjeg+zcbPrf2J/OV3emWJHCucvKSbfinJttZjWOWdwWeNU5dzhvna3+jyYiEjpWbl9JQkoC36z7hoY1G5LULIlzzjinUDMU9Bj6+cB1ZjbXzGaZWX1/hhIRCRVZOVm8/M3LXPrupSzZuoSRLUYy/cHphV7mUPC7LRYDygFXA/WBL8yslnPOHbuimSUCiQDVq1cvaE4RkaCzYOMC4pLjWLxlMXfUvoO3m75N5dMqe5anoHvoGcA4d8Q8IBcof7wVnXNJzrlo51x0hQonvZ2viEjQO5B1gO7TunPViKvYsn8L4+4ex5i7x3ha5lDwPfSvgIZAmpmdD0QB2/2WSkQkSM38bSYJKQn8uutX4uvF0++mfpQtWdbrWEA+Ct3MRgOxQHkzywB6A6OAUXmnMmYCDx/vcIuISLjYfWg33ad1Z/jC4dQqV4vUh1JpWLOh17H+P/k5y6XVPzz1gJ+ziIgEpa9WfEW7Ce3Ysn8LT8Q8wQs3vECp4qW8jvU3hTqCTkQklGzet5mOkzoyZtkY6laqS3KrZKLPivY61j9SoYuIHMM5x/uL3ufxqY+zP2s/fRv25ckGT1K8aHGvo52QCl1E5Chrdq2h9fjWTF8znWurX8vw5sO5sPyFXsfKFxW6iAiQk5vDW3Pf4pmZz1DEijD4lsG0iW5DEQudexiq0EUk4i3ZsoT4lHjmbZjHrefdytBbh1Lt9Gpex/rXVOgiErEOZx+m7zd9eWXOK5QtWZZP//cp99a5FzPzOlqBqNBFJCJ9t/474pPjWb59OQ/UfYCBNw+kfKnjXvAeMlToIhJR/jj8B0+nPs3gHwZTtUxVJt43kabnNfU6ll+o0EUkYkxaPYk2E9qwfs962tdvz8uNXua0Eqd5HctvVOgiEva2H9hO1yld+finj7mw/IXMeWwODao18DqW36nQRSRs/TnXs9PkTuw+tJtn//ssva7rRYliJbyOFhAqdBEJS+v3rKfdxHaMXzWe+mfVZ2SLkVxS6RKvYwWUCl1Ewsqxcz0H3DSATld1omiRol5HCzgVuoiEjaPnejau1ZhhzYZRq1wtr2MVGhW6iIS8rJws+n3Xjz6z+nBK8VMY1WIUj1z2SMheIFRQKnQRCWnzN84nPjmexVsWc+dFd/J207c589QzvY7lCRW6iISkA1kH6D2zNwO+H0Cl0pUYd/c4bq99u9exPHXS24iZ2Sgz25o3bu7Y554wM2dmoX29rIiElBm/zaDu0Lr0T+/PY5c9xrL2yyK+zCEfhQ68DzQ5dqGZVQNuBNb5OZOIyHHtPrSbhOQEGn3YCIAZD81geIvhQTOk2Wv5mSk628xqHOepgUB34Gs/ZxIR+Zsvl39J+4nt2bJ/C082eJLnY58PyrmeXirQMXQzawFscM4tPtmnyGaWCCQCVK9evSCbE5EItnnfZjpM7MDY5WO5tNKlpLRK4YqzrvA6VlD614VuZqWAXsBN+VnfOZcEJAFER0e7f7s9EYlMzjneW/Qej099nINZB3m54cs80eCJoJ/r6aWC7KGfA9QE/tw7rwosNLMrnXOb/RlORCLTml1rSExJJPW3VK6rfh3Dmw/ngvIXeB0r6P3rQnfOLQEq/vnYzH4Hop1z2/2YS0QiUE5uDoPmDuKZGc9QrEgxht46lMQrEkNqrqeXTlroZjYaiAXKm1kG0Ns5NzLQwUQksvy05Sfik+P5YeMPNDu/GUNuGRKScz29lJ+zXFqd5PkafksjIhHncPZhXpr9Eq9++yrlSpZj9B2juefieyLusn1/0JWiIuKZo+d6Plj3QQbcPCDk53p6SYUuIoXu6Lme1U6vxqT7J9Hk3L9dvyj/kgpdRArVpNWTaD2+NRl7M+hwZQf6NuwbVnM9vaRCF5FCsf3AdrpM7sInSz6hdvnafPvYt8RUi/E6VlhRoYtIQDnnGP3zaDpP7syeQ3t47r/P8fR1T4ftXE8vqdBFJGDW71lPmwltmLh6IldWuZKRLUZSp2Idr2OFLRW6iPhdrstl6A9DeSr1KXJdLgNvHkjHKztGxFxPL6nQRcSvVmxfQUJKAnPWzaFxrcYkNUuiZrmaXseKCCp0EfGLrJwsXv/2dfrM7kPp4qV5r+V7PHzpw7pAqBCp0EXEZ/M3zicuOY6ftvzEXRfdxVtN34rYuZ5eUqGLSIEdyDrAczOfY+D3Aznz1DP56p6vaHlhS69jRSwVuogUyIzfZpCQknDkVreXJ/Laja9pFJzHVOgi8q/sOriLJ6c9ycgfR3LuGecy8+GZxNaI9TqWoEIXkX9h3PJxtJ/Ynm37t9G9QXeej32eU4qf4nUsyaNCF5GT2vTHJjpM6sC45eO47MzLmHDfBC6vfLnXseQYKnQR+Uea6xla8jOxaBTQDNjqnKuTt6wf0BzIBH4FHnXO7Q5kUBEpXJrrGXryM6jvfeDYGxVPA+o45+oCq4Cefs4lIh7Jzs3mje/eoM6QOszbMI+htw4l7ZE0lXkIyM8IutlmVuOYZVOPevg9cKd/Y4mIF37a8hNxyXHM3zif5uc3Z8itQ6hapqrXsSSf/HEM/THgcz+8joh45FD2IV6a/RKvffsa5UqW47M7PuPui+/WZfshxqdCN7NeQDbwyQnWSQQSAapXr+7L5kQkAOasm0NCSgIrtq/goUsfYsBNA/hPqf94HUsKID/H0I/LzB7myIel9zvn3D+t55xLcs5FO+eiK1SoUNDNiYif7T28l/YT2nPde9dxMOsgk++fzAe3faAyD2EF2kM3syZAD+B659wB/0YSkUCbuHoibca3IWNvBp2u7ETfRn05NepUr2OJj/Jz2uJoIBYob2YZQG+OnNVSApiWd4zte+dcmwDmFBE/2LZ/G12mdOHTJZ9yUYWLNNczzOTnLJdWx1k8MgBZRCRAnHN8uuRTukzpwp5De+h9fW96XttTcz3DjK4UFQlz6/aso+2EtkxcPZGrqlzFiBYjNNczTKnQRcKU5npGHhW6SBhasX0F8cnxfLv+W26sdSPDmg3TXM8IoEIXCSOZOZm8/u3rvDj7RUoXL837Ld/noUsf0gVCEUKFLhImftjwA3HJcSzZuoS7L76bt5q8RaVTK3kdSwqRCl0kxB071/Pre7+mxQUtvI4lHlChi4Sw1DWpJI5PZM2uNbS+ojWvNX6N00ue7nUs8YgKXSQE7Tq4iyemPsGoRaM494xzSXs4jetrXO91LPGYCl0kxIxdNpYOkzqwbf82elzTg97X99ZcTwFU6CIhY9Mfm2g/sT1frviSemfW01xP+RsVukiQc84x8seRPDH1CQ7nHObVRq/SLaab5nrK36jQRYLYLzt/ITElkZm/z+T6s69nePPhnPef87yOJUFKhS4ShLJzs3nz+zd5buZzFC9anGHNhhF/eTxFrMAjDCQCqNBFgszizYuJS45jwaYFtLigBUNuGUKVMlW8jiUhQIUuEiQOZR/ixVkv8vp3r3PGKWfw+Z2fc9dFd+myfck3FbpIEJizbg7xyfGs3LFScz2lwFToIh7ae3gvPaf3ZMj8IZx9+tlMvn8yN597s9exJETlZwTdKI4Mg97qnKuTt+wM4HOgBvA7cLdzblfgYoqEnwmrJtBmQhs27N1A56s681LDlzTXU3ySn4/M3weaHLPsKSDVOXcekJr3WETyYdv+bdw39j6ajW7G6SVO57u473izyZsqc/FZfmaKzjazGscsbsmRwdEAHwBpQA8/5hIJO845PlnyCV0md2Hv4b08f/3z9LyuJ1FFo7yOJmGioMfQKznnNgE45zaZWUU/ZhIJO+v2rKPN+DZM+mUSV1e9mhHNR3BxxYu9jiVhJuAfippZIpAIUL169UBvTiSo5LpchvwwhJ6pPXHOMajJINrXb6+5nhIQBS30LWZWOW/vvDKw9Z9WdM4lAUkA0dHRroDbEwk5y7ctJz4lnu/Wf8fN59zMu83epUbZGl7HkjBW0OuIk4GH875+GPjaP3FEQl9mTiYvznqRy4ZdxortK/jwtg+ZdP8klbkEXH5OWxzNkQ9Ay5tZBtAbeBX4wszigHXAXYEMKRIq5m2YR3xyPEu2LuGei+9hUJNBmusphSY/Z7m0+oenGvk5i0jI2p+5n2dnPsuguYOofGplzfUUT+hKUREfTV8zncSURH7b/RttrmjDq41f1VxP8YQKXaSAdh3cxeNTH+e9Re9x3hnnaa6neE6FLlIAY5eNpf3E9mw/sJ2nrnmK565/TnM9xXMqdJF/YeMfG+kwsQNfrviSyytfzqT7J1Gvcj2vY4kAKnSRfDl2rudrjV+jW0w3ihXRj5AED303ipyE5npKqFChi/yD7NxsBqYP5Lm054gqGqW5nhL0VOgix7Fo8yLik+NZsGkBLS9oyeBbBmuupwQ9FbrIUQ5lH6LPrD68/u3r/KfUf/jizi+486I7NddTQoIKXSTPN2u/IT4lnlU7VvHIZY/wxk1vcMYpZ3gdSyTfVOgS8fYe3stT059i6Pyh1Chbg6kPTOXGc270OpbIv6ZCl4g2ftV42k5oy4a9G+h6dVdevOFFSkeV9jqWSIGo0CUibd2/lc6TO/PZz59Rp2Idxtw1hquqXuV1LBGfqNAlojjn+Pinj+k6pSt7D+/lhdgXeOrapzTXU8KCCl0ixtrda2kzoQ2Tf5lMTNUYRrQYwUUVLvI6lojfqNAl7OXk5vw11xPQXE8JWz4Vupl1BeIBBywBHnXOHfJHMBF/WLZtGfHJ8aRnpHPzOTczrNkwzi57ttexRAKiwNcwm1kVoBMQ7ZyrAxQF7vVXMBFfZOZk0mdWH+oNq8fKHSv/muupMpdw5ushl2LAKWaWBZQCNvoeScQ38zbMIy45jp+3/sy9de5lUJNBVCxd0etYIgFX4D1059wGoD9HhkRvAvY456b6K5jIv7U/cz/dpnQjZmQMuw7uIqVVCqPvGK0yl4jhyyGXckBLoCZwFlDazB44znqJZjbfzOZv27at4ElFTmD6mulcMvQSBn4/kNZXtGZZ+2U0O7+Z17FECpUv9wFtDPzmnNvmnMsCxgENjl3JOZfknIt2zkVXqFDBh82J/N3Ogzt59OtHufGjGyletDizH5nNkFuHUKZEGa+jiRQ6X46hrwOuNrNSwEGgETDfL6lETsI5x5hlY+g4qSPbD2yn57U9ee765yhZrKTX0UQ8U+BCd87NNbMxwEIgG/gRSPJXMJF/svGPjbSb0I6vV37N5ZUvZ/IDk7nszMu8jiXiOZ/OcnHO9QZ6+ymLyAnlulxGLBzBk9OeJDMnk9cbv07XmK6a6ymSRz8JEhJ+2fkLCSkJpP2eRmyNWIY3H865Z5zrdSyRoKJCl6CWnZvNgPQB9E7rTYmiJUhqlkT85fGaICRyHCp0CVqLNi8iLjmOhZsWctuFtzH4lsGcddpZXscSCVoqdAk6R8/1LF+qPP931/9xR+07tFcuchIqdAkqs9fOJiElgVU7VvHoZY/S/6b+muspkk8qdAkKew/vpce0Hry74F1qlq3JtAen0bhWY69jiYQUFbp4LmVlCm0ntGXTvk10u7obfW7oo7meIgWgQhfPbN2/lU6TOvH50s+pU7EO4+4Zx5VVrvQ6lkjIUqFLoftzrmeXKV3Yl7mPPrF96HFtD831FPGRCl0K1drda2k9vjVTfp1Cg2oNGN58uOZ6iviJCl0KRU5uDoN/GMzTqU9jZrzd9G3a1W9HEfPlhp8icjQVugTcsm3LiEuO4/uM72l6blPebfYu1U+v7nUskbCjQpeAyczJ5JVvXqHvN30pU6IMH93+Efdfcr8uEBIJEBW6BMTcjLnEJcexdNtSWtVpxaAmg6hQWgNORAJJhS5+tT9zP8/MeIZBcwdRpUwVxrcaz63n3+p1LJGIoEIXv5n26zQSxyfy++7faRvdllcbv6pRcCKFSIUuPtt5cCfdpnTjg8UfcP5/zmf2I7O57uzrvI4lEnF8KnQzKwuMAOoADnjMOZfuj2AS/P6c69lhUgd2HtzJ09c+zbPXP6u5niIe8XUPfRAw2Tl3p5lFAaX8kElCwIa9G2g/sT1fr/yaKypfwdQHpnLpmZd6HUskohW40M2sDPBf4BEA51wmkOmfWBKsjp7rmZWTRb8b+9Hl6i6a6ykSBHz5KawFbAPeM7NLgQVAZ+fcfr8kk6CzesdqElISmLV2FjfUuIHhzYdzzhnneB1LRPL4ct11MeByYKhzrh6wH3jq2JXMLNHM5pvZ/G3btvmwOfFKdm42r815jbrv1mXR5kWMaD6C1IdSVeYiQcaXPfQMIMM5Nzfv8RiOU+jOuSQgCSA6Otr5sD3xwI+bfiQuOY4fN//I/2r/j3eavkPl0yp7HUtEjqPAe+jOuc3AejO7IG9RI2CZX1KJ5w5mHaTn9J7UH16fjX9sZMxdYxh791iVuUgQ8/WTrI7AJ3lnuKwBHvU9knht9trZxCfHs3rnah677DH639SfcqeU8zqWiJyET4XunFsERPspi3hsz6E99Jjeg2ELhlGrXC2mPzidRrUaeR1LRPJJ55oJAMkrk2k3oZ3meoqEMBV6hNuybwudJnfii6VfcEnFS/jyni+pX6W+17FEpABU6BHKOcdHP31E1yld2Ze5jxdveJHu13TXXE+REKZCj0C/7/6d1uNbM/XXqVxT7RqGNx9O7Qq1vY4lIj5SoQep9PR00tLSiI2NJSYmxi+vmZObwzvz3qHXjF6YGe80fYe29dtqrqdImFChB6H09HQaNWpEZmYmUVFRpKam+lzqS7cuJT4lXnM9RcKYds2CUFpaGpmZmeTk5JCZmUlaWlqBXyszJ5MX0l6g3rB6rN6xmo9v/5gJ901QmYuEIe2hB6HY2FiioqL+2kOPjY0t0Ot8n/E98cnxLN22lPsuuY83b35Tcz1FwpgKPQjFxMSQmppa4GPo+zL38cyMZ3hr7ltULVNVcz1FIoQKPUjFxMQU6Lj51F+nkpiSyNo9a2lfvz2vNHqF00qcFoCEIhJsVOhhYseBHXSb2o0PF3/IheUvZM6jc7im+jVexxKRQqRCD3HOOb5Y+gWdJndi58Gd9LquF8/89xnN9RSJQCr0ELZh7wbaTWxH8spkos+KZtqD06hbqa7XsUTEIyr0EJTrchm+YDjdp3cnKyeL/jf2p/PVnTXXUyTCqQFCzKodq0hMSWTW2lk0rNmQpGZJGgUnIoAKPWRk5WTxRvobPJ/2PCWLlWRE8xE8Vu8xzMzraCISJFToIWDhpoXEJ8drrqeInJDPhW5mRYH5wAbnXDPfI8mfDmYd5IVZL9D/u/5UKF2BsXeP5X+1/+d1LBEJUv7YQ+8MLAfK+OG1JM+s32eRkJLA6p2riasXR78b+2mup4ickE835zKzqsCtwAj/xJE9h/bQOqU1sR/EkuNySH0olREtRqjMReSkfN1DfxPoDvzjteVmlggkAlSvrjv8nUjyymTaTmjL5n2beSLmCV644QVKFS/ldSwRCREF3kM3s2bAVufcghOt55xLcs5FO+eiK1TQnf6OZ8u+Ldwz5h5aftaS8qXKMzd+Lv1u6qcyF5F/xZc99GuAFmZ2C1ASKGNmHzvnHvBPtPDnnOPDxR/SdUpX9mft56UbXqL7Nd0pXrS419FEJAQVuNCdcz2BngBmFgs8oTLPv2Pneo5oMYILy1/odSwRCWE6D72Q5eTm8Pa8t+k1oxdFrAiDbxlMm+g2muspIj7zS6E759KANH+8Vjj7eevPxCfHM3fDXG457xaG3jpUo+BExG+0h14IDmcf5uVvXuaVOa9wesnT+eR/n9CqTitdti8ifqVCD7DvM74nLjmOZduWcf8l9/NmkzcpX6q817FEJAyp0ANkX+Y+eqX24u15b1O1TFUm3jeRpuc19TqWiIQxFXoATPllCq3Ht2bdnnW0r9+elxu9rLmeIhJwKvSTSE9PJy0tjdjY2JMObd5xYAddp3Tlo58+OjLX87E5NKjWoJCSikikU6GfQHp6Oo0aNSIzM5OoqChSU1OPW+p/zvXsOKkjuw7t4tn/Pkuv63pRolgJD1KLSKRSoZ9AWloamZmZ5OTkkJmZSVpa2t8KPWNvBu0mtCNlVQr1z6rP9BbTNddTRDyhQj+B2NhYoqKi/tpDj42N/eu5XJdL0oIkuk/rTnZuNm/c9Aadr+pM0SJFvQssIhFNhX4CMTExpKam/u0Y+qodq0hISWD22tk0qtmIpOZJ1CpXy+O0IhLpVOgnERMT81eRHz3X85TipzCqxSgeuewRXSAkIkFBhZ5PCzctJC45jkWbF3HnRXfydtO3OfPUM72OJSLyFxX6SRzMOsjzac/zRvobVCxdkXF3j+P22rd7HUtE5G9U6CeQ9nsaCSkJ/LLzF+LrxdPvpn6ULVnW61giIselQj+O3Yd2031ad4YvHM455c5hxkMzuKHmDV7HEhE5IRX6Mb5a8RXtJrRjy/4tmuspIiFFhZ5n877NdJzUkTHLxlC3Ul2SWyUTfVa017FERPKtwIVuZtWAD4EzgVwgyTk3yF/BCotzjg8Wf0C3Kd04kHWAvg378mSDJzXXU0RCji976NnA4865hWZ2GrDAzKY555b5KVvA/bbrN1qPb820NdO4tvq1jGg+ggvKX+B1LBGRAvFlSPQmYFPe13+Y2XKgChD0hZ6Tm8Nbc9/imZnPUNSKMuSWIbSObq25niIS0vxyDN3MagD1gLn+eL1AOnquZ7PzmzHkliFUO72a17FERHzmc6Gb2anAWKCLc27vcZ5PBBIBqlf3biDy0XM9y5Ysy+g7RnPPxffosn0RCRs+FbqZFedImX/inBt3vHWcc0lAEkB0dLTzZXsFlb4+nbjkOJZvX84DdR9g4M0DNddTRMKOL2e5GDASWO6cG+C/SP6zL3MfT6c+zTvz3qHa6dU011NEwpove+jXAA8CS8xsUd6yp51zE32P5bvJv0ym9fjWrN+zng5XdqBvw76a6ykiYc2Xs1zmAEF3AProuZ61y9fWXE8RiRhhc6Woc47Pl35Op0mdNNdTRCJSWBR6xt4M2k5oy/hV47myypWkNk/lkkqXeB1LRKRQhXSh57pchs0fRo/pPchxOQy4aQCdruqkuZ4iEpFCttBXbl9JQkoC36z7hsa1GpPULIma5Wp6HUtExDMhV+hZOVn0+64ffWb1oVTxUrzX8j0evvRhXSAkIhEvpAp9wcYFxCXHsXjLYu666C7eavqW5nqKiOQJiUI/kHXgr7melUpX4st7vuS2C2/zOpaISFAJiUK//b3bmbppKi2qtOCDBz7QXE8RkeMI+kJPT09nVp9ZFClRhGmbprH84uXExMR4HUtEJOgE/Q3A09LSyN6cTe6aXDIzM0lLS/M6kohIUAr6Qo+NjSUqKoqiRYsSFRVFbGys15FERIJS0B9yiYmJITU1lbS0NGJjY3W4RUTkHwR9ocORUleRi4icWNAfchERkfxRoYuIhAkVuohImFChi4iECRW6iEiYUKGLiIQJc84V3sbMtgFrC/jXywPb/RgnFOg9Rwa958jgy3s+2zlX4WQrFWqh+8LM5jvnor3OUZj0niOD3nNkKIz3rEMuIiJhQoUuIhImQqnQk7wO4AG958ig9xwZAv6eQ+YYuoiInFgo7aGLiMgJhEShm1kTM1tpZr+Y2VNe5wk0M6tmZjPNbLmZLTWzzl5nKgxmVtTMfjSz8V5nKQxmVtbMxpjZirx/67C/paiZdc37nv7ZzEabWUmvM/mbmY0ys61m9vNRy84ws2lmtjrvz3KB2HbQF7qZFQUGA02Bi4BWZnaRt6kCLht43DlXG7gaaB8B7xmgM7Dc6xCFaBAw2Tl3IXApYf7ezawK0AmIds7VAYoC93qbKiDeB5ocs+wpINU5dx6QmvfY74K+0IErgV+cc2ucc5nAZ0BLjzMFlHNuk3NuYd7Xf3DkB72Kt6kCy8yqArcCI7zOUhjMrAzwX2AkgHMu0zm329tUhaIYcIqZFQNKARs9zuN3zrnZwM5jFrcEPsj7+gPgtkBsOxQKvQqw/qjHGYR5uR3NzGoA9YC53iYJuDeB7kCu10EKSS1gG/Be3mGmEWZW2utQgeSc2wD0B9YBm4A9zrmp3qYqNJWcc5vgyA4bUDEQGwmFQrfjLIuIU3PM7FRgLNDFObfX6zyBYmbNgK3OuQVeZylExYDLgaHOuXrAfgL0a3iwyDtu3BKoCZwFlDazB7xNFV5CodAzgGpHPa5KGP6adiwzK86RMv/EOTfO6zwBdg3Qwsx+58ghtYZm9rG3kQIuA8hwzv35m9cYjhR8OGsM/Oac2+acywLGAQ08zlRYtphZZYC8P7cGYiOhUOg/AOeZWU0zi+LIhyjJHmcKKDMzjhxbXe6cG+B1nkBzzvV0zlV1ztXgyL/vDOdcWO+5Oec2A+vN7IK8RY2AZR5GKgzrgKvNrFTe93gjwvyD4KMkAw/nff0w8HUgNhL0Q6Kdc9lm1gGYwpFPxUc555Z6HCvsmFd0AAAAhElEQVTQrgEeBJaY2aK8ZU875yZ6mEn8ryPwSd6OyhrgUY/zBJRzbq6ZjQEWcuRMrh8JwytGzWw0EAuUN7MMoDfwKvCFmcVx5D+2uwKybV0pKiISHkLhkIuIiOSDCl1EJEyo0EVEwoQKXUQkTKjQRUTChApdRCRMqNBFRMKECl1EJEz8PxsF4IZekf4AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import SGDRegressor\n",
    "\n",
    "X = np.array([0,1,2])\n",
    "Y = np.array([0.9, 3.1, 5.1])\n",
    "#X = np.array([39.93, 42.05, 43.18])\n",
    "#Y = np.array([199,   290,   298])\n",
    "\n",
    "model = SGDRegressor(max_iter=1000, tol=0.01)\n",
    "model.fit(X.reshape(-1,1), Y.ravel())\n",
    "\n",
    "plt.plot(X, Y, 'k.')\n",
    "\n",
    "x = [[0],[10]]\n",
    "#x = [[30],[60]]\n",
    "y = model.predict(x)\n",
    "\n",
    "b = y[0]\n",
    "a = (y[1]-b)/x[1][0]\n",
    "print(\"a = {0}, b = {1}\".format(a, b))\n",
    "\n",
    "plt.plot(x, y, 'g-')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用批量梯度下降法 BGD 时，每次迭代是对所有样本进行计算，当目标函数为凸函数时，BGD 一定能够得到全局最优解，并且计算过程可以转换为矩阵计算，易于并行实现；不过当样本数目很多时，训练过程会很慢。使用随机梯度下降法 SGD 可以加快训练速度，不过缺点也显而易见，它的准确度要低很多，得到的可能不是全局最优解。为了在两种方法的性能之间取得一个折衷，可以采用 **小批量梯度下降法**（Mini-batch Gradient Descent，简称 MBGD），MBGD 在每次更新参数时使用 b 个样本（一般为10）。\n",
    "\n",
    "批量梯度下降的更新公式为：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j - \\alpha \\frac{1}{m} \\sum_{i=1}^{m} (h_{\\theta}(x^{(i)})-y^{(i)})x_j^{(i)}\n",
    "$$\n",
    "\n",
    "随机梯度下降的更新公式为（注意没有求和符号）：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j - \\alpha  (h_{\\theta}(x^{(i)})-y^{(i)})x^{(i)}_j\n",
    "$$\n",
    "\n",
    "小批量梯度下降的更新公式为：\n",
    "\n",
    "$$\n",
    "\\theta_j := \\theta_j - \\alpha \\frac{1}{10} \\sum_{k=i}^{(i+9)}(h_{\\theta}(x^{(k)})-y^{(k)})x_j^{(k)}\n",
    "$$\n",
    "\n",
    "### 参考\n",
    "\n",
    "* [\\[Machine Learning\\] 梯度下降法的三种形式BGD、SGD以及MBGD](https://www.cnblogs.com/maybe2030/p/5089753.html)\n",
    "* [批量梯度下降(BGD)、随机梯度下降(SGD)以及小批量梯度下降(MBGD)的理解 - LLLiuye - 博客园](https://www.cnblogs.com/lliuye/p/9451903.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
