{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第一部分　分类\n",
    "\n",
    "## 第1章　机器学习基础\n",
    "\n",
    "### 1.1 　何谓机器学习\n",
    "#### 1.1.1 　传感器和海量数据\n",
    "#### 1.1.2 　机器学习非常重要\n",
    "### 1.2 　关键术语\n",
    "### 1.3 　机器学习的主要任务\n",
    "### 1.4 　如何选择合适的算法\n",
    "### 1.5 　开发机器学习应用程序的步骤\n",
    "### 1.6 　Python语言的优势\n",
    "#### 1.6.1 　可执行伪代码\n",
    "#### 1.6.2 　Python比较流行\n",
    "#### 1.6.3 　Python语言的特色\n",
    "#### 1.6.4 　Python语言的缺点\n",
    "### 1.7 　NumPy函数库基础\n",
    "### 1.8 　本章小结\n",
    "\n",
    "## 第2章　k-近邻算法\n",
    "\n",
    "### 2.1 　k-近邻算法概述\n",
    "#### 2.1.1 　准备：使用Python导入数据\n",
    "#### 2.1.2 　从文本文件中解析数据\n",
    "#### 2.1.3 　如何测试分类器\n",
    "### 2.2 　示例：使用k-近邻算法改进约会网站的配对效果\n",
    "#### 2.2.1 　准备数据：从文本文件中解析数据\n",
    "#### 2.2.2 　分析数据：使用Matplotlib创建散点图\n",
    "#### 2.2.3 　准备数据：归一化数值\n",
    "#### 2.2.4 　测试算法：作为完整程序验证分类器\n",
    "#### 2.2.5 　使用算法：构建完整可用系统\n",
    "### 2.3 　示例：手写识别系统\n",
    "#### 2.3.1 　准备数据：将图像转换为测试向量\n",
    "#### 2.3.2 　测试算法：使用k-近邻算法识别手写数字\n",
    "### 2.4 　本章小结\n",
    "\n",
    "## 第3章　决策树\n",
    "\n",
    "### 3.1 　决策树的构造\n",
    "#### 3.1.1 　信息增益\n",
    "#### 3.1.2 　划分数据集\n",
    "#### 3.1.3 　递归构建决策树\n",
    "### 3.2 　在Python中使用Matplotlib注解绘制树形图\n",
    "#### 3.2.1 　Matplotlib注解\n",
    "#### 3.2.2 　构造注解树\n",
    "### 3.3 　测试和存储分类器\n",
    "#### 3.3.1 　测试算法：使用决策树执行分类\n",
    "#### 3.3.2 　使用算法：决策树的存储\n",
    "### 3.4 　示例：使用决策树预测隐形眼镜类型\n",
    "### 3.5 　本章小结\n",
    "\n",
    "## 第4章　基于概率论的分类方法：朴素贝叶斯\n",
    "\n",
    "### 4.1 　基于贝叶斯决策理论的分类方法\n",
    "### 4.2 　条件概率\n",
    "### 4.3 　使用条件概率来分类\n",
    "### 4.4 　使用朴素贝叶斯进行文档分类\n",
    "### 4.5 　使用Python进行文本分类\n",
    "#### 4.5.1 　准备数据：从文本中构建词向量\n",
    "#### 4.5.2 　训练算法：从词向量计算概率\n",
    "#### 4.5.3 　测试算法：根据现实情况修改分类器\n",
    "#### 4.5.4 　准备数据：文档词袋模型\n",
    "### 4.6 　示例：使用朴素贝叶斯过滤垃圾邮件\n",
    "#### 4.6.1 　准备数据：切分文本\n",
    "#### 4.6.2 　测试算法：使用朴素贝叶斯进行交叉验证\n",
    "### 4.7 　示例：使用朴素贝叶斯分类器从个人广告中获取区域倾向\n",
    "#### 4.7.1 　收集数据：导入RSS源\n",
    "#### 4.7.2 　分析数据：显示地域相关的用词\n",
    "### 4.8 　本章小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第5章　Logistic回归\n",
    "\n",
    "在这一节我们将首次接触最优化问题和算法。\n",
    "\n",
    "### 5.1 　基于Logistic回归和Sigmoid函数的分类\n",
    "\n",
    "对于二分类问题，通常使用 **海维塞德阶跃函数**（Heaviside step function）作为预测函数，也称为 **单位阶跃函数**，但是该函数在跳跃点上从 0 瞬间跳跃到 1，不是一个连续函数，所以使用 **Sigmoid 函数** 来近似：\n",
    "\n",
    "$$\n",
    "\\delta(z) = \\frac{1}{1 + e^{-z}}\n",
    "$$\n",
    "\n",
    "### 5.2 　基于最优化方法的最佳回归系数确定\n",
    "\n",
    "#### 5.2.1 　梯度上升法\n",
    "\n",
    "梯度算法的迭代公式：\n",
    "\n",
    "$$\n",
    "w := w + \\alpha \\nabla_w f(w)\n",
    "$$\n",
    "\n",
    "梯度上升法用来求函数的最大值，梯度下降法用来求函数的最小值。\n",
    "\n",
    "准备如下数据集，并通过梯度上升法求解 Logistic 回归："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAG3xJREFUeJzt3X+QXWWd5/H3J+nwQ+igY3oUksZYKwUySBjp6uxUVhcLZULGFafKqYJdVmp0TWIpq1VrRR0KmdWdrZ04OzO1MCOEgZVZI+4PB4aSoDDjTKGUNOkwCcEJaIZSOwlCI5KQAoUm3/2jbzOd2/dn33Pvec45n1dVV+4999x7v7dv5/me53m+5zmKCMzMrHqW5B2AmZnlwwnAzKyinADMzCrKCcDMrKKcAMzMKsoJwMysopwAzMwqygnAzKyinADMzCpqKO8AWlmxYkWsXr067zDMzApj165dz0TESCf7Jp0AVq9ezeTkZN5hmJkVhqQfd7qvh4DMzCrKCcDMrKKcAMzMKqrjBCDpVklPS3p03rbfl3RQ0u7az4Ymz10v6XFJ+yV9JovAzcysN930AL4MrG+w/U8i4oLaz476ByUtBf4MuBQ4F7hC0rmLCdbMzLLTcQKIiPuBZxfxHuPA/oh4IiJeAr4GXLaI1zEzswxlMQfwcUmP1IaIXtfg8ZXA1Lz7B2rbzMwsR70mgC8B/wK4AHgS+O8N9lGDbU2vQylpo6RJSZPT09M9hmdWfFOHp7h6x9WM3zzO1TuuZurwVPsnmXWgpxPBIuKpuduSbga+0WC3A8DovPurgEMtXnMbsA1gbGzMFyy2Sps6PMWaG9dw9KWjvHzsZXb/dDfb925nz+Y9jJ422v4FzFroqQcg6fR5d38beLTBbjuBsyS9WdIJwOXAXb28r1lVbH1g66uNP8DLx17m6EtH2frA1pwjszLouAcg6XbgImCFpAPAdcBFki5gdkjnR8Cm2r5nAH8RERsiYkbSx4FvAUuBWyPi+5l+CrOSmjg48WrjP+flYy/z0MGHcorIyqTjBBARVzTYfEuTfQ8BG+bd3wEsKBE1s9bWrlzL7p/uPi4JLFuyjPGV4zlGZWXhM4HNErZl3RZOPeFUli1ZBsw2/qeecCpb1m3JOTIrAycAs4SNnjbKns172HThJsbPGGfThZs8AWyZSXo5aLOymDo8xdYHtjJxcIK1K9eyZd2Wjhvx0dNGuX7D9X2O0KrICcCsz1zKaanyEJBZn7mU01LlBGDWZy7ltFQ5AZj1wfzlG34x8wuGdPxoayqlnF5moto8B2CWsfox/6ElQ7wSrzCkIWZiJplSTs9NmHsAZhmrH/OfOTbD0JIh3jry1qRKOT03Ye4BmGWs2Zj/yUMnM/GRiZyiWshzE+YegFnG1q5c++qZu3NSGfOfryhxWv84AZhlrCjLNxQlTusfJwCzjBVl+YZBxOkqo7QpIt1rroyNjcXk5GTeYZjZItRXGc31MFJMhmUiaVdEjHWyr3sAZtYXrjJKnxOAmfWFq4zS5wRgZn3hKqP0OQGYlVxeE7H9rDLy5HI2PAlsVmJ5T8TOXQfhOz/5DsfiGEu0hHec+Y6urofQ6DU9udxcXyaBJd0q6WlJj87b9kVJj0l6RNIdkl7b5Lk/krRX0m5JbtHNBiTvidjR00bZsm4LPzn8Ex575jH2PLWHm3bdxJob1yz6qD3vz1Qm3QwBfRlYX7ftPuC8iDgf+AHw2RbPf1dEXNBpZjKz4y1m2COFidisG+wUPlNZdJwAIuJ+4Nm6bfdGxEzt7oPAqgxjM7OauWGPm3bdxM5DOzs+ik5hIjbrBjuFz1QWWU4Cfwi4p8ljAdwraZekja1eRNJGSZOSJqenpzMMz6y4rv32tTz3i+e6PopOYbmHrBvsFD5TWWSSACRdA8wA25vssi4i3g5cCnxM0jubvVZEbIuIsYgYGxkZySI8GzBXaGRr6vAUX9n7FYLjCzY6OYpOYVmKrBvsFD5TWXRVBSRpNfCNiDhv3rargM3AxRHxQgev8fvA0Yj4o3b7ugqoeFyhkb2rd1zNn+/8c45x7LjtQnzw/A8yfOIwEwcnWLtybU/VNf00Vw300MGHGF85nmycZdBNFVBP1wOQtB74NPCvmzX+kk4BlkTE87XblwCf7+V9LV2tJvyu33B9ztEV08TBiQWNP8wmgDsfv5MXXn4h+St6jZ426u8/Qd2Ugd4OfA84W9IBSR8GbgCGgftqJZ431vY9Q9KO2lPfAHxX0h7gIeDuiPhmpp/CkuEKjew1GkNfwhLOev1Zrzb+4HJI617HPYCIuKLB5lua7HsI2FC7/QSwZlHRWeGsXbmW3T/dfVwSSLlCY25oIuUhlC3rtrB97/YFw2onDZ3kZGs98VIQlqkiVWgstrRy0JpNer7jzHe4HNJ64qUgLHNFmfC7esfV3LTrpgW9lU0XbirEeHVKE+71Pakrz7+SrzzylaR7VmXVzSSwE4AV1tThKa799rXcs/8eEFz6lkv5wru+0HFDM37zODsP7Vy4/YzxpC7e3koKybY+EQ1piFfiFZYuWcrMsRlXgg3YwKqAzPIydXiKt33pbRz+5eFXt9225zbufOxO9n50b0cNTdHmKxpJobqmvvJrprY4wMyx2X9dCZYuzwFYUjo9iWzrA1s58ssjC7Y//8vnO66CKdJ8RcoaVX7V8+R0mtwDsGTUDyW0qmufODix4MxYgGMc67ihmZtczXsIpega9aTqFa1nVRXuAVgyulk1cu3KtQgt2L6EJV01NHNDKBMfmeD6Dde78V+E+p7UkIYQYmjJ7PGle1bpcgKwZHRzEtmWdVtYfuLyBduHTxyuREOT0npL9WWqm8c2870Pf4/NF272Wj2J8xCQJaObSdnR00bZ+9G9LauAUj7Jq5fY6ofK/uHJf+Dmh2/mnBXn9Hy1rcVqNBm9dtXagcZg3XMZqCUjy7r2lGrks46t0fkLc1L6nJaPvlwS0qzfslzmN+XLBvYaW6uqm5Q+p6XPQ0CWlKzq2vuxKF1WQ0q9xtau6qbda6U8NGaD5QRgpZT1SV7dlKj2O7b6xeHqtXqtLD+HFZ+HgCwJWVe1ZH2SV5ZDSr3GNn+obM0b1nDi0hM7LrlMeWjMBs89AMtdP45Ksz7Jq5dhm0ZDLr3GNn+orJv1gHy9BpvPCcBy16+riLWaT+h2HPzckXPZdWjXcVfm6mTYplVyy2pdnG7mTcqw/pFlx0NAlrtBH5V2ex2AqcNT3PnYnQsuy/iaZa9pO2yT2pBLWdY/SulEuCJzArDcNbrkYT+PSrttlLc+sJUXXj7+ktdCvP/s97cdtkltyCXLUtu8FOVCPkXQVQKQdKukpyU9Om/br0i6T9IPa/++rslzr6rt80NJV/UauJXHoI9Ku22UG+0fBPue2df2vQad3DpR9PWPUutVFVm3PYAvA+vrtn0G+NuIOAv429r940j6FeA6YC0wDlzXLFFY9Qz6qLTbRrmXRrwsQy4pSa1XVWRdJYCIuB94tm7zZcBttdu3Ae9v8NTfBO6LiGcj4ufAfSxMJFZhgzwq7bZR7qURH1Ryq9KYeIq9qqLqei0gSauBb0TEebX7z0XEa+c9/vOIeF3dcz4FnBQR/6V2/1rgxYj4o1bv5bWArF+6vZRiCpdebCbldY/6oWqft1spXhJy4cLtNLiaByBpI7AR4Mwzz+xnTFZh3S45MYhLLy52iYZ+ldGmyhfyyU4WCeApSadHxJOSTgeebrDPAeCiefdXAX/f6MUiYhuwDWZ7ABnEZ5a8Xk6GS2FMvJPkleUaRClcC7kMskgAdwFXAf+t9u9fN9jnW8B/nTfxewnw2Qze2zLmhcLy0ewo/re++lucNHRSy++i2eJwL868yNThqb5/f50kL69BlKZuy0BvB74HnC3pgKQPM9vwv0fSD4H31O4jaUzSXwBExLPAF4CdtZ/P17ZZQlxfnZ9mR/F7n97b9ruYm6SeWw9ozr7pfQP5/jopy3TpZpq6rQK6IiJOj4hlEbEqIm6JiJ9FxMURcVbt32dr+05GxH+Y99xbI+IttZ//mfUHsd75P2l+GlW2zNfsu5jrsY2eNsryE46/ROZMzAzk++tkCOr+n9yf+zCVLeS1gOxVKYwlV1W7JZ5h4XdRP6zSyXP6od36QlOHp3j8mccXPG9IQy7dzJmXgrBXub46P/XnC7ztV9/GkI4/Pqv/Lup7bI0M4vtrd57E1ge28kq8suB5S5cs9QlxOXMCsFc1+4985flXFvokoxROkuokhvknw939b+9m+MThlieftbo0ZLPn9EO7k90mDk4wc2xmwfPOWXGOJ4Bz5ovC23HqT3i68vwruXT7pYU96SaFk4YWG0O7k88aXRx+2ZJlnLPiHE4eOjmZ+vhmcW66cJNLOfugmxPBnACspaL/500h/n7FkEJy60RR4iyLbhKAh4CspaJPDKcQf79iKMrSzkWJs4pcBWQtFf0KUinE32kMizkJryhnxBYlzqrxEJC1VPTuewrxdxJDCnFaOXgIyDJT9O57CvF3EoNPwrM8uAdgloDxm8fZeWjnwu1njDPxkYkcIrKBW74cnn9+4fbhYThypOOXcQ/ArGAyOQlv+XKQFv4sX97+uZa/Ro1/q+0ZcAIwS0Aml47MoQGxYnMCMEtACnMVVj0uAzVrYZDXR3CppA2aE4BZE76IiZWdh4DMmnBppg3U8HB32zPgHoBZEyksI9GV4eHmZYSWvi5KPbPiHoBZE4W7PsKRIxCx8CeHhqUvXOaauZ4TgKSzJe2e93NE0ifr9rlI0uF5+3yu1/c167dMSjNLIIXrKQAuc+2DTM8ElrQUOAisjYgfz9t+EfCpiHhvN6/nM4Etb+3W5C+7pNYokpo/lvCKBoPWzZnAWc8BXAz80/zG36zIql6a2WoivMq/l7LIeg7gcuD2Jo/9hqQ9ku6R9GsZv6+Z9UHhJsKtK5klAEknAO8D/m+Dhx8G3hQRa4DrgTtbvM5GSZOSJqenp7MKz8wWoXAT4daVLHsAlwIPR8RT9Q9ExJGIOFq7vQNYJmlFoxeJiG0RMRYRYyMjIxmGZylJZmLRWkpqIjyHOvmyy3IO4AqaDP9IeiPwVESEpHFmE8/PMnxvKxCfYduDjJYM7tTcGkVJTISXpZw1IZkkAEmvAd4DbJq3bTNARNwIfAD4qKQZ4EXg8kj5QgTWV55Y7EHWpZAdJJSqT4SXWSYJICJeAF5ft+3GebdvAG7I4r0qacBHff3miUXS+U5dW19pPhO4CEr2n9QTi5TuO7VicgKwgUtqYtGswpwAbOB88ROzNHg1UMvFoCYWB3lBl4Hwip/5S2X+JgNOAFZapSw3zbqBcULpXonmbzwEVAQ+AWZRkr6gSyrfadmXkLaW3AMoAv9nXJSky039nVoC3AOw0kqi3LQMFzEpw2ewhpwALHOprPOTRLlpGcaLy/AZrKFMLwiTNV8QpniSuoAICVzQpduLmKRYYeILsRwvxe9onjwvCGMVl9o6P4Vbx8ZH2+lLoJHPioeALFNJT7w24vHt1vx7KDUnAMtUEhOv3fARd2v+PZSaE4BlKomJ15SkUu/fD2X4DBXnBGCZ8jo/dcp8otWgPoOH6frGk8CWucJNvKbESzMs5GG6vnECMEtJGXoGVhgeArJqK/MYfRb8+yk19wCs2nzE3Zp/P6WWWQ9A0o8k7ZW0W9KC03c1639I2i/pEUlvz+q9zaxgBjGx68njtrLuAbwrIp5p8tilwFm1n7XAl2r/mllRLXZZhG4mdhc7Me7J47YGOQR0GfCXMbv40IOSXivp9Ih4coAxmFmWBtHIehiqb7KcBA7gXkm7JG1s8PhKYP6ykAdq244jaaOkSUmT09PTGYZntkgeSrCSyjIBrIuItzM71PMxSe+se7zRkoILlhKMiG0RMRYRYyMjIxmGZ7kqciPqoQQrqcwSQEQcqv37NHAHUL/4ywFg/umgq4BDWb2/Jc6NqFlyMkkAkk6RNDx3G7gEeLRut7uAD9aqgf4lcNjj/2YVNYjzC3wOQ1tZTQK/AbhDsxeOGAK+GhHflLQZICJuBHYAG4D9wAvA72b03mazEr9QRykttkJnEN+Hv/O2MkkAEfEEsKbB9hvn3Q7gY1m8n1lDHmYaPDeyhealIMza8VCClZSXgrDBKPIqlz7KtZJyD8AGo6jr4he5fNWsDScAs1Y8r2Al5gRgZlZRTgBmZhXlBGBmVlFOAGZmFeUEYNZK3ucAuArJ+sjnAZi1kneZqquQrI/cA7DyyOpo3UfdVhFOAFYeWZ1s5qPuzqSYKFOMKWFOAGa2OCkmyhRjSpgTgJlZRTkBmKUs7yokKzUnALOs9GP8uaiL6FkhOAGY1VvsUbfHn61gnADKzlUR3evHUXeKv/9e/zZSHJ5KMaaE9ZwAJI1K+jtJ+yR9X9InGuxzkaTDknbXfj7X6/tah3xUmqZB/f5bNfK9/m2kODyVYkwJy+JM4BngP0XEw5KGgV2S7ouIf6zb7zsR8d4M3s/MOuUDAGuh5x5ARDwZEQ/Xbj8P7ANW9vq6ZmbWX5nOAUhaDfw6MNHg4d+QtEfSPZJ+Lcv3NUuCx5mtYDJLAJJOBb4OfDIi6gfcHgbeFBFrgOuBO1u8zkZJk5Imp6enswrPrP8ajT9XlYsPCiGTBCBpGbON//aI+Kv6xyPiSEQcrd3eASyTtKLRa0XEtogYi4ixkZGRLMKrNldF5Cvl338/Y/PcQyH0PAksScAtwL6I+OMm+7wReCoiQtI4s4nnZ72+t3XA1Q/5yvv3PzzcuNEdHs4/NstdFlVA64B/D+yVtLu27feAMwEi4kbgA8BHJc0ALwKXR1S5f2yWkWblnHMNvBt5a6HnBBAR3wXUZp8bgBt6fS8zq+OhFuuBzwQ2M6soJwBLlytJiivlyW97lROApcvDG4OXVdL1kgyF4ARgZv/MSbdSnADMisxDLdaDLMpAzSwvHlKxHrgHYNXgCWWzBZwALF1ZDm94bNtsAQ8BWbo8vDF4rZaOsNJxD8AsLykOS7l8s1KcAMzy4mEpy5kTgJlZRTkBWDW4Xt5sAScAqwaPbWcrxfkL65oTgA1W2RqOsn2eTnn+ohScAGywytZw9PJ5PCxlOfN5AGZ58fCT5cw9gLKq6tCEmXUskwQgab2kxyXtl/SZBo+fKOl/1x6fkLQ6i/e1Fso21FImTs6WiJ4TgKSlwJ8BlwLnAldIOrdutw8DP4+ItwB/Avxhr+9rOXMjtnhlSM6evyiFLHoA48D+iHgiIl4CvgZcVrfPZcBttdv/D7hYUssLyVviFtuIla3hKNvn6ZTLakshi0nglcDUvPsHgLXN9omIGUmHgdcDz2Tw/lYkZWsgyvZ5rFKy6AE0OpKPRewzu6O0UdKkpMnp6emegzMzs8aySAAHgNF591cBh5rtI2kIOA14ttGLRcS2iBiLiLGRkZEMwquoqg5NmFnHskgAO4GzJL1Z0gnA5cBddfvcBVxVu/0B4NsR0bAHYBnxGG26nJwtET3PAdTG9D8OfAtYCtwaEd+X9HlgMiLuAm4B/pek/cwe+V/e6/taTpYvbz3R60asPSdhS0QmZwJHxA5gR922z827/Qvgd7J4L8tZq8bfnTqzQvGZwGaD5nMoLBFOAGaDVoYTwawUnADMzCrKCcDMrKKcAKw7LmE0Kw1fD8C64xJGs9JwD8Bs0NyLskS4B2A2aO5FWSLcA7BycY29WcecAKxcXGNv1rFyJQAf/Vm/+G/LSqhcCcBHf9XWz8bYf1tWQuVKAFZtbozNuuIEYGZWUU4AVg2usTdbwAnAqsG192YLlCsB+AxLa6bXKh7/bVkJletMYB/lVdvwcOOJ4GbbofOJY/9tWQmVqwdg6etnPf2RI7OXpaz/ceNt1lBPPQBJXwT+DfAS8E/A70bEcw32+xHwPPAKMBMRY728rxWY6+nNktFrD+A+4LyIOB/4AfDZFvu+KyIucONvZpaGnhJARNwbETO1uw8Cq3oPyczMBiHLOYAPAfc0eSyAeyXtkrSx1YtI2ihpUtLk9PR0huFZpbmKx2yBtnMAkv4GeGODh66JiL+u7XMNMANsb/Iy6yLikKRfBe6T9FhE3N9ox4jYBmwDGBsbiw4+g1l7ngg2W6BtAoiId7d6XNJVwHuBiyOiYYMdEYdq/z4t6Q5gHGiYAKzkWpVqmtlA9TQEJGk98GngfRHxQpN9TpE0PHcbuAR4tJf3tQJzqaZZMnqdA7gBGGZ2WGe3pBsBJJ0haUdtnzcA35W0B3gIuDsivtnj+5qZWY96Og8gIt7SZPshYEPt9hPAml7exyps+fLmQ0buNZj1xGcCW9p84phZ3zgBmJlVlBOAmVlFOQGYmVWUE4CZWUU5AVjavISDWd+U64IwVj4u9TTrG/cAzMwqygnAzKyinADMzCrKCcDMrKKcAMzMKkpNlvBPgqRp4Md5x9GFFcAzeQexCI57cIoYMzjuQeo15jdFxEgnOyadAIpG0mQRL3rvuAeniDGD4x6kQcbsISAzs4pyAjAzqygngGxtyzuARXLcg1PEmMFxD9LAYvYcgJlZRbkHYGZWUU4AGZP0BUmPSNot6V5JZ+QdUyckfVHSY7XY75D02rxjakfS70j6vqRjkpKv9JC0XtLjkvZL+kze8XRC0q2Snpb0aN6xdErSqKS/k7Sv9vfxibxj6oSkkyQ9JGlPLe7/3Pf39BBQtiQtj4gjtdv/ETg3IjbnHFZbki4Bvh0RM5L+ECAiPp1zWC1JeitwDLgJ+FRETOYcUlOSlgI/AN4DHAB2AldExD/mGlgbkt4JHAX+MiLOyzueTkg6HTg9Ih6WNAzsAt5fgN+1gFMi4qikZcB3gU9ExIP9ek/3ADI21/jXnAIUIsNGxL0RMVO7+yCwKs94OhER+yLi8bzj6NA4sD8inoiIl4CvAZflHFNbEXE/8GzecXQjIp6MiIdrt58H9gEr842qvZh1tHZ3We2nr+2HE0AfSPoDSVPAvwM+l3c8i/Ah4J68gyiZlcDUvPsHKECjVHSSVgO/DkzkG0lnJC2VtBt4GrgvIvoatxPAIkj6G0mPNvi5DCAiromIUWA78PF8o/1n7eKu7XMNMMNs7LnrJOaCUINthegdFpWkU4GvA5+s65knKyJeiYgLmO2Bj0vq67Cbrwi2CBHx7g53/SpwN3BdH8PpWLu4JV0FvBe4OBKZHOrid526A8DovPurgEM5xVJ6tTH0rwPbI+Kv8o6nWxHxnKS/B9YDfZuAdw8gY5LOmnf3fcBjecXSDUnrgU8D74uIF/KOp4R2AmdJerOkE4DLgbtyjqmUapOptwD7IuKP846nU5JG5qrvJJ0MvJs+tx+uAsqYpK8DZzNbnfJjYHNEHMw3qvYk7QdOBH5W2/Rg6tVLkn4buB4YAZ4DdkfEb+YbVXOSNgB/CiwFbo2IP8g5pLYk3Q5cxOwKlU8B10XELbkG1YakfwV8B9jL7P9DgN+LiB35RdWepPOB25j9+1gC/J+I+Hxf39MJwMysmjwEZGZWUU4AZmYV5QRgZlZRTgBmZhXlBGBmVlFOAGZmFeUEYGZWUU4AZmYV9f8BNV36sVspJZsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# 加载数据集\n",
    "def loadDataSet(dataPath):\n",
    "    dataMat = []\n",
    "    labelMat = []\n",
    "    fr = open(dataPath)\n",
    "    for line in fr.readlines():\n",
    "        lineArr = line.strip().split()\n",
    "        dataMat.append([1.0, float(lineArr[0]), float(lineArr[1])])\n",
    "        labelMat.append(int(lineArr[2]))\n",
    "    return dataMat, labelMat\n",
    "\n",
    "# 画散点图\n",
    "def plotDataSet(dataMat, labelMat):\n",
    "    dataArr = np.array(dataMat)\n",
    "    n = np.shape(dataArr)[0]\n",
    "    xcord1 = []\n",
    "    ycord1 = []\n",
    "    xcord2 = []\n",
    "    ycord2 = []\n",
    "    for i in range(n):\n",
    "        if int(labelMat[i]) == 1:\n",
    "            xcord1.append(dataArr[i, 1])\n",
    "            ycord1.append(dataArr[i, 2])\n",
    "        else:\n",
    "            xcord2.append(dataArr[i, 1])\n",
    "            ycord2.append(dataArr[i, 2])\n",
    "            \n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    ax.scatter(xcord1, ycord1, s=30, c='red', marker='s')\n",
    "    ax.scatter(xcord2, ycord2, s=30, c='green')\n",
    "    plt.show()\n",
    "\n",
    "dataMat, labelMat = loadDataSet('./data/1.txt')\n",
    "plotDataSet(dataMat, labelMat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2.2 　训练算法：使用梯度上升找到最佳参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2.77955136]\n",
      " [ 0.36268202]\n",
      " [-0.44923697]]\n"
     ]
    }
   ],
   "source": [
    "def sigmoid(inX):\n",
    "    return 1.0 / (1 + np.exp(-inX))\n",
    "\n",
    "def gradAscent(dataMatIn, classLabels):\n",
    "    dataMatrix = np.mat(dataMatIn)\n",
    "    labelMat = np.mat(classLabels).transpose()\n",
    "    m, n = np.shape(dataMatrix)\n",
    "    alpha = 0.001\n",
    "    maxCycles = 200\n",
    "    weights = np.ones((n, 1))\n",
    "    for k in range(maxCycles):\n",
    "        h = sigmoid(dataMatrix * weights)\n",
    "        error = (labelMat - h)\n",
    "        weights = weights + alpha * dataMatrix.transpose() * error\n",
    "    return weights\n",
    "\n",
    "weights = gradAscent(dataMat, labelMat)\n",
    "print(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2.3 　分析数据：画出决策边界\n",
    "\n",
    "在 Sigmoid 函数 $\\delta(z) = \\frac{1}{1 + e^{-z}}$ 中，当 $z=0$ 时，$\\delta(z)=0.5$ 为决策边界。所以我们令：\n",
    "\n",
    "$$\n",
    "w_0 + w_1 x + w_2 y = 0\n",
    "$$\n",
    "\n",
    "这就是 Logistic 回归的决策边界：\n",
    "\n",
    "$$\n",
    "y = \\frac{-w_0 - w_1 x}{w_2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt03OV95/H3V5Lv0siAZVuS72CwLQXD4iOHdZOSJSHgXKA96R7YpaFNNjZpw6Z7muMkzUnSJds9W6fbZg+0AVPYkIaQ9mwC5SQmgTbtIeEEY5vasWUbsM1FtnyRbawLvkr67h8zkqXRzGhG85v5/Wbm8zpHR5qZn2a+o7Gf7+/3PN/neczdERGRylMVdgAiIhIOJQARkQqlBCAiUqGUAEREKpQSgIhIhVICEBGpUEoAIiIVSglARKRCKQGIiFSomrADyGTWrFm+aNGisMMQESkZ27dvP+HuDdkcG+kEsGjRIrZt2xZ2GCIiJcPM3sr2WHUBiYhUKCUAEZEKpQQgIlKhsk4AZvaYmR03s90j7vtTMztsZjsSX2vT/O6tZvaqme03sy8FEbiIiOQnlyuA7wC3prj/r9z9usTX5uQHzawa+GvgNmAFcJeZrZhIsCIiEpysE4C7vwCcmsBrtAH73f2gu18AfgDcPoHnERGRAAUxBvA5M/t1oovoshSPNwMdI24fStwnIiIhyjcBfBu4ErgOOAL87xTHWIr70u5DaWbrzGybmW3r6urKMzyR0tfR3cF9m++j7ZE27tt8Hx3dHeP/kkgW8poI5u7Hhn42s0eAH6c47BAwf8TteUBnhufcBGwCWLVqlTYslorW0d3ByodW0nehj4uDF9lxdAdP7HqCnffuZH79/PGfQCSDvK4AzKxxxM3fAnanOGwrsNTMFpvZZOBO4Jl8XlekUmx8ceNw4w9wcfAifRf62PjixpAjk3KQ9RWAmT0J3ATMMrNDwNeBm8zsOuJdOm8C6xPHNgF/6+5r3b3fzD4H/AyoBh5z9/ZA34VImdpyeMtw4z/k4uBFXj78ckgRSTnJOgG4+10p7n40zbGdwNoRtzcDY0pERSSz1c2r2XF0x6gkMKlqEm3NbSFGJeVCM4FFImzDmg3UTq5lUtUkIN74106uZcOaDSFHJuVACUAkwubXz2fnvTtZf8N62praWH/Deg0AS2AivRy0SLno6O5g44sb2XJ4C6ubV7NhzYasG/H59fN5YO0DBY5QKpESgEiBqZRTokpdQCIFplJOiSolAJECUymnRJUSgEgBjFy+4Vz/OWpsdG9rVEo5tcxEZdMYgEjAkvv8a6pqGPABaqyGfu+PTCmnxiZEVwAiAUvu8+8f7KemqoblDcsjVcqpsQnRFYBIwNL1+U+rmcaWz2wJKaqxNDYhugIQCdjq5tXDM3eHRKXPf6RSiVMKRwlAJGClsnxDqcQphaMEIBKwUlm+oRhxqsoo2sw9unuurFq1yrdt2xZ2GCIyAclVRkNXGFFMhuXEzLa7+6psjtUVgIgUhKqMok8JQEQKQlVG0acEICIFoSqj6FMCEClzYQ3EFrLKSIPLwdAgsEgZC3sgdmgfhF+8/QsGfZAqq+J9C96X034IqZ5Tg8vpFWQQ2MweM7PjZrZ7xH3fNLN9ZvZrM3vKzGam+d03zWyXme0wM7XoIkUS9kDs/Pr5bFizgbe732bfiX3sPLaTh7c/zMqHVk74rD3s91ROcukC+g5wa9J9zwOt7n4t8Brw5Qy//wF3vy7bzCQio02k2yMKA7FBN9hReE/lIusE4O4vAKeS7nvO3fsTN18C5gUYm4gkDHV7PLz9YbZ2bs36LDoKA7FBN9hReE/lIshB4E8Bz6Z5zIHnzGy7ma3L9CRmts7MtpnZtq6urgDDEyldX/35Vzl97nTOZ9FRWO4h6AY7Cu+pXASSAMzsK0A/8ESaQ9a4+78DbgP+0Mzen+653H2Tu69y91UNDQ1BhCdFpgqNYHV0d/C9Xd/DGV2wkc1ZdBSWpQi6wY7CeyoXOVUBmdki4Mfu3jrivnuAe4Gb3f1MFs/xp0Cfu//FeMeqCqj0qEIjePdtvo+/2fo3DDI46n7D+OS1n6RuSh1bDm9hdfPqvKprCmmoGujlwy/T1twW2TjLQS5VQHntB2BmtwJfBH4zXeNvZjOAKnfvTfx8C3B/Pq8r0ZVpwO+BtQ+EHF1p2nJ4y5jGH+IJ4OlXn+bMxTOR39Frfv18ff4RlEsZ6JPAr4BrzOyQmX0aeBCoA55PlHg+lDi2ycw2J351DvBLM9sJvAz8xN1/Gui7kMhQhUbwUvWhV1HF0iuWDjf+oHJIyV3WVwDufleKux9Nc2wnsDbx80Fg5YSik5Kzunk1O47uGJUEolyhMdQ1EeUulA1rNvDErifGdKtNrZmqZCt50VIQEqhSqtCYaGllsaUb9HzfgvepHFLyoqUgJHClMuB33+b7eHj7w2OuVtbfsL4k+qujNOCefCV197V3871ffy/SV1blKpdBYCUAKVkd3R189edf5dn9z4LBbVfdxjc+8I2sG5q2R9rY2rl17P1NbZHavD2TKCTb5ERUYzUM+ADVVdX0D/arEqzIilYFJBKWju4O3vPt99B9vnv4vsd3Ps7T+55m12d3ZdXQlNp4RSpRqK5JrvzqTywO0D8Y/65KsOjSGIBESraTyDa+uJGe8z1j7u8935t1FUwpjVdEWarKr2QanI4mXQFIZCR3JWSqa99yeMuYmbEAgwxm3dAMDa6G3YVS6lJdSSUrtSurSqErAImMXFaNXN28GsPG3F9FVU4NzVAXypbPbOGBtQ+o8Z+A5CupGqvBMGqq4ueXurKKLiUAiYxcJpFtWLOB2JTYmPvrptRVREMTpfWWkstU7111L7/69K+494Z7tVZPxKkLSCIjl0HZ+fXz2fXZXRmrgKI8ySuf2JK7yv7tyL/xyCuPsGzWsrx325qoVIPRq+etLmoMkjuVgUpkBFnXHqUa+aBjSzV/YUiU3qeEoyBbQooUWpDL/EZ528B8Y8tUdROl9ynRpy4giZSg6toLsShdUF1K+cY2XtXNeM8V5a4xKS4lAClLQU/yyqVEtdCxJS8OlyzTcwX5PqT0qQtIIiHoqpagJ3kF2aWUb2wju8pWzlnJlOopWZdcRrlrTIpPVwASukKclQY9ySufbptUXS75xjayqyyX9YC0X4OMpAQgoSvULmKZxhNy7Qdf0bCC7Z3bR+3MlU23TabkFtS6OLmMm5TD+kcSHHUBSeiKfVaa6z4AHd0dPL3v6THbMk6fNH3cbpuodbmUy/pHUZoIV8qUACR0qbY8LORZaa6N8sYXN3Lm4ugtrw3jjmvuGLfbJmpdLkGW2oalVDbyKQU5JQAze8zMjpvZ7hH3XW5mz5vZ64nvl6X53XsSx7xuZvfkG7iUj2KflebaKKc63nH2ntg77msVO7llo9TXP4raVVUpy/UK4DvArUn3fQn4Z3dfCvxz4vYoZnY58HVgNdAGfD1dopDKU+yz0lwb5Xwa8XLpcomSqF1VlbKcEoC7vwCcSrr7duDxxM+PA3ek+NUPA8+7+yl3fwd4nrGJRCpYMc9Kc22U82nEi5XcKqlPPIpXVaUq57WAzGwR8GN3b03cPu3uM0c8/o67X5b0O18Aprr7/0jc/ipw1t3/ItNraS0gKZRct1KMwtaL6UR53aNCqLT3m6sobgk5duF2UuzmAZjZOmAdwIIFCwoZk1SwXJecKMbWixNdoqFQZbRRpY18ghNEAjhmZo3ufsTMGoHjKY45BNw04vY84F9TPZm7bwI2QfwKIID4RCIvn8lwUegTzyZ5BbkGURT2Qi4HQSSAZ4B7gP+V+P6PKY75GfA/Rwz83gJ8OYDXloBpobBwpDuL/8j3P8LUmqkZP4t0i8Od7T9LR3dHwT+/bJKX1iCKplzLQJ8EfgVcY2aHzOzTxBv+D5nZ68CHErcxs1Vm9rcA7n4K+AawNfF1f+I+iRDVV4cn3Vn8ruO7xv0shgaph9YDGrK3a29RPr9syjJVuhlNuVYB3eXuje4+yd3nufuj7n7S3W9296WJ76cSx25z9/8y4ncfc/erEl//N+g3IvnTf9LwpKpsGSndZzF0xTa/fj6xyaO3yOz3/qJ8ftl0Qb3w9guhd1PJWFoLSIZFoS+5Uo23xDOM/SySu1Wy+Z1CGG99oY7uDl498eqY36uxGpVuhkxLQcgw1VeHJ3m+wHtmv4caG31+lvxZJF+xpVKMz2+8eRIbX9zIgA+M+b3qqmpNiAuZEoAMS/cf+e5r7y7pSUZRmCSVTQwjJ8P95D/9hLopdRknn2XaGjLd7xTCeJPdthzeQv9g/5jfWzZrmQaAQ6ZN4WWU5AlPd197N7c9cVvJTrqJwqShicYw3uSzVJvDT6qaxLJZy5hWMy0y9fHp4lx/w3qVchZALhPBlAAko1L/zxuF+AsVQxSSWzZKJc5ykUsCUBeQZFTqA8NRiL9QMZTK0s6lEmclUhWQZFTqO0hFIf5sY5jIJLxSmRFbKnFWGnUBSUalfvkehfiziSEKcUp5UBeQBKbUL9+jEH82MWgSnoRBVwAiEdD2SBtbO7eOvb+pjS2f2RJCRFIs7o6ZQSwGvb1jD6irg56erJ8vistBi0gGgYxVBNSASOEc7zlHe2cPuw93x793dnPPjYv4zPuXpP7sIP39AVACEImA5KUgJjSJK4QGRFJzdzpOnaW9s5vdnYnG/nAPJ/rODx+zeNYMVs6fyeJZM0KLUwlAJAK0yUnpGhh0Dnb1DZ/Z7+7sZk9nDz3n4rOfq6uMpbNr+c2rG2hpitHaXM/yxjrqpqZf/K9YlABEMijm/ggqlYy+8/0DvHa0j/bOS104+470cvZifK2jKTVVLGuM8dGVTfHGvqmea+bWMXVSdciRp6YEIJKGNjGpbH3n+9l7pIf2w93s7uyhvbOH14/10j8YL5ypm1LD8qYYd7UtoKUpRktzjKsaaqmpLp3iSiUAkTQqba/dSvbOuxeGz+jbO+ON/hsn32WoSHJW7WRWNNXzgWsaaGmqp6UpxoLLp1NVlWq78wmqq0s/iF8gSgAiaURhGYmchNCAlBp351jP+VFVOHs6ezh8+uzwMc0zp9HSFOOO65uH++xn102Jl2oWUgiVWkoAImlEYRmJnJR7qWeOZa6Dg87bp86MOrPf09nNib4LAJjFK3FuWHgZn7xxIa3N9axojHHZjMmFfieRkXcCMLNrgL8fcdcS4Gvu/q0Rx9xEfLP4NxJ3/cjd78/3tUUKKZDSzDJQzIHwjDKUufYPDLK/q4/2wz3DDf7ezh56z8crcWqqjKVz6rjpmtm0DlfixJgxpbLPgQOdCWxm1cBhYLW7vzXi/puAL7j7R3N5Ps0ElrCNtyZ/uYvUGkWJLphz1ZN4tWERu+dcSXvia++CZVzoHwRg6qQqljfGhqtwWprquXpuLVNqolmJE7QwZwLfDBwY2fiLlLJKL80MeyC899xF9iQqcHav/W/smbOE12ctYKAq3pjHzvXRcuwA99y4kJamelqbYyyeVUt1kIOzZSzoBHAn8GSax240s51AJ/GrgfaAX1tEAlbMgfATfefjFTgjKnHePHlm+PGGxdfTeuwAH9y/hZZjB2k9doB53ccwgCf/JPB4KkFgCcDMJgMfB76c4uFXgIXu3mdma4GngaVpnmcdsA5gwYIFQYUnIhNQiIFwd6ez+9xwJU574vvRnnPDx8y/fBotjfV84oZ5w2WXs+un5fVeZKzAxgDM7HbgD939liyOfRNY5e4nMh2nMYDyFZmBRcko3zGAwUHnjZPvjmrod3d2c/pMPKFUGVzZUBufSNVUT0tzjJbGeuqnp1gmQYvdZSWsMYC7SNP9Y2ZzgWPu7mbWRnwfgpMBvraUEM2wzUORG8Fc1ii6ODDI68cuLZPQnqixf/dCfJmEydVVXD23lltb5tLSHD+rXz43xrTJWQ7OqpEPXCAJwMymAx8C1o+4714Ad38I+ATwWTPrB84Cd3qUNyKQggp7YLGkBb3iZxYJJdVA+NkLA+w72sPuRG397sM9vHq0lwsD8Uqc6ZOrWd4Yi3fhNNfT2lTPVbNrmVxTOsskVIJAEoC7nwGuSLrvoRE/Pwg8GMRrVaQyu/QtuRm2hRCVzzSLhNJ99uLw2fzQipcHuvpILInDzOmTaGmK8XtrFg3PnF10xQxV4pSAyp4FUSrKbJ33kpthWwgR/UyPz5g5XFvf/r3t7O7spuPUpWUS5sam0tIU47bWS904zTOnFX6ZBCkIJQApOs2wDZ+7c+ids7RffSPtc66MT6qavYTjdZcu5Bcd6eHa5pmJ1S7jjf2s2ikhRi1B057ApSDT2VWEP79MKn2G7YQ/0wn83sCg88aJvsTA7KXtCLvPxq/AqgcHuOpkBy1HD9By/AAtxw6y4thBYuffzeadSMRoT2CJvGLNsC27ctNxVvw83z+QVInTw57OnuENSybXVLF8bh1r39MY76//7VtY1vUmU/svFPNdlLaojN8EQFcApaAMrwCKIVLr2CQL4DM9cyG+Ycnuw/GSy92He3j9eC8XB+K/P2NyNS1N9axIDMy2Nse4sqGWSSM3LCmjxqxoIv7/UVcA5UbrvE9IpMtNc/xMT5+5MFxbP9TgHzxxacOSK2ZMZkVTjPdfvYTW5vikqoXZbFiiRr6iKQGUAv0nnZBIl5um+UzdneM950Y19LsPj96wpKl+Kiua6vnYyqb4apfNMebGpqoSR3KmBCBlKxLlphm6WLy7m7dPnRlu6IfO8Ic2LAFYMmsG1y+Yyd3vXTh8Zn95sTcsUTdR2dIYgAQuKgOvkRgDSJyV91sVB6+YN7yG/e45V7Jn6fWjNiy5anYtrYna+qENS2qjsGFJxPu8iy7ifw+NAUhoorTOTy7r2ATp3MUBXjvWGy+5vOUPaJ+zhL0Nizk/KV5DP/XiOZZ1vcnt1zcN19dfPaeOqZOqdbZdCspoTE5XABKo+zbfx8PbHx7T7bL+hvXhD7wWQN/5oUqc7uEa+/3H++hPrJNQd65veO36lmMHaDl+kCUnD1Hjg6nPFqN4dhnFmCQtXQFIaCI98JpKDmfcp969MGpwdk9nD2+cvFSJM6t2Mi1N9dy8fPbwmf2CWbWU9NBsLBZ2BFJASgASqEgMvOYiRePvwBGm0L7n2KVNSzq7OdJ9acOS5pnTaG2Occf1zcODs7PrppRfJU6Jrjcl2VECkECV2jo/gxhvXdaYGJxdMrwQ2qnp9fDdbVhiw5K2xZcPbzK+oinGzOlZVuKUUX/xGOXwHiqcxgAkcFFd5+fiwCAHuvoulV0e7mHPq4fomzIdgEkDF7m66y1ajh+k9egBWp76O5Y31jF9chHPk6LW3x6FeDQwnhONAUioirXOTybnLg6w72jvqD77fUd7udAf37Bk2qRqVjTF+O3dP48vgHb0AFefeJvJg/2XnmThZcUPvJyvGCYqoktnlwMlACl5PecusidRgbMnsefsga53GUhU4sSm1tDaXM89Ny6kpame1uZ6Fs9KbFjyB2tCjj6JzmiliJQApKR09Z4fNWu2vbOHt06eGX58TmwKLU31fLhl7vBG4/Muy7Bhic64M9Pfp6wpAUgkuTuHT5+NN/RDNfad3RzrOT98zILLp9PaHOM/rpo/3Ng31OW4YYnOuDPT36esBZYAzOxNoBcYAPqTByEsfgr2f4C1wBng99z9laBeX0pXfMOSd8ec2Z8+Ey8lrUpU4vz7K2cNN/QrmmLUT5sUcuQyYcUY2NXg8biCvgL4gLufSPPYbcDSxNdq4NuJ71JBLvQP8tqx3uG++vbOHvYe6eHMhcSGJdVVXDO3jtta57KiqZ7WphjL5saYNrk65MglpYk2srkM7E60G0qDx+MqZhfQ7cB3PV53+pKZzTSzRnc/UsQYpIjiG5b0sidRibO7s5vXjo3esGRF06UunNbmeq6anbRhiURbMRpZna0XTJAJwIHnzMyBh919U9LjzUDHiNuHEveNSgBmtg5YB7BgwYIAw5NC6j5zkfYj8dr69s5udnf2cLCrj0QhDpdNn0Rrcz2f+o3FtCYqcbLasCQK1JUgZSrIBLDG3TvNbDbwvJntc/cXRjye6n/6mJkkicSxCeITwQKMTwIS37BkxO5UR7rpOHVpw5LG+qm0NMX4yNC+s831NC6Yg5VqI6quBClTgSUAd+9MfD9uZk8BbcDIBHAIGDkddB7QGdTrS/DcnY5TZxNn9Jc2Ge/qvVSJs+iK6Vw7byZ3tS2I19g3xbiiNkUljhpRkcgJJAGY2Qygyt17Ez/fAtyfdNgzwOfM7AfEB3+71f8fHQODzsGuvnhDn+iv39PZQ8+5+MzY6ipj6exa3r+0IVGJE2NFU4y6qarEkQkoxvwCzWEYV1BXAHOApxKTbWqA77v7T83sXgB3fwjYTLwEdD/xMtDfD+i1JUfn+wd47WjfqDP7vUd6OHcxvkzClJoqljXG+NjKpsTM2REblkSZ+uqLb6KNbDE+D33m49JicGXu3eQNSzp7eP1Y76UNS6bUsDyxyuXQssZXNsygJuhKnGIsKlao14jCgmgiWdJicBXqnXcvXBqcTXx/48SlDUuumDGZFU0xbrpmCa1DG5aUSiVOmNSVIGVKCaAEuTvHes6POKuP99cfPn2pEqd55jRWNMX4+Mqm4bLLObEQNywp5UZUXQlSppQAIm5w0Hn71Jnhhn5obZyT714A4r0Ti2fN4IaFl/HJGxfS2lzPisYYl83IcsOSYinVRlTjClLGlAAipH9gkP1dfYnJVPEGf29nD73n45U4NVXG1XPq+A/LZg/X1y9vjDFjij7GglH5qpQxtRwhSd6wZE9iw5LziQ1Lpk6qYnljfM/ZoQXQrp5by5SaiFfiiEjJUAIogt6hDUs6L21FuL+rb9SGJS1N9fzue+NdOC1NMZY01MY3LBERKRAlgICd6Ds/aknj9sPdvDliw5KGuim0NsW4pWVOdhuWiIgUiBJAQF7cf4I//oedHO05N3zf/Mun0dJYzydumEdL4sx+dt3UEKMUEblECSAgc+un8t4ll8ercJpitDTWUz9dyySUvLDLV1WFJAWkBBCQKxtq+dad14cdhgQt7EZWVUhSQNp5Q8pHurPyXM/WY7H4BIvkr1gs/xhFIkQJQMpHT098bZ7kr1zP4nXWnZ0oJsooxhRhSgAiMjFRTJRRjCnClABERCqUEoBIlAU1riGSghKASFAK0f8c1LiGSApKACLJJnrWrf5nKTFKAOVOVRG5K8RZdxT//vn+24hi91QUY4qwvBOAmc03s38xs71m1m5mn09xzE1m1m1mOxJfX8v3dSVLOiuNpmL9/TM18vn+24hi91QUY4qwIGYC9wN/7O6vmFkdsN3Mnnf3PUnH/cLdPxrA64lItnQCIBnkfQXg7kfc/ZXEz73AXqA53+cVEZHCCnQMwMwWAdcDW1I8fKOZ7TSzZ82sJcjXFYkE9TNLiQksAZhZLfBD4I/cPbnD7RVgobuvBB4Ans7wPOvMbJuZbevq6goqPJHCS9X/XKlUfFASAkkAZjaJeOP/hLv/KPlxd+9x977Ez5uBSWY2K9Vzufsmd1/l7qsaGhqCCK+yqSoiXFH++xcyNo09lIS8B4EtvpXVo8Bed//LNMfMBY65u5tZG/HEczLf15YsqPohXGH//TPtZxB2bBK6IKqA1gC/C+wysx2J+/4EWADg7g8BnwA+a2b9wFngTvdKvj4WCch4G8aokZcM8k4A7v5LIOOGtu7+IPBgvq8lIknU1SJ50ExgEZEKpQQg0aVKktIV5cFvGaYEINGl7o3iCyrpakmGkqAEICKXKOlWFCUAkVKmrhbJQxBloCISFnWpSB50BSCVQQPKImMoAUh0Bdm9ob5tkTHUBSTRpe6N4su0dISUHV0BiIQlit1SKt+sKEoAImFRt5SETAlARKRCKQFIZVC9vMgYSgBSGdS3Hawojl9IzpQApLjKreEot/eTLY1flAUlACmucms48nk/6paSkGkegEhY1P0kIdMVQLmq1K4JEclaIAnAzG41s1fNbL+ZfSnF41PM7O8Tj28xs0VBvK5kUG5dLeVEyVkiIu8EYGbVwF8DtwErgLvMbEXSYZ8G3nH3q4C/Av4839eVkKkRm7hySM4avygLQVwBtAH73f2gu18AfgDcnnTM7cDjiZ//H3CzmWXcSF4ibqKNWLk1HOX2frKlstqyEMQgcDPQMeL2IWB1umPcvd/MuoErgBMBvL6UknJrIMrt/UhFCeIKINWZvE/gmPiBZuvMbJuZbevq6so7OBERSS2IBHAImD/i9jygM90xZlYD1AOnUj2Zu29y91XuvqqhoSGA8CpUpXZNiEjWgkgAW4GlZrbYzCYDdwLPJB3zDHBP4udPAD9395RXABIQ9dFGl5KzRETeYwCJPv3PAT8DqoHH3L3dzO4Htrn7M8CjwN+Z2X7iZ/535vu6EpJYLPNArxqx8SkJS0QEMhPY3TcDm5Pu+9qIn88BvxPEa0nIMjX+uqgTKSmaCSxSbJpDIRGhBCBSbOUwEUzKghKAiEiFUgIQEalQSgCSG5UwipQN7QcguVEJo0jZ0BWASLHpKkoiQlcAIsWmqyiJCF0BSHlRjb1I1pQApLyoxl4ka+WVAHT2J4Wif1tShsorAejsr7IVsjHWvy0pQ+WVAKSyqTEWyYkSgIhIhVICkMqgGnuRMZQApDKo9l5kjPJKAJphKenkW8Wjf1tShsprJrDO8ipbXV3qgeB090P2A8f6tyVlqLyuACT6CllP39MT35Yy+UuNt0hKeV0BmNk3gY8BF4ADwO+7++kUx70J9AIDQL+7r8rndaWEqZ5eJDLyvQJ4Hmh192uB14AvZzj2A+5+nRp/EZFoyCsBuPtz7t6fuPkSMC//kEREpBiCHAP4FPBsmscceM7MtpvZukxPYmbrzGybmW3r6uoKMDypaKriERlj3DEAM/snYG6Kh77i7v+YOOYrQD/wRJqnWePunWY2G3jezPa5+wupDnT3TcAmgFWrVnkW70FkfBoIFhlj3ATg7h/M9LiZ3QN8FLjZ3VM22O7emfh+3MyeAtqAlAlAylymUk0RKaq8uoDM7Fbgi8DH3f1MmmNmmFnd0M/ALcDufF7BN3xHAAAEEUlEQVRXSphKNUUiI98xgAeBOuLdOjvM7CEAM2sys82JY+YAvzSzncDLwE/c/ad5vq6IiOQpr3kA7n5Vmvs7gbWJnw8CK/N5HalgsVj6LiNdNYjkRTOBJdo0cUykYJQAREQqlBKAiEiFUgIQEalQSgAiIhVKCUCiTUs4iBRMeW0II+VHpZ4iBaMrABGRCqUEICJSoZQAREQqlBKAiEiFUgIQEalQlmYJ/0gwsy7grbDjyMEs4ETYQUyA4i6eUowZFHcx5RvzQndvyObASCeAUmNm20px03vFXTylGDMo7mIqZszqAhIRqVBKACIiFUoJIFibwg5gghR38ZRizKC4i6loMWsMQESkQukKQESkQikBBMzMvmFmvzazHWb2nJk1hR1TNszsm2a2LxH7U2Y2M+yYxmNmv2Nm7WY2aGaRr/Qws1vN7FUz229mXwo7nmyY2WNmdtzMdocdS7bMbL6Z/YuZ7U38+/h82DFlw8ymmtnLZrYzEfd/L/hrqgsoWGYWc/eexM//FVjh7veGHNa4zOwW4Ofu3m9mfw7g7l8MOayMzGw5MAg8DHzB3beFHFJaZlYNvAZ8CDgEbAXucvc9oQY2DjN7P9AHfNfdW8OOJxtm1gg0uvsrZlYHbAfuKIG/tQEz3L3PzCYBvwQ+7+4vFeo1dQUQsKHGP2EGUBIZ1t2fc/f+xM2XgHlhxpMNd9/r7q+GHUeW2oD97n7Q3S8APwBuDzmmcbn7C8CpsOPIhbsfcfdXEj/3AnuB5nCjGp/H9SVuTkp8FbT9UAIoADP7MzPrAP4z8LWw45mATwHPhh1EmWkGOkbcPkQJNEqlzswWAdcDW8KNJDtmVm1mO4DjwPPuXtC4lQAmwMz+ycx2p/i6HcDdv+Lu84EngM+FG+0l48WdOOYrQD/x2EOXTcwlwlLcVxJXh6XKzGqBHwJ/lHRlHlnuPuDu1xG/Am8zs4J2u2lHsAlw9w9meej3gZ8AXy9gOFkbL24zuwf4KHCzR2RwKIe/ddQdAuaPuD0P6AwplrKX6EP/IfCEu/8o7Hhy5e6nzexfgVuBgg3A6wogYGa2dMTNjwP7woolF2Z2K/BF4OPufibseMrQVmCpmS02s8nAncAzIcdUlhKDqY8Ce939L8OOJ1tm1jBUfWdm04APUuD2Q1VAATOzHwLXEK9OeQu4190PhxvV+MxsPzAFOJm466WoVy+Z2W8BDwANwGlgh7t/ONyo0jOztcC3gGrgMXf/s5BDGpeZPQncRHyFymPA19390VCDGoeZ/QbwC2AX8f+HAH/i7pvDi2p8ZnYt8Djxfx9VwD+4+/0FfU0lABGRyqQuIBGRCqUEICJSoZQAREQqlBKAiEiFUgIQEalQSgAiIhVKCUBEpEIpAYiIVKj/D00j0e4gAo5pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画散点图和分界线\n",
    "def plotBestFit(dataMat, labelMat, weights):\n",
    "    dataArr = np.array(dataMat)\n",
    "    n = np.shape(dataArr)[0]\n",
    "    xcord1 = []\n",
    "    ycord1 = []\n",
    "    xcord2 = []\n",
    "    ycord2 = []\n",
    "    for i in range(n):\n",
    "        if int(labelMat[i]) == 1:\n",
    "            xcord1.append(dataArr[i, 1])\n",
    "            ycord1.append(dataArr[i, 2])\n",
    "        else:\n",
    "            xcord2.append(dataArr[i, 1])\n",
    "            ycord2.append(dataArr[i, 2])\n",
    "            \n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    ax.scatter(xcord1, ycord1, s=30, c='red', marker='s')\n",
    "    ax.scatter(xcord2, ycord2, s=30, c='green')\n",
    "    \n",
    "    # 决策边界\n",
    "    x = np.arange(-3.0, 3.0, 0.1)\n",
    "    y = (-weights[0,0]-weights[1,0]*x)/weights[2,0]\n",
    "    ax.plot(x, y)\n",
    "    \n",
    "    plt.show()\n",
    "    \n",
    "plotBestFit(dataMat, labelMat, weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2.4 　训练算法：随机梯度上升\n",
    "\n",
    "上面的算法需要大量的计算，它在每次更新回归系数时都需要遍历整个数据集，可以改进为一次只用一个样本来更新回归系数，这被称为 **随机梯度上升算法**。由于可以在新样本到来时对分类器进行增量式更新，所以它也是一种 **在线学习算法**，于此相对应，一次处理所有数据的算法被称为 **批处理**。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.963951   0.9826866  0.49153886]\n"
     ]
    }
   ],
   "source": [
    "def stocGradAscent0(dataMatIn, classLabels):\n",
    "    dataMatrix = np.array(dataMatIn)\n",
    "    m, n = np.shape(dataMatrix)\n",
    "    alpha = 0.001\n",
    "    weights = np.ones(n)\n",
    "    for i in range(m):\n",
    "        h = sigmoid(np.sum(dataMatrix[i] * weights))\n",
    "        error = (labelMat[i] - h)\n",
    "        weights = weights + alpha * error * dataMatrix[i]\n",
    "    return weights\n",
    "\n",
    "weights = stocGradAscent0(dataMat, labelMat)\n",
    "print(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "随机梯度上升算法和梯度上升算法在代码上很相似，但也有一些区别：第一，梯度上升算法中 h 和 error 是向量，而随机梯度上升算法中是数值；第二，随机梯度上升算法中没有矩阵的转换过程。\n",
    "\n",
    "画出分界线，可以看出随机梯度上升的拟合效果没有上面的梯度上升算法完美，但是，这并不能表示这种算法不好，判断优化算法优劣的一个方法是看它是否收敛，可以在数据集上多运行几次，看参数是否达到一个稳定值。\n",
    "\n",
    "针对上面的随机梯度上升算法，还可以进行一些改进：\n",
    "\n",
    "* alpha 每次迭代时需要调整：alpha = 4/(1.0+i+j)+0.01\n",
    "* 随机选择样本来更新回归系数\n",
    "\n",
    "这样可以缓解参数收敛的数据波动。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 　示例：从疝气病症预测病马的死亡率\n",
    "#### 5.3.1 　准备数据：处理数据中的缺失值\n",
    "#### 5.3.2 　测试算法：用Logistic回归进行分类\n",
    "### 5.4 　本章小结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第6章　支持向量机\n",
    "\n",
    "### 6.1 　基于最大间隔分隔数据\n",
    "### 6.2 　寻找最大间隔\n",
    "#### 6.2.1 　分类器求解的优化问题\n",
    "#### 6.2.2 　SVM应用的一般框架\n",
    "### 6.3 　SMO高效优化算法\n",
    "#### 6.3.1 　Platt的SMO算法\n",
    "#### 6.3.2 　应用简化版SMO算法处理小规模数据集\n",
    "### 6.4 　利用完整Platt SMO算法加速优化\n",
    "### 6.5 　在复杂数据上应用核函数\n",
    "#### 6.5.1 　利用核函数将数据映射到高维空间\n",
    "#### 6.5.2 　径向基核函数\n",
    "#### 6.5.3 　在测试中使用核函数\n",
    "### 6.6 　示例：手写识别问题回顾\n",
    "### 6.7 　本章小结\n",
    "\n",
    "## 第7章　利用AdaBoost元算法提高分类性能\n",
    "\n",
    "### 7.1 　基于数据集多重抽样的分类器\n",
    "#### 7.1.1 　bagging：基于数据随机重抽样的分类器构建方法\n",
    "#### 7.1.2 　boosting\n",
    "### 7.2 　训练算法：基于错误提升分类器的性能\n",
    "### 7.3 　基于单层决策树构建弱分类器\n",
    "### 7.4 　完整AdaBoost算法的实现\n",
    "### 7.5 　测试算法：基于AdaBoost的分类\n",
    "### 7.6 　示例：在一个难数据集上应用AdaBoost\n",
    "### 7.7 　非均衡分类问题\n",
    "#### 7.7.1 　其他分类性能度量指标：正确率、召回率及ROC曲线\n",
    "#### 7.7.2 　基于代价函数的分类器决策控制\n",
    "#### 7.7.3 　处理非均衡问题的数据抽样方法\n",
    "### 7.8 　本章小结\n",
    "\n",
    "# 第二部分　利用回归预测数值型数据\n",
    "\n",
    "## 第8章　预测数值型数据：回归\n",
    "\n",
    "### 8.1 　用线性回归找到最佳拟合直线\n",
    "### 8.2 　局部加权线性回归\n",
    "### 8.3 　示例：预测鲍鱼的年龄\n",
    "### 8.4 　缩减系数来“理解”数据\n",
    "#### 8.4.1 　岭回归\n",
    "#### 8.4.2 　lasso\n",
    "#### 8.4.3 　前向逐步回归\n",
    "### 8.5 　权衡偏差与方差\n",
    "### 8.6 　示例：预测乐高玩具套装的价格\n",
    "#### 8.6.1 　收集数据：使用Google购物的API\n",
    "#### 8.6.2 　训练算法：建立模型\n",
    "### 8.7 　本章小结\n",
    "\n",
    "## 第9章　树回归\n",
    "\n",
    "### 9.1 　复杂数据的局部性建模\n",
    "### 9.2 　连续和离散型特征的树的构建\n",
    "### 9.3 　将CART算法用于回归\n",
    "#### 9.3.1 　构建树\n",
    "#### 9.3.2 　运行代码\n",
    "### 9.4 　树剪枝\n",
    "#### 9.4.1 　预剪枝\n",
    "#### 9.4.2 　后剪枝\n",
    "### 9.5 　模型树\n",
    "### 9.6 　示例：树回归与标准回归的比较\n",
    "### 9.7 　使用Python的Tkinter库创建GUI\n",
    "#### 9.7.1 　用Tkinter创建GUI\n",
    "#### 9.7.2 　集成Matplotlib和Tkinter\n",
    "### 9.8 　本章小结\n",
    "\n",
    "# 第三部分　无监督学习\n",
    "\n",
    "## 第10章　利用K-均值聚类算法对未标注数据分组\n",
    "\n",
    "### 10.1 　K-均值聚类算法\n",
    "### 10.2 　使用后处理来提高聚类性能\n",
    "### 10.3 　二分K-均值算法\n",
    "### 10.4 　示例：对地图上的点进行聚类\n",
    "#### 10.4.1 　Yahoo! PlaceFinder API\n",
    "#### 10.4.2 　对地理坐标进行聚类\n",
    "### 10.5 　本章小结\n",
    "\n",
    "## 第11章　使用Apriori算法进行关联分析\n",
    "\n",
    "### 11.1 　关联分析\n",
    "### 11.2 　Apriori原理\n",
    "### 11.3 　使用Apriori算法来发现频繁集\n",
    "#### 11.3.1 　生成候选项集\n",
    "#### 11.3.2 　组织完整的Apriori算法\n",
    "### 11.4 　从频繁项集中挖掘关联规则\n",
    "### 11.5 　示例：发现国会投票中的模式\n",
    "#### 11.5.1 　收集数据：构建美国国会投票记录的事务数据集\n",
    "#### 11.5.2 　测试算法：基于美国国会投票记录挖掘关联规则\n",
    "### 11.6 　示例：发现毒蘑菇的相似特征\n",
    "### 11.7 　本章小结\n",
    "\n",
    "## 第12章　使用FP-growth算法来高效发现频繁项集\n",
    "\n",
    "### 12.1 　FP树：用于编码数据集的有效方式\n",
    "### 12.2 　构建FP树\n",
    "#### 12.2.1 　创建FP树的数据结构\n",
    "#### 12.2.2 　构建FP树\n",
    "### 12.3 　从一棵FP树中挖掘频繁项集\n",
    "#### 12.3.1 　抽取条件模式基\n",
    "#### 12.3.2 　创建条件FP树\n",
    "### 12.4 　示例：在Twitter源中发现一些共现词\n",
    "### 12.5 　示例：从新闻网站点击流中挖掘\n",
    "### 12.6 　本章小结\n",
    "\n",
    "# 第四部分　其他工具\n",
    "\n",
    "## 第13章　利用PCA来简化数据\n",
    "\n",
    "### 13.1 　降维技术\n",
    "### 13.2 　PCA\n",
    "#### 13.2.1 　移动坐标轴\n",
    "#### 13.2.2 　在NumPy中实现PCA\n",
    "### 13.3 　示例：利用PCA对半导体制造数据降维\n",
    "### 13.4 　本章小结\n",
    "\n",
    "## 第14章　利用SVD简化数据\n",
    "\n",
    "### 14.1 　SVD的应用\n",
    "#### 14.1.1 　隐性语义索引\n",
    "#### 14.1.2 　推荐系统\n",
    "### 14.2 　矩阵分解\n",
    "### 14.3 　利用Python实现SVD\n",
    "### 14.4 　基于协同过滤的推荐引擎\n",
    "#### 14.4.1 　相似度计算\n",
    "#### 14.4.2 　基于物品的相似度还是基于用户的相似度？\n",
    "#### 14.4.3 　推荐引擎的评价\n",
    "### 14.5 　示例：餐馆菜肴推荐引擎\n",
    "#### 14.5.1 　推荐未尝过的菜肴\n",
    "#### 14.5.2 　利用SVD提高推荐的效果\n",
    "#### 14.5.3 　构建推荐引擎面临的挑战\n",
    "### 14.6 　基于SVD的图像压缩\n",
    "### 14.7 　本章小结\n",
    "\n",
    "## 第15章　大数据与MapReduce\n",
    "\n",
    "### 15.1 　MapReduce：分布式计算的框架\n",
    "### 15.2 　Hadoop流\n",
    "#### 15.2.1 　分布式计算均值和方差的mapper\n",
    "#### 15.2.2 　分布式计算均值和方差的reducer\n",
    "### 15.3 　在Amazon网络服务上运行Hadoop程序\n",
    "#### 15.3.1 　AWS上的可用服务\n",
    "#### 15.3.2 　开启Amazon网络服务之旅\n",
    "#### 15.3.3 　在EMR上运行Hadoop作业\n",
    "### 15.4 　MapReduce上的机器学习\n",
    "### 15.5 　在Python中使用mrjob来自动化MapReduce\n",
    "#### 15.5.1 　mrjob与EMR的无缝集成\n",
    "#### 15.5.2 　mrjob的一个MapReduce脚本剖析\n",
    "### 15.6 　示例：分布式SVM的Pegasos算法\n",
    "#### 15.6.1 　Pegasos算法\n",
    "#### 15.6.2 　训练算法：用mrjob实现MapReduce版本的SVM\n",
    "### 15.7 　你真的需要MapReduce吗？\n",
    "### 15.8 　本章小结\n",
    "\n",
    "## 附录A 　Python入门\n",
    "\n",
    "## 附录B 　线性代数\n",
    "\n",
    "## 附录C 　概率论复习\n",
    "\n",
    "## 附录D 　资源"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}