数据预处理

发表于 2019-10-30 | 分类于人工智能，特征工程

预处理
归一化	列：preprocessing.MinMaxScaler preprocessing.StandardScaler 行：preprocessing.Normalizer
分类与编码	preprocessing.OrdinalEncoder() preprocessing.OneHotEncoder()
数据离散化
多项式化	PolynomialFeatures(degree=2,include_bias=False)
缺失值处理

阅读全文 »

距离

发表于 2019-10-29 | 分类于数学基础，高数

距离公理

距离必须满足以下条件。设$X$是任一非空集，对$X$中任意两点$x,y$有一实数$d(x,y)$与之对应且满足：

正定性：非负性、唯一性。$d(x,y) \geq 0$ ，且 $d(x,y) = 0$当且仅当$x=y$
对称性：$d(x,y) =d(y,x)$
三角不等式：$d(x,y) \leq d(x,z)+d(y,z)$，意两边之和大于、等于第三边。

AB测

发表于 2019-10-28 | 分类于人工智能，评估

应用场景

A / B测试，即你设计的页面有两个版本（A和B），A为现行的设计， B是新的设计。比较这两个版本之间你所关心的数据（转化率，业绩，跳出率等）。A/B测试的目的就是尝试了解新的设计是否在统计上显著地改变了转化率。

流程

对用户进行分组，分为A组与B组。A组是对照组，B组是在A组的基础上有一些改变的实验组。

分组的重点是正交分流，也就是2组的人群分布一致。在实际操作过程中，经常会出现多组实验同时进行，为了避免实验相互干扰，保证每组实验的独立性；就应该保证每个实验过程中，实验组与对照组的人群分布一致，即正交分流。

例如淘宝App对用户登陆页面与用户注册页面都进行了修改（用户登陆页面—>跳转注册页面）。在对注册页面进行AB测试的过程中，需要保证AB两组中用户登陆页面的人群分布一致；如果A组中的用户80%来自登陆页面修改之后的用户，B组用户20%来自登陆页面修改之后的用户，这样很难确定是用户登陆页面还是注册页面对AB两组产生了影响。
使用显著性检验，观察A组与B组是否发生了在统计上发生了显著性的变化。从而判断B组的改变是否有效。这就是数学原理的关键所在。

数学原理

分流引擎

AB测副作用

探索性难题

多臂赌博机

terminal

发表于 2019-10-27 | 分类于 linux ， wsl

windows terminal

添加到右键

检查常量

1
2
3

echo %USERPROFILE%

echo %LOCALAPPDATA%

创建文件夹用来存放图标与注册表

创建文件夹terminal

1	mkdir "%USERPROFILE%\AppData\Local\Terminal"

存放图标

将terminal的图标存放在mkdir "%USERPROFILE%\AppData\Local\Terminal"中

添加注册表

创建 addwt.reg文件，在文件中写入以下内容，并使用管理员身份运行。

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\Background\shell\wt]
@="Windows Terminal here"
"Icon"="%USERPROFILE%\\AppData\\Local\\Terminal\\terminal.ico"

[HKEY_CLASSES_ROOT\Directory\Background\shell\wt\command]
@="C:\\Users\\[你的电脑用户名！你的电脑用户名！]\\AppData\\Local\\Microsoft\\WindowsApps\\wt.exe"

配置

{
    "$schema": "https://aka.ms/terminal-profiles-schema",
    "globals" : 
    {
        "alwaysShowTabs" : true,
        "copyOnSelect" : false,
        "defaultProfile" : "{c6eaf9f4-32a7-5fdc-b5cf-066e8a4b1e40}",     //默认终端
        "initialCols" : 120,
        "initialRows" : 30,
        "keybindings" : 
        [
            {
                "command" : "closePane",
                "keys" : 
                [
                    "ctrl+shift+w"
                ]
            },
            {
                "command" : "copy",
                "keys" : 
                [
                    "ctrl+shift+c"
                ]
            },
            {
                "command" : "duplicateTab",
                "keys" : 
                [
                    "ctrl+shift+d"
                ]
            },
            {
                "command" : "newTab",
                "keys" : 
                [
                    "ctrl+shift+t"
                ]
            },
            {
                "command" : "newTabProfile0",
                "keys" : 
                [
                    "ctrl+shift+1"
                ]
            },
            {
                "command" : "newTabProfile1",
                "keys" : 
                [
                    "ctrl+shift+2"
                ]
            },
            {
                "command" : "newTabProfile2",
                "keys" : 
                [
                    "ctrl+shift+3"
                ]
            },
            {
                "command" : "newTabProfile3",
                "keys" : 
                [
                    "ctrl+shift+4"
                ]
            },
            {
                "command" : "newTabProfile4",
                "keys" : 
                [
                    "ctrl+shift+5"
                ]
            },
            {
                "command" : "newTabProfile5",
                "keys" : 
                [
                    "ctrl+shift+6"
                ]
            },
            {
                "command" : "newTabProfile6",
                "keys" : 
                [
                    "ctrl+shift+7"
                ]
            },
            {
                "command" : "newTabProfile7",
                "keys" : 
                [
                    "ctrl+shift+8"
                ]
            },
            {
                "command" : "newTabProfile8",
                "keys" : 
                [
                    "ctrl+shift+9"
                ]
            },
            {
                "command" : "nextTab",
                "keys" : 
                [
                    "ctrl+tab"
                ]
            },
            {
                "command" : "openNewTabDropdown",
                "keys" : 
                [
                    "ctrl+shift+space"
                ]
            },
            {
                "command" : "openSettings",
                "keys" : 
                [
                    "ctrl+,"
                ]
            },
            {
                "command" : "paste",
                "keys" : 
                [
                    "ctrl+shift+v"
                ]
            },
            {
                "command" : "prevTab",
                "keys" : 
                [
                    "ctrl+shift+tab"
                ]
            },
            {
                "command" : "scrollDown",
                "keys" : 
                [
                    "ctrl+shift+down"
                ]
            },
            {
                "command" : "scrollDownPage",
                "keys" : 
                [
                    "ctrl+shift+pgdn"
                ]
            },
            {
                "command" : "scrollUp",
                "keys" : 
                [
                    "ctrl+shift+up"
                ]
            },
            {
                "command" : "scrollUpPage",
                "keys" : 
                [
                    "ctrl+shift+pgup"
                ]
            },
            {
                "command" : "switchToTab0",
                "keys" : 
                [
                    "ctrl+alt+1"
                ]
            },
            {
                "command" : "switchToTab1",
                "keys" : 
                [
                    "ctrl+alt+2"
                ]
            },
            {
                "command" : "switchToTab2",
                "keys" : 
                [
                    "ctrl+alt+3"
                ]
            },
            {
                "command" : "switchToTab3",
                "keys" : 
                [
                    "ctrl+alt+4"
                ]
            },
            {
                "command" : "switchToTab4",
                "keys" : 
                [
                    "ctrl+alt+5"
                ]
            },
            {
                "command" : "switchToTab5",
                "keys" : 
                [
                    "ctrl+alt+6"
                ]
            },
            {
                "command" : "switchToTab6",
                "keys" : 
                [
                    "ctrl+alt+7"
                ]
            },
            {
                "command" : "switchToTab7",
                "keys" : 
                [
                    "ctrl+alt+8"
                ]
            },
            {
                "command" : "switchToTab8",
                "keys" : 
                [
                    "ctrl+alt+9"
                ]
            }
        ],
        "requestedTheme" : "system",
        "showTabsInTitlebar" : true,
        "showTerminalTitleInTitlebar" : true,
        "wordDelimiters" : " ./\\()\"'-:,.;<>~!@#$%^&*|+=[]{}~?\u2502"
    },
    "profiles" : 
    [
        {                           
            "acrylicOpacity" : 0.75,                    // 亚克力效果透明度
        //    "background" : "#123456",                   // 背景颜色
            "backgroundImage" : "C://terminalpicture//powershell.png", // 背景图片地址，路径中不能有中文
            "backgroundImageOpacity" : 0.3,   // 透明度
            "backgroundImageStretchMode" : "uniformToFill",
            "closeOnExit" : true,
            "colorScheme" : "Campbell",                // 配色方案
            "commandline" : "powershell.exe",          // 打开终端的命令
            "cursorColor" : "#FFFFFF",                 // 游标的颜色
            "cursorShape" : "bar",
            "fontFace" : "Consolas",
            "fontSize" : 10,                           // 字体大小
            "guid" : "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
            "historySize" : 9001,
            "icon" : "ms-appx:///ProfileIcons/{61c54bbd-c2c6-5271-96e7-009a87ff44bf}.png",    // 图标地址
            "name" : "Windows PowerShell",             // 终端的名称
            "padding" : "0, 0, 0, 0",
            "snapOnInput" : true,
            "startingDirectory" : "./",
            "useAcrylic" : false                       // 使用亚克力效果
        },
        {
            "acrylicOpacity" : 0.75,
          //  "background" : "#0C0C0C",                  // 背景颜色
            "backgroundImage" : "C://terminalpicture//cmd.jpg",       // 背景图片地址
            "backgroundImageOpacity" : 0.9,            // 透明度
            "backgroundImageStretchMode" : "uniformToFill",
            "closeOnExit" : true,
            "colorScheme" : "Campbell",
            "commandline" : "cmd.exe",
            "cursorColor" : "#FFFFFF",
            "cursorShape" : "bar",
            "fontFace" : "Consolas",
            "fontSize" : 10,
            "guid" : "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}",
            "historySize" : 9001,
            "icon" : "ms-appx:///ProfileIcons/{0caa0dad-35be-5f56-a8ff-afceeeaa6101}.png",
            "name" : "cmd",
            "padding" : "0, 0, 0, 0",
            "snapOnInput" : true,
            "startingDirectory" : "./",
            "useAcrylic" : false
        },
        {
            "acrylicOpacity" : 0.59999999999999998,
            "closeOnExit" : false,
            "colorScheme" : "Vintage",
            "commandline" : "Azure",
            "connectionType" : "{d9fcfdfa-a479-412c-83b7-c5640e61cd62}",
            "cursorColor" : "#FFFFFF",
            "cursorShape" : "bar",
            "fontFace" : "Consolas",
            "fontSize" : 10,
            "guid" : "{b453ae62-4e3d-5e58-b989-0a998ec441b8}",
            "historySize" : 9001,
            "icon" : "ms-appx:///ProfileIcons/{b453ae62-4e3d-5e58-b989-0a998ec441b8}.png",
            "name" : "Azure Cloud Shell",
            "padding" : "0, 0, 0, 0",
            "snapOnInput" : true,
            "startingDirectory" : "./",
            "useAcrylic" : true
        },
        {
            "acrylicOpacity" : 0.5,
            "background" : "#0C0C0C",                   // 背景颜色
            "backgroundImage" : "C://terminalpicture//ubuntu.jpg",       // 背景图片地址
            "backgroundImageOpacity" : 0.3,             // 透明度
            "backgroundImageStretchMode" : "uniformToFill",
            "closeOnExit" : true,
            "colorScheme" : "Campbell",
            "commandline" : "wsl.exe -d Ubuntu-18.04",
            "cursorColor" : "#FFFFFF",
            "cursorShape" : "bar",
            "fontFace" : "Consolas",
            "fontSize" : 10,
            "guid" : "{c6eaf9f4-32a7-5fdc-b5cf-066e8a4b1e40}",
            "historySize" : 9001,
            "icon" : "ms-appx:///ProfileIcons/{9acb9455-ca41-5af7-950f-6bca1bc9722f}.png",
            "name" : "Ubuntu-18.04",
            "padding" : "0, 0, 0, 0",
            "snapOnInput" : true,
            "startingDirectory" : "./",
            "useAcrylic" : false
        }
    ],
    "schemes" : 
    [
        {
            "background" : "#0C0C0C",
            "black" : "#0C0C0C",
            "blue" : "#0037DA",
            "brightBlack" : "#767676",
            "brightBlue" : "#3B78FF",
            "brightCyan" : "#61D6D6",
            "brightGreen" : "#16C60C",
            "brightPurple" : "#B4009E",
            "brightRed" : "#E74856",
            "brightWhite" : "#F2F2F2",
            "brightYellow" : "#F9F1A5",
            "cyan" : "#3A96DD",
            "foreground" : "#CCCCCC",
            "green" : "#13A10E",
            "name" : "Campbell",
            "purple" : "#881798",
            "red" : "#C50F1F",
            "white" : "#CCCCCC",
            "yellow" : "#C19C00"
        },
        {
            "background" : "#000000",
            "black" : "#000000",
            "blue" : "#000080",
            "brightBlack" : "#808080",
            "brightBlue" : "#0000FF",
            "brightCyan" : "#00FFFF",
            "brightGreen" : "#00FF00",
            "brightPurple" : "#FF00FF",
            "brightRed" : "#FF0000",
            "brightWhite" : "#FFFFFF",
            "brightYellow" : "#FFFF00",
            "cyan" : "#008080",
            "foreground" : "#C0C0C0",
            "green" : "#008000",
            "name" : "Vintage",
            "purple" : "#800080",
            "red" : "#800000",
            "white" : "#C0C0C0",
            "yellow" : "#808000"
        },
        {
            "background" : "#282C34",
            "black" : "#282C34",
            "blue" : "#61AFEF",
            "brightBlack" : "#5A6374",
            "brightBlue" : "#61AFEF",
            "brightCyan" : "#56B6C2",
            "brightGreen" : "#98C379",
            "brightPurple" : "#C678DD",
            "brightRed" : "#E06C75",
            "brightWhite" : "#DCDFE4",
            "brightYellow" : "#E5C07B",
            "cyan" : "#56B6C2",
            "foreground" : "#DCDFE4",
            "green" : "#98C379",
            "name" : "One Half Dark",
            "purple" : "#C678DD",
            "red" : "#E06C75",
            "white" : "#DCDFE4",
            "yellow" : "#E5C07B"
        },
        {
            "background" : "#FAFAFA",
            "black" : "#383A42",
            "blue" : "#0184BC",
            "brightBlack" : "#4F525D",
            "brightBlue" : "#61AFEF",
            "brightCyan" : "#56B5C1",
            "brightGreen" : "#98C379",
            "brightPurple" : "#C577DD",
            "brightRed" : "#DF6C75",
            "brightWhite" : "#FFFFFF",
            "brightYellow" : "#E4C07A",
            "cyan" : "#0997B3",
            "foreground" : "#383A42",
            "green" : "#50A14F",
            "name" : "One Half Light",
            "purple" : "#A626A4",
            "red" : "#E45649",
            "white" : "#FAFAFA",
            "yellow" : "#C18301"
        },
        {
            "background" : "#002B36",
            "black" : "#073642",
            "blue" : "#268BD2",
            "brightBlack" : "#002B36",
            "brightBlue" : "#839496",
            "brightCyan" : "#93A1A1",
            "brightGreen" : "#586E75",
            "brightPurple" : "#6C71C4",
            "brightRed" : "#CB4B16",
            "brightWhite" : "#FDF6E3",
            "brightYellow" : "#657B83",
            "cyan" : "#2AA198",
            "foreground" : "#839496",
            "green" : "#859900",
            "name" : "Solarized Dark",
            "purple" : "#D33682",
            "red" : "#DC322F",
            "white" : "#EEE8D5",
            "yellow" : "#B58900"
        },
        {
            "background" : "#FDF6E3",
            "black" : "#073642",
            "blue" : "#268BD2",
            "brightBlack" : "#002B36",
            "brightBlue" : "#839496",
            "brightCyan" : "#93A1A1",
            "brightGreen" : "#586E75",
            "brightPurple" : "#6C71C4",
            "brightRed" : "#CB4B16",
            "brightWhite" : "#FDF6E3",
            "brightYellow" : "#657B83",
            "cyan" : "#2AA198",
            "foreground" : "#657B83",
            "green" : "#859900",
            "name" : "Solarized Light",
            "purple" : "#D33682",
            "red" : "#DC322F",
            "white" : "#EEE8D5",
            "yellow" : "#B58900"
        }
    ]
}

anaconda与conda

发表于 2019-10-25 | 分类于编程语言， python

conda环境管理

conda info --envs     #查看环境
conda create -n my_env_name python=3.6  #创建环境
conda remove -n my_env_name --all #删除环境
source activate my_env_name #激活环境
source deactivate my_env_name #关闭环境

linux软件安装原理

发表于 2019-09-26 | 分类于 linux ，环境

在linux下所有的东西都是以“文件”的形式表示的，那些可以运行的程序有2种，

脚本文件，脚本文件由解释程序执行，一般有shell脚本、perl脚本、python脚本等等。
二进制文件，也就是经过编译器编译、联接形成的只有0和1组成的文件（计算机只运行0和1组成的程序），c、java等程序都是这种程序。

因此，只要某个程序所需要的全部“文件”都存在于正确的位置上，那么这个程序就可以运行。（这个现象在windows下是不全部适用的，如果你复制某个缺少的文件到windows的系统目录，并不一定能使你需要的程序运行起来。）对程序而言，还有一些是以纯文本形式存在的配置文件，用户可以通过定制配置文件来控制程序的运行结果等等。

阅读全文 »

身体部位

发表于 2019-09-21

肩部

t-SNE

发表于 2019-09-19 | 分类于人工智能，特征工程

t-SNE 是一种非线性降维算法，非常适用于高维数据降维到2维或者3维，进行可视化。

http://www.datakit.cn/blog/2017/02/05/t_sne_full.html

pytest

发表于 2019-09-18 | 分类于编程语言， python

目录结构

应用

├── feature        
    ├── in_fc.py             #开发文件
    └── test                 #测试目录
        ├── __init__.py      #子目录需要添加
        └── test_in_fc.py    #测试

阅读全文 »

ssh

发表于 2019-09-03 | 分类于 linux ，远程登陆

生成ssh密钥对

1	ssh-keygen -t rsa -C 'xxx@xxx.com'

按3次回车（不使用密码）生成下图。密钥对在~/.ssh文件夹中,将id_rsa.pub（公钥）提交到github或者gitlab中

1568460515778

安装ssh

1 2	sudo apt-get install openssh-client #客户端类似于xshell，登陆服务器 sudo apt-get install openssh-server #服务端用于服务器，让其他客户端远程登陆

远程登陆服务器

shell脚本远程自动登陆服务器

目前远程登陆服务器一般需要1. 登陆跳板机；2. 登陆服务器。

ssh跳板机->输入静态密码+token码
在跳板机ssh服务器->输入密码

解决方法

1	sudo apt-get install expect #安装expect

在expect中写入，不同公司的登陆流程不一样，需要修改登陆流程。但是登陆模板如图所示。其中set是设置变量；expect是设置返回的字符串；send是发送的字符串。

#!/usr/bin/expect

set salt [lindex $argv 0]   #跳板机的token码
set username 跳板机地址
set password 跳板机静态密码
set serverIP 服务器IP地址
set server 服务器地址
set serverpass  服务器密码
spawn ssh "$username"
expect {
    "*yes/no*" {            #如果遇到yes/no输入yes
        send "yes\n";
        exp_continue;
    }
    "*Password:" {          #如果遇到Password
        send "$password $salt\n";    #静态密码+' '+token密码
        exp_continue;
    }
    "Please input server keyword" {  #输入服务器IP地址
        send $serverIP;
        exp_continue;
    }
    "Select account" {               #选择账户
        send "powerop\r";
        exp_continue;
    }
    "server" {                       #从跳板机ssh服务器    
        send "ssh $server\r";
        exp_continue;
    }
    "*password:" {                   #输入服务器密码
        send "$serverpass\r";
        exp_continue;
    }
}
interact