Use cmd to get the terminal
Then type SET to get the environmental variables
C:\Users\AJAY>SET
Month: July 2017
test and control split prior to modeling in SAS language #sasstats
https://github.com/decisionstats/sas-for-data-science/blob/master/split%20test%20and%20control.sas
| data cars; |
| set sashelp.cars; |
| run; |
| data cars2; |
| set sashelp.cars; |
| where ranuni(12) <=.25; |
| run; |
|
data cars2;
set sashelp.cars;
where ranuni(12) <=.25;
run;
NOTE: There were 114 observations read from the data set SASHELP.CARS.
WHERE RANUNI(12)<=0.25;
NOTE: The data set WORK.CARS2 has 114 observations and 15 variables.
|
| data cars3 cars4; |
| set sashelp.cars; |
| if ranuni(12)<=.25 then output cars3; |
| else output cars4; |
| run; |
|
data cars3 cars4;
set sashelp.cars;
if ranuni(12)<=.25 then output cars3;
else output cars4;
run;
NOTE: There were 428 observations read from the data set SASHELP.CARS.
NOTE: The data set WORK.CARS3 has 114 observations and 15 variables.
NOTE: The data set WORK.CARS4 has 314 observations and 15 variables.
|
| PROC SURVEYSELECT DATA=cars OUT=test_cars METHOD=srs SAMPRATE=0.25; |
| RUN; |
| PROC SURVEYSELECT DATA=cars outall OUT=test_cars2 METHOD=srs SAMPRATE=0.25; |
| RUN; |
|
PROC SURVEYSELECT DATA=cars OUT=test_cars METHOD=srs SAMPRATE=0.25;
RUN;
NOTE: The data set WORK.TEST_CARS has 107 observations and 15 variables.
NOTE: PROCEDURE SURVEYSELECT used (Total process time):
PROC SURVEYSELECT DATA=cars outall OUT=test_cars2 METHOD=srs SAMPRATE=0.25;
RUN;
NOTE: The data set WORK.TEST_CARS2 has 428 observations and 16 variables.
NOTE: PROCEDURE SURVEYSELECT used (Total process time):
|
| proc print data=test_cars2 (obs=6); |
| var selected; |
| run; |
| proc freq data=test_cars2; |
| tables selected/norow nocol nocum nopercent; |
| run; |
| data test ; |
| set test_cars2; |
| where selected=0 ; |
| run; |
| data control ; |
| set test_cars2; |
| where selected=1 ; |
| run; |
Output
Automated Machine Learning using Python TPot
Python Tpot offers automated machine learning.
From
Click to access olson_tpot_2016.pdf
Machine learning is commonly described as a field of study that gives computers the ability to learn without being explicitly programmed”(Simon, 2013). Despite this common claim, machine learning practitioners know that designing effective machine learning pipelines is often a tedious endeavour, and typically requires considerable experience with machine learning algorithms, expert knowledge of the problem domain, and brute force search to accomplish (Olson et al., 2016a). Thus, contrary to what machine learning enthusiasts would have us believe, machine learning still requires considerable explicit programming. In response to this challenge, several automated machine learning methods have been developed over the years (Hutter et al., 2015).Over the past year, we have been developing a Tree-based Pipeline Optimization Tool (TPOT) that automatically designs and optimizes machine learning pipelines for a given problem domain (Olson et al., 2016b), without any need for human intervention. In short, TPOT optimizes machine learning pipelines using a version of genetic programming (GP)
TPOT is built on top of scikit-learn, so all of the code it generates should look familiar… if you’re familiar with scikit-learn, anyway.
and
Citation: Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science
Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, and Jason H. Moore (2016). Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science. Proceedings of GECCO 2016, pages 485-492
Decision Trees in R simplified
library(rattle)## Rattle: A free graphical interface for data mining with R.
## Version 4.1.0 Copyright (c) 2006-2015 Togaware Pty Ltd.
## Type 'rattle()' to shake, rattle, and roll your data.library(party)## Loading required package: grid## Loading required package: mvtnorm## Loading required package: modeltools## Loading required package: stats4## Loading required package: strucchange## Loading required package: zoo##
## Attaching package: 'zoo'## The following objects are masked from 'package:base':
##
## as.Date, as.Date.numeric## Loading required package: sandwichdata(iris)
model=ctree(Species~.,data = iris)
plot(model)
table(predict(model),iris$Species)##
## setosa versicolor virginica
## setosa 50 0 0
## versicolor 0 49 5
## virginica 0 1 45library(rpart) model2=rpart(Species~.,data = iris) fancyRpartPlot(model2)
library(randomForest)## randomForest 4.6-12## Type rfNews() to see new features/changes/bug fixes.model3=randomForest(Species~.,data = iris)
table(predict(model3),iris$Species)##
## setosa versicolor virginica
## setosa 50 0 0
## versicolor 0 47 4
## virginica 0 3 46K Means Clustering in Python
from https://github.com/decisionstats/pythonfordatascience/blob/master/2%2BClustering%2B-K%2BMeans.ipynb
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.cluster import KMeans
import sklearn.metrics as sm
import pandas as pd
import numpy as np
In [2]:
wine=pd.read_csv("http://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data",header=None)
In [3]:
wine.head()
Out[3]:
In [4]:
wine.columns=['winetype','Alcohol','Malic acid','Ash','Alcalinity of ash','Magnesium','Total phenols','Flavanoids','Nonflavanoid phenols','Proanthocyanins','Color intensity','Hue','OD280/OD315 of diluted wines','Proline']
In [5]:
wine.head()
Out[5]:
In [6]:
wine.info()
In [7]:
wine.describe()
Out[7]:
In [8]:
pd.value_counts(wine['winetype'])
Out[8]:
In [9]:
x=wine.ix[:,1:14]
y=wine.ix[:,:1]
In [10]:
x.columns
Out[10]:
In [11]:
y.columns
Out[11]:
In [12]:
x.head()
Out[12]:
In [13]:
y.head()
Out[13]:
In [14]:
# K Means Cluster
model = KMeans(n_clusters=3)
model.fit(x)
Out[14]:
In [15]:
model.labels_
Out[15]:
In [16]:
pd.value_counts(model.labels_)
Out[16]:
In [17]:
pd.value_counts(y['winetype'])
Out[17]:
In [18]:
# We convert all the 1s to 0s and 0s to 1s.
predY = np.choose(model.labels_, [2, 1, 3]).astype(np.int64)
In [19]:
pd.value_counts(y['winetype'])
Out[19]:
In [20]:
pd.value_counts(model.labels_)
Out[20]:
In [21]:
pd.value_counts(predY)
Out[21]:
In [22]:
# Performance Metrics
sm.accuracy_score(y, predY)
Out[22]:
In [23]:
# Confusion Matrix
sm.confusion_matrix(y, predY)
Out[23]:
In [24]:
from ggplot import *
%matplotlib inline
In [25]:
p = ggplot(aes(x='Alcohol', y='Ash',color="winetype"), data=wine)
p + geom_point()
Out[25]:
In [26]:
p2 = ggplot(aes(x='Alcohol', y='Ash',color="predY"), data=wine)
p2 + geom_point()
Out[26]:
Connecting Python with a RDBMS (Postgres)
In [1]:
import psycopg2
import pandas as pd
import sqlalchemy as sa
import time
import seaborn as sns
import re
In [2]:
!pip install psycopg2
In [3]:
parameters = {
'username': 'postgres',
'password': 'root',
'server': 'localhost',
'database': 'datascience'
}
In [4]:
connection= 'postgresql://{username}:{password}@{server}:5432/{database}'.format(**parameters)
In [5]:
print (connection)
In [6]:
engine = sa.create_engine(connection, encoding="utf-8")
In [7]:
insp = sa.inspect(engine)
db_list = insp.get_schema_names()
print(db_list)
In [8]:
print(insp)
In [9]:
engine.table_names()
Out[9]:
In [11]:
data3= pd.read_sql_query('select * from "sales77" limit 10',con=engine)
In [12]:
print(data3)
DECISION STATS 