• Data competition: From 0 to 1: Part I
  • 1. Data competition Introduction
  • 2. Example: Credit Fraud Detector
    • EDA(Exploratory Data Analysis)
      • Why taking log transformation of continuous variables?
    • Outliers Detection
    • Unbalance
    • Metrics
    • Resampling
    • Cross-validation: evaluating estimator performance
      • Cross validation iterators
      • Cross-validation iterators for grouped data
      • Cross validation of time series data
    • Modeling

Data competition: From 0 to 1: Part I

1. Data competition Introduction

A typical data science process might look like this:

• Project Scoping / Data Collection
• Exploratory Analysis
• Data Cleaning
• Feature Engineering
• Model Training (including cross-validation to tune hyper-parameters)
• Project Delivery / Insights
  

2. Example: Credit Fraud Detector

Competition Describe

• The datasets contains transactions made by credit cards in September 2013 by european cardholders.
• Features V1, V2, ... V28 are the principal components obtained with PCA
• Feature ‘Time‘ contains the seconds elapsed between each transaction and the first transaction in the dataset.
• The feature ‘Amount’ is the transaction Amount.
• Feature ‘Class‘ is the response variable and it takes value 1 in case of fraud and 0 otherwise.

EDA(Exploratory Data Analysis)

• Gathering a basic sense of data
• Example:
  • Shape(Traing Size, Test size)
  • Label(Binary or Multi or Regression, Distribution)
  • Columns(Meaning, Numerical or Time or Category)
  • Null Values, how to deal with
  • Numerical variable: Distribution
  • Outliers

# import library
import pandas as pd pd.set_option(‘display.max_column‘,100) import matplotlib.pyplot as plt from pylab import rcParams rcParams[‘figure.figsize‘] = 14, 10 # plt.figure(figsize=(14, 10)) import numpy as np import seaborn as sns %matplotlib inline # 可以省略 plt.show() import warnings warnings.filterwarnings(‘ignore‘) 

 

# simple ead ways
data = pd.read_csv(‘data/creditcard.csv‘) data.head() data.shape data.info() data.columns data.describe() data[‘Class‘].value_counts() data.isnull().sum() data.isnull().sum().max() # 包含缺失最多的一列有多少 data.isnull().any() data.isnull().values.any() # 数据中是否有缺失值 

 

# Amount-histogram
Fraud_transacation = data[data["Class"]==1] Normal_transacation= data[data["Class"]==0] plt.figure(figsize=(10,6)) plt.subplot(121) Fraud_transacation.Amount.plot.hist(title="Fraud Transacation") plt.yscale(‘log‘) plt.subplot(122) Normal_transacation.Amount.plot.hist(title="Normal Transaction") plt.yscale(‘log‘) # Amount-Time f, (ax1, ax2) = plt.subplots(2, 1, sharex=True) f.suptitle(‘Time of transaction vs Amount by class‘) ax1.scatter(Fraud.Time, Fraud.Amount) ax1.set_title(‘Fraud‘) ax2.scatter(Normal.Time, Normal.Amount) ax2.set_title(‘Normal‘) plt.xlabel(‘Time (in Seconds)‘) plt.ylabel(‘Amount‘) plt.show() # correlation correlation_matrix = data.corr() fig = plt.figure(figsize=(12,9)) sns.heatmap(correlation_matrix,vmax=0.8,square = True) plt.show() ## After sampling ? 

data[‘hour‘] = data[‘Time‘].apply(lambda x: np.ceil(float(x)/3600) % 24) data.pivot_table(values=‘Amount‘,index=‘hour‘,columns=‘Class‘,aggfunc=‘count‘) # https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot_table.html bins = np.arange(data[‘hour‘].min(),data[‘hour‘].max()+2) # plt.figure(figsize=(15,4)) sns.distplot(data[data[‘Class‘]==0.0][‘hour‘], norm_hist=True, bins=bins, kde=False, color=‘b‘, hist_kws={‘alpha‘:.5}, label=‘Legit‘) sns.distplot(data[data[‘Class‘]==1.0][‘hour‘], norm_hist=True, bins=bins, kde=False, color=‘r‘, label=‘Fraud‘, hist_kws={‘alpha‘:.5}) plt.xticks(range(0,24)) plt.legend() 

# https://github.com/pandas-profiling/pandas-profiling
import pandas_profiling profile = df.profile_report(title="Credit Fraud Detector") profile.to_file(output_file=Path("./credit_fraud_detector.html")) 

EDA Conclusion

• The data set is highly skewed, consisting of 492 frauds in a total of 284,807 observations. This resulted in only 0.172% fraud cases.
• There is no missing value in the dataset.
• The ‘Time’ and ‘Amount’ features are not transformed data.

Why taking log transformation of continuous variables?

Amount distribution is "power law", meaning that the vast majority of amount are small and very few are big.

\begin{array}{c}{\log \left(10^{4}\right)=4 * \log (10)} \\ {\log \left(10^{3}\right)=3 * \log (10)} \\ {10^{4}-10^{3}} \\ {4-3}\end{array}log(104)=4∗log(10)log(103)=3∗log(10)104−1034−3?

which transforms a huge difference in a smaller one. Logarithm naturally reduces the dynamic range of a variable so the differences are preserved while the scale is not that dramatically skewed.

Outliers Detection

• Point anomalies: A single instance of data is anomalous if it‘s too far off from the rest.
• Contextual anomalies: The abnormality is context specific. This type of anomaly is common in time-series data.

\mathrm{IQR}=Q_{3}-Q_{1}IQR=Q3?−Q1?

\text{Outliers}: >Q_3+k \cdot IQROutliers:>Q3?+k⋅IQR

\text{Outliers}: <Q_1-k \cdot IQROutliers:<Q1?−k⋅IQR

The higher kk is (ex: 3), the less outliers will detect, and the lower kk is (ex: 1.5) the more outliers it will detect.

We want to focus more on "extreme outliers" rather than just outliers.

Unbalance

count_classes = pd.value_counts(df[‘Class‘], sort = True).sort_index() count_classes.plot(kind = ‘bar‘) plt.title("Fraud class histogram") plt.xlabel("Class") plt.ylabel("Frequency") 

1. Collect more data
2. Using the weights parameters

LogisticRegression(class_weight=‘balanced‘) # How to choose weights? 

3. Changing the performance metric:
   2.1 Precision, Recall
   2.2 F1-score
   2.3 ROC curves
4. Resampling the dataset
   3.1 OVER-sampling
   3.2 UNDER-sampling
   3.3 SMOTE

Metrics

Confusion Matrix

• True Positives : The cases in which we predicted YES and the actual output was also YES.
• True Negatives : The cases in which we predicted NO and the actual output was NO.
• False Positives : The cases in which we predicted YES and the actual output was NO.
• False Negatives : The cases in which we predicted NO and the actual output was YES.
  

## homework ##
def find_TP(y_true, y_pred): # counts the number of true positives (y_true = 1, y_pred = 1) return sum((y_true == 1) & (y_pred == 1)) def find_FN(y_true, y_pred): # counts the number of false negatives (y_true = 1, y_pred = 0) return # your code here def find_FP(y_true, y_pred): # counts the number of false positives (y_true = 0, y_pred = 1) return # your code here def find_TN(y_true, y_pred): # counts the number of true negatives (y_true = 0, y_pred = 0) return # your code here import numpy as np def find_conf_matrix_values(y_true,y_pred): # calculate TP, FN, FP, TN TP = find_TP(y_true,y_pred) FN = find_FN(y_true,y_pred) FP = find_FP(y_true,y_pred) TN = find_TN(y_true,y_pred) return TP,FN,FP,TN def my_confusion_matrix(y_true, y_pred): TP,FN,FP,TN = find_conf_matrix_values(y_true,y_pred) return np.array([[TN,FP],[FN,TP]]) 

from sklearn.metrics import confusion_matrix confusion_matrix(label, prediction) 

Accuracy
\text{Accuracy}=\frac{\text{True Positives}+\text{False Negatives}}{\text{Total Number of Predictions}}Accuracy=Total Number of PredictionsTrue Positives+False Negatives?

• It works well only if there are equal number of samples belonging to each class.

# homework
def my_accuracy_score(y_true, y_pred): # calculates the fraction of samples predicted correctly TP,FN,FP,TN = find_conf_matrix_values(y_true,y_pred) return # your code here 

from sklearn.metrics import accuracy_score accuracy_score(label, prediction) 

Precision
\text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Positives}}Precision=True Positives+False PositivesTrue Positives?

# homework
def my_precision_score(y_true, y_pred): # calculates the fraction of predicted positives samples that are actually positive TP,FN,FP,TN = find_conf_matrix_values(y_true,y_pred) return # your code here 

from sklearn.metrics import precision_score precision_score(label, prediction) 

Recall
\text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}Recall=True Positives+False NegativesTrue Positives?

# homework
def my_recall_score(y_true, y_pred): # calculates the fraction of positive samples predicted correctly TP,FN,FP,TN = find_conf_matrix_values(y_true,y_pred) return # your code here 

from sklearn.metrics import recall_score recall_score(label, prediction) 

F1 Score
The f1 score is the harmonic mean(调和平均) of recall and precision, with a higher score as a better model.
F 1=\frac{2}{\frac{1}{\text { precision }}+\frac{1}{\text { recall }}}=\frac{2 * \text { (precision * recall) }}{\text { precision }+\text {recall}}F1= precision 1?+ recall 1?2?= precision +recall2∗ (precision * recall) ?

# homework
def my_f1_score(y_true, y_pred): # calculates the F1 score recall = my_recall_score(y_true,y_pred) precision = my_precision_score(y_true,y_pred) return # your code here 

from sklearn.metrics import f1_score f1_score(label, prediction) 

We often assume that we defined a threshold of 0.5 for selecting which samples are predicted as positive. If we change this threshold the performance metrics will change It would be nice to be able to evaluate the performance of a model without the need to select an arbitrary threshold. This is precisely what AUC-ROC is providing.

Area Under Curve (AUC)--Area Under the curve of the Receiver Operating Characteristic (AUROC)

AUC is used for binary classification problem.

• True Positive Rate (Sensitivity) : the proportion of positive data points that are correctly considered as positive, with respect to all positive data points.\text{True Positive Rate}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}True Positive Rate=True Positives+False NegativesTrue Positives?
• False Positive Rate (Specificity) : the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.\text{False Positive Rate}=\frac{\text{False Positives}}{\text{False Positives}+\text{True Negatives}}False Positive Rate=False Positives+True NegativesFalse Positives?

FPR and TPR bot hare computed at threshold values such as (0.00, 0.02, 0.04, …., 1.00)(0.00,0.02,0.04,….,1.00) and a graph is drawn. AUC is the area under the curve of plot False Positive Rate vs True Positive Rate at different points in [0, 1][0,1]. The resulting curve is called ROC curve (Receiver Operating Characteristic curve).

from sklearn.metrics import roc_curve fpr, tpr, thresholds = roc_curve(label, prediction) import matplotlib.pyplot as plt plt.plot(fpr, tpr) plt.legend() plt.xlabel(‘False Positive Rate‘) plt.ylabel(‘True Positive Rate‘) plt.show() from sklearn.metrics import roc_auc_score auc = roc_auc_score(label, prediction) 

from numba import jit @jit def fast_auc(y_true, y_prob): y_true = np.asarray(y_true) y_true = y_true[np.argsort(y_prob)] nfalse = 0 auc = 0 n = len(y_true) for i in range(n): y_i = y_true[i] nfalse += (1 - y_i) auc += y_i * nfalse auc /= (nfalse * (n - nfalse)) return auc 

## test speed
y_true = np.random.randint(0,2,1000000) y_pred = np.random.rand(1000000) fast_auc(y_true, y_pred) from sklearn.metrics import roc_auc_score roc_auc_score(y_true, y_pred) %timeit fast_auc(y_true, y_pred) %timeit roc_auc_score(y_true, y_pred) 

You can learn more about ROC on the Wikipedia page and What does AUC stand for and what is it?.

Mean Absolute Error(MAE)
\text {MAE}=\frac{1}{N} \sum_{j=1}^{N}\left|y_{j}-\hat{y}_{j}\right|MAE=N1?j=1∑N?∣yj?−y^?j?∣

Mean Squared Error(MSE)
\text {MSE}=\frac{1}{N} \sum_{j=1}^{N}\left(y_{j}-\hat{y}_{j}\right)^{2}MSE=N1?j=1∑N?(yj?−y^?j?)2

Log Loss
AUC only takes into account the order of probabilities and hence it does not take into account the model’s capability to predict higher probability for samples more likely to be positive.

\text {Log Loss}=-\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{M} y_{i j} * \log \left(p(y_{i j})\right)Log Loss=−N1?i=1∑N?j=1∑M?yij?∗log(p(yij?))

• y_{ij}yij?: whether sample ii belongs to class jj or not
• p(y_{i j})p(yij?): the probability of sample ii belonging to class jj

\text {Log Loss}=-\frac{1}{N} \sum_{i=1}^{N} y_{i} \cdot \log \left(p\left(y_{i}\right)\right)+\left(1-y_{i}\right) \cdot \log \left(1-p\left(y_{i}\right)\right)Log Loss=−N1?i=1∑N?yi?⋅log(p(yi?))+(1−yi?)⋅log(1−p(yi?))

Resampling

Under-Sampling: samples from the majority class
Over-Sampling: adding more examples from the minority class

Under-Sampling Drawback: Removing information that may be valuable. This could lead to underfitting and poor generalization to the test set.

### Create A Small Unbalanced Sample Dataset
from sklearn.utils import resample from sklearn.datasets import make_classification X, y = make_classification( n_classes=2, class_sep=1.5, weights=[0.9, 0.1], n_informative=3, n_redundant=1, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10 ) df = pd.DataFrame(X) df[‘target‘] = y df.target.value_counts().plot(kind=‘bar‘, title=‘Count (target)‘) ### Oversample minority class minority_data = df[df.target==1] magority_data = df[df.target==0] # oversample minority upsampled_data = resample(minority_data, replace=True, # sample with replacement n_samples=len(magority_data), # match number in majority class random_state=27) # reproducible results # combine majority and upsampled minority upsampled = pd.concat([upsampled_data, magority_data]) # check new class counts upsampled.target.value_counts() ### Undersample majority class downsampled_data = resample(magority_data, replace = False, # sample without replacement n_samples = len(minority_data), # match minority n random_state = 27) # reproducible results # combine minority and downsampled majority downsampled = pd.concat([minority_data, downsampled_data]) # checking counts downsampled.target.value_counts() 

Imbalanced-Learn

def plot_2d_space(X_train, y_train,X=X,y=y ,label=‘Classes‘): colors = [‘#1F77B4‘, ‘#FF7F0E‘] markers = [‘o‘, ‘s‘] fig,(ax1,ax2)=plt.subplots(1,2, figsize=(10,5)) for l, c, m in zip(np.unique(y), colors, markers): ax1.scatter( X_train[y_train==l, 0], X_train[y_train==l, 1], c=c, label=l, marker=m ) for l, c, m in zip(np.unique(y), colors, markers): ax2.scatter( X[y==l, 0], X[y==l, 1], c=c, label=l, marker=m ) ax1.set_title(label) ax2.set_title(‘original data‘) plt.legend(loc=‘upper right‘) plt.show() import imblearn ### Random under-sampling with imbalanced-learn from imblearn.under_sampling import RandomUnderSampler ran = RandomUnderSampler(return_indices=True) ##intialize to return indices of dropped rows X_rs,y_rs,dropped = ran.fit_sample(X,y) print("The number of removed indices are ",len(dropped)) plot_2d_space(X_rs, y_rs, X, y,‘Random under sampling‘) ### Random over-sampling with imbalanced-learn from imblearn.over_sampling import RandomOverSampler ran=RandomOverSampler() X_ran,y_ran= ran.fit_resample(X,y) print(‘The new data contains {} rows ‘.format(X_ran.shape[0])) plot_2d_space(X_ran,y_ran,X,y,‘over-sampled‘) 

Under-sampling: Tomek links
Tomek links are pairs of very close instances, but of opposite classes. Removing the instances of the majority class of each pair increases the space between the two classes, facilitating the classification process.

from imblearn.under_sampling import TomekLinks tl = TomekLinks(return_indices=True, ratio=‘majority‘) X_tl, y_tl, id_tl = tl.fit_sample(X, y) #print(‘Removed indexes:‘, id_tl) plot_2d_space(X_tl, y_tl,X,y, ‘Tomek links under-sampling‘) 

Over-sampling: SMOTE
SMOTE (Synthetic Minority Oversampling TEchnique) consists of synthesizing elements for the minority class, based on those that already exist. It works randomly picingk a point from the minority class and computing the k-nearest neighbors for this point. The synthetic points are added between the chosen point and its neighbors.

from imblearn.over_sampling import SMOTE smote = SMOTE(ratio=‘minority‘) X_sm, y_sm = smote.fit_sample(X, y) plot_2d_space(X_sm, y_sm,X,y, ‘SMOTE over-sampling‘) 

Cross-validation: evaluating estimator performance

Trained model would have a perfect score on training data but would fail to predict anything useful on yet-unseen data. This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set X_test, y_test. Here is a flowchart of typical cross validation workflow in model training.

k-folds:

• A model is trained using k-1 of the folds as training data;
• the resulting model is validated on the remaining part of the data.

The performance measure reported by k-fold cross-validation is then the average of the values computed in the loop. This approach can be computationally expensive, but does not waste too much data.

Cross validation iterators

Cross-validation iterators for i.i.d. data
K-fold

from sklearn.model_selection import KFold X = ["a", "b", "c", "d"] kf = KFold(n_splits=2) for train, test in kf.split(X): print("%s %s" % (train, test)) 

Random permutations cross-validation a.k.a. Shuffle & Split

from sklearn.model_selection import ShuffleSplit X = np.arange(10) ss = ShuffleSplit(n_splits=5, test_size=0.25, random_state=0) for train_index, test_index in ss.split(X): print("%s %s" % (train_index, test_index)) 

Cross-validation iterators with stratification based on class labels
Some classification problems can exhibit a large imbalance in the distribution of the target classes.

Stratified k-fold

from sklearn.model_selection import StratifiedKFold X = np.ones(10) y = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1] skf = StratifiedKFold(n_splits=3) for train, test in skf.split(X, y): print("%s %s" % (train, test)) 

Stratified Shuffle Split

Cross-validation iterators for grouped data

The i.i.d. assumption is broken if the underlying generative process yield groups of dependent samples.

Group k-fold
GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.

from sklearn.model_selection import GroupKFold X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 8.8, 9, 10] y = ["a", "b", "b", "b", "c", "c", "c", "d", "d", "d"] groups = [1, 1, 1, 2, 2, 2, 3, 3, 3, 3] gkf = GroupKFold(n_splits=3) for train, test in gkf.split(X, y, groups=groups): print("%s %s" % (train, test)) 

Group Shuffle Split

from sklearn.model_selection import GroupShuffleSplit X = [0.1, 0.2, 2.2, 2.4, 2.3, 4.55, 5.8, 0.001] y = ["a", "b", "b", "b", "c", "c", "c", "a"] groups = [1, 1, 2, 2, 3, 3, 4, 4] gss = GroupShuffleSplit(n_splits=4, test_size=0.5, random_state=0) for train, test in gss.split(X, y, groups=groups): print("%s %s" % (train, test)) 

Leave P Groups Out

from sklearn.model_selection import LeavePGroupsOut X = np.arange(6) y = [1, 1, 1, 2, 2, 2] groups = [1, 1, 2, 2, 3, 3] lpgo = LeavePGroupsOut(n_groups=2) for train, test in lpgo.split(X, y, groups=groups): print("%s %s" % (train, test)) 

Cross validation of time series data

k-fold 假设数据是 iid 的，但是时间序列存在相关性，因此不能简单使用 k-fold。

from sklearn.model_selection import TimeSeriesSplit X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]]) y = np.array([1, 2, 3, 4, 5, 6]) tscv = TimeSeriesSplit(n_splits=3) for train, test in tscv.split(X): print("%s %s" % (train, test)) 

Modeling

from sklearn.preprocessing import StandardScaler data[‘normAmount‘] = StandardScaler().fit_transform(data[‘Amount‘].reshape(-1, 1)) data = data.drop([‘Amount‘],axis=1) 

# train_test_split
from sklearn.cross_validation import train_test_split # Whole dataset X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.3, random_state = 0) from sklearn.linear_model import LogisticRegression lr = LogisticRegression(C = c_param, penalty = ‘l1‘) lr.fit(X_train, y_train) y_pred = lr.predict(X_test) # compare y_test with y_pred