Describe how the multi-class classification is different for SVC and LinearSVC. Be explicit, don't just describe what's in the documentation. For example, what does 'one-against-one' and 'one-vs-the-rest' mean?
The one-against-one classifier trains binary classifer for N class (multi class) data set. Each classifier recieves a pair of classes from the training set and we learn to classify between these two labels/classes. On the other hand, One versus Rest approach, we train on classifier per class, with the samples from that class labelled as Postive class and the rest as Negative class, and repeating these N times gives us a N class classifier. Now, all the samples are given Weights (probablity) for each class and from them we choose a winner class, giving the final Label. In order to perform Multi class classification we need to transform into a set of binary classification problem. When it comes to multi class classification The main difference between SVC and LinearSVC is they use One Vs One and One Vs Rest approach. One clear difference in SVC and Linear SVC is: SVC offers us different Kernels (rbf or poly) while LinearSVC just produces a linear margin of seperation. While in SVC the max iterations are infinite, LinearSVC limits them to 1000.
The last major difference is, in LinearSVC we have an option to choose between dual form of SVM or single form. In SVC we do no have that option.¶
Importing the packages and data with Tensorflow¶
from scipy.stats import mode
import numpy as np
#from mnist import MNIST
from time import time
import pandas as pd
import os
import matplotlib.pyplot as matplot
import matplotlib
%matplotlib inline
import random
matplot.rcdefaults()
from IPython.display import display, HTML
from itertools import chain
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
import seaborn as sb
from sklearn.model_selection import ParameterGrid
from sklearn.svm import SVC, LinearSVC
import warnings
warnings.filterwarnings('ignore')
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data/')
train = mnist.train.images
validation = mnist.validation.images
test = mnist.test.images
trlab = mnist.train.labels
vallab = mnist.validation.labels
tslab = mnist.test.labels
train = np.concatenate((train, validation), axis=0)
trlab = np.concatenate((trlab, vallab), axis=0)
Imp thing to remember: Data is 0-1 normalized¶
We save a lot of compute time by keeping the data that way and we dont lose any significant amount of accuracy
Linear SVC¶
Running a Sample Linear SVM classifier on default values to see how the model does on MNIST data
svm = LinearSVC(dual=False)
svm.fit(train, trlab)
svm.coef_
svm.intercept_
pred = svm.predict(test)
accuracy_score(tslab, pred) # Accuracy
cm = confusion_matrix(tslab, pred)
matplot.subplots(figsize=(10, 6))
sb.heatmap(cm, annot = True, fmt = 'g')
matplot.xlabel("Predicted")
matplot.ylabel("Actual")
matplot.title("Confusion Matrix")
matplot.show()
As we can see that the SVM does a pretty decent job at classifying, we still get the usual misclassification on 5-8, 2-8, 5-3, 4-9. However, accuracy of 91.82% is good
acc = []
acc_tr = []
coefficient = []
for c in [0.0001,0.001,0.01,0.1,1,10,100,1000,10000]:
svm = LinearSVC(dual=False, C=c)
svm.fit(train, trlab)
coef = svm.coef_
p_tr = svm.predict(train)
a_tr = accuracy_score(trlab, p_tr)
pred = svm.predict(test)
a = accuracy_score(tslab, pred)
coefficient.append(coef)
acc_tr.append(a_tr)
acc.append(a)
c = [0.0001,0.001,0.01,0.1,1,10,100,1000,10000]
matplot.subplots(figsize=(10, 5))
matplot.semilogx(c, acc,'-gD' ,color='red' , label="Testing Accuracy")
matplot.semilogx(c, acc_tr,'-gD' , label="Training Accuracy")
#matplot.xticks(L,L)
matplot.grid(True)
matplot.xlabel("Cost Parameter C")
matplot.ylabel("Accuracy")
matplot.legend()
matplot.title('Accuracy versus the Cost Parameter C (log-scale)')
matplot.show()
We clearly see a bias variance trade off in the graph. As the cost increases, the Training accuracy increases, so as the test accuracy, but only till c=1, then we see over fitting. From, c=10 to 1000 we see the model overfitting and we see Low Bias and High Variance
So as we go from Left to Right: Bias Decreases and Variance Increases
svm_coef = coefficient[4]
svm_coef.shape
matplot.subplots(2,5, figsize=(24,10))
for i in range(10):
l1 = matplot.subplot(2, 5, i + 1)
l1.imshow(svm_coef[i].reshape(28, 28), cmap=matplot.cm.RdBu)
l1.set_xticks(())
l1.set_yticks(())
l1.set_xlabel('Class %i' % i)
matplot.suptitle('Class Coefficients')
matplot.show()
These images look nothing like the images we saw in Logistic regression or Naive Bayes. In Naive Bayes, the underlying number was clearly visible, while in Logistice regression the pattern seemed quite distinct between all the classes. However, here you dont see any apparant patterns or distinctness and really hard to differentiate.
acc = []
acc_tr = []
coefficient = []
for c in [0.0001,0.001,0.01,0.1,1,10,100,1000,10000]:
svm = LinearSVC(dual=False, C=c, penalty='l1')
svm.fit(train, trlab)
coef = svm.coef_
p_tr = svm.predict(train)
a_tr = accuracy_score(trlab, p_tr)
pred = svm.predict(test)
a = accuracy_score(tslab, pred)
coefficient.append(coef)
acc_tr.append(a_tr)
acc.append(a)
c = [0.0001,0.001,0.01,0.1,1,10,100,1000,10000]
matplot.subplots(figsize=(10, 5))
matplot.semilogx(c, acc,'-gD' ,color='red' , label="Testing Accuracy")
matplot.semilogx(c, acc_tr,'-gD' , label="Training Accuracy")
#matplot.xticks(L,L)
matplot.grid(True)
matplot.xlabel("Cost Parameter C")
matplot.ylabel("Accuracy")
matplot.legend()
matplot.title('Accuracy versus the Cost Parameter C (log-scale)')
matplot.show()
Exact same thing with just a slight difference is clearly observed here as well. We see a bias variance trade off in the graph. As the cost increases, the Training accuracy increases, so as the test accuracy, but only till c=1, then we see over fitting. From, c=10 to 1000 we see the model overfitting and we see Low Bias and High Variance. Only thing is with L1 Penalty we have a lesser effect of overfitting and the model performs really poorly with lesser cost values.
Again, as we go from Left to Right: Bias Decreases and Variance Increases
Once more, we choose the model with best testing accuracy i.e. c = 1¶
svm_coef = coefficient[4]
svm_coef.shape
matplot.subplots(2,5, figsize=(24,10))
for i in range(10):
l1 = matplot.subplot(2, 5, i + 1)
l1.imshow(svm_coef[i].reshape(28, 28), cmap=matplot.cm.RdBu)
l1.set_xticks(())
l1.set_yticks(())
l1.set_xlabel('Class %i' % i)
matplot.suptitle('Class Coefficients')
matplot.show()
It reflects my views on Linear SVC with L2 (default) penalty, these images look very vaguely like the original images, also different than we saw in Logistic regression or Naive Bayes. In Naive Bayes, the underlying number was clearly visible, while in Logistice regression the pattern seemed quite distinct between all the classes. However, here you dont see any apparant patterns or distinctness and hard to differentiate between classes. However, we can also interpret some numbers like 0,5,6,8
Another important observation is, the images are also different looking than its L2 siblings as well, not by a bigger margin, but still different none the less.
