top of page

What is Support Vector Machines(SVMs)?



Support vectors are data points that maximize the margin around a hyperplane that separates positive and negative instances in dataset.










Basic idea:

Another method for building a classifier where we view the data “spatially”, and predict a new instance’s class based on where it is “located in space”


For simplicity, we’ll assume decision is binary (positive/negative, yes/no, etc.)


Assume numeric attributes (easier to work with as “points in space” and perform arithmetic calculations); however, not restricted to 2D


Hyperplane: plane (or line if 2D space) that separates positive and negative instances in dataset


Support vectors: data points (actual instances from training dataset) that maximize the margin around the hyperplane; they “anchor” the hyperplane


Linear separability

One equation for defining a line is ax + by = c








Find a, b, c such that ax + by ≥ c for red points and ax + by ≤ c for green points In general, many possible solutions for a, b, c


Defining the hyperplane (assuming you knew the support vectors)

Ex: In the dataset below, assume decision attribute is z











Easy to see that support vectors are s1 = (1 0), s2 = (3 1), and s3 = (3 -1)


Augment vectors with 1 more dimension; assign value of 1 for this dimension in each support vector s1 = (1 0 1), s2 = (3 1 1), s3 = (3 -1 1)


Set up equations we need to solve:

φ is a function that maps instance data to a “feature space”

α‘s are parameters which, when “combined” with support vectors and respective decisions, will help us define the hyperplane


α1φ(s1) • φ(s1) + α2φ(s2) • φ(s1) + α3φ(s3) • φ(s1) = -1 s1’s decision

α1φ(s1) • φ(s2) + α2φ(s2) • φ(s2) + α3φ(s3) • φ(s2) = 1 s2’s decision

α1φ(s1) • φ(s3) + α2φ(s2) • φ(s3) + α3φ(s3) • φ(s3) = 1 s3’s decision


Let φ( ) be I (identity vector):

Since data in this example are linearly separable, we’re going to be able to use a linear SVM; so we’re just going to use identify function for φ

α1s1 • s1 + α2s2 • s1 + α3s3 • s1 = -1

α1s1 • s2 + α2s2 • s2 + α3s3 • s2 = 1

α1s1 • s3 + α2s2 • s3 + α3s3 • s3 = 1


Compute dot product in each equation:

2α1 + 4α2 + 4α3 = -1 e.g., s2 • s1 = (3 1 1) • (1 0 1) = (3*1) + (1*0) + (1*1) = 4

4α1 + 11α2 + 9α3 = 1

4α1 + 9α2 + 11α3 = 1


Solve for α1, α2, and α3:

α1 = -3.5, α2 = 0.75, α3 = 0.75


Define the discriminating hyperplane:

w’ = α1s1 + α2s2 + α3s3

= -3.5 (1 0 1) + 0.75 (3 1 1) + 0.75 (3 -1 1)

= (1 0 -2)


For hyperplane in 2D, (a b c) translates to ax + by + c = 0

For hyperplane in 3D, (a b c d) translates to ax + by + cz + d = 0 etc.


So in this case (1 0 -2) translates to 1x + 0y + -2 = 0 or simply x = 2












Doing Linear SVM in Python


Import Libraries

import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt 
from sklearn.svm import SVC 

Read Data

df = pd.read_csv('svm_exampleData.csv')

Change non numeric value into numeric value

# Just for graphing purposes, change non-decision attr's to have numeric values 
df= df.replace({'z': r'positive'}, {'z':1}, regex=True) 
df= df.replace({'z': r'negative'}, {'z':-1}, regex=True)
X = df.iloc[:, 0:2].values 
y = df.iloc[:, 2].values  
r,_ = df.shape 
# Display original data points 
plt.scatter(X[:, 0], X[:, 1], c=y, s=r, cmap='winter'); 
plt.show() 

Make Linear SVM

# Make a linear SVM 
model = SVC(kernel='linear') 
model.fit(X, y) 
print(model.support_vectors_)         # The support vectors 
print(model.coef_)                    # The weights (x and y) 
print(model.intercept_)               # The intercept 

# Function to plot SVM boundary lines - cool!
def plot_svc_decision_function(model, ax=None, plot_support=True):
 	if ax is None:
 		ax = plt.gca()
 	xlim = ax.get_xlim()
 	ylim = ax.get_ylim()
 
 	# Create grid to evaluate model
 	x = np.linspace(xlim[0], xlim[1], 30)
 	y = np.linspace(ylim[0], ylim[1], 30)
 	Y, X = np.meshgrid(y, x)
 	xy = np.vstack([X.ravel(), Y.ravel()]).T
 	P = model.decision_function(xy).reshape(X.shape)
 
 	# Plot decision boundary and margins
 	ax.contour(X, Y, P, colors='k',
 		levels=[-1, 0, 1], alpha=0.5,
 		linestyles=['--', '-', '--'])
 
 	# Plot support vectors
 	if plot_support:
 		ax.scatter(model.support_vectors_[:, 0],
 		model.support_vectors_[:, 1],
 		s=300, linewidth=1, facecolors='none');
 	ax.set_xlim(xlim)
 	ax.set_ylim(ylim)

# Call the function to show points and SV boundaries
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='winter')
plot_svc_decision_function(model);
plt.show()

# Make predictions (will be array([-1]) or array([1]))
model.predict([[-3, 4]]) 
model.predict([[7, 5]])


Thanks for your support!

You can send your request directly at realcode4you@gmail.com if you need any SVMs related help, machine learning related help, data science assignment help, python assignment help or any project related to machine learning and get instant help with an affordable prices.

Comments


bottom of page