Search

# What is Support Vector Machines(SVMs)? Support vectors are data points that maximize the margin around a hyperplane that separates positive and negative instances in dataset. Basic idea:

Another method for building a classifier where we view the data “spatially”, and predict a new instance’s class based on where it is “located in space”

For simplicity, we’ll assume decision is binary (positive/negative, yes/no, etc.)

Assume numeric attributes (easier to work with as “points in space” and perform arithmetic calculations); however, not restricted to 2D

Hyperplane: plane (or line if 2D space) that separates positive and negative instances in dataset

Support vectors: data points (actual instances from training dataset) that maximize the margin around the hyperplane; they “anchor” the hyperplane

Linear separability

One equation for defining a line is ax + by = c Find a, b, c such that ax + by ≥ c for red points and ax + by ≤ c for green points In general, many possible solutions for a, b, c

Defining the hyperplane (assuming you knew the support vectors)

Ex: In the dataset below, assume decision attribute is z Easy to see that support vectors are s1 = (1 0), s2 = (3 1), and s3 = (3 -1)

Augment vectors with 1 more dimension; assign value of 1 for this dimension in each support vector s1 = (1 0 1), s2 = (3 1 1), s3 = (3 -1 1)

Set up equations we need to solve:

φ is a function that maps instance data to a “feature space”

α‘s are parameters which, when “combined” with support vectors and respective decisions, will help us define the hyperplane

α1φ(s1) • φ(s1) + α2φ(s2) • φ(s1) + α3φ(s3) • φ(s1) = -1 s1’s decision

α1φ(s1) • φ(s2) + α2φ(s2) • φ(s2) + α3φ(s3) • φ(s2) = 1 s2’s decision

α1φ(s1) • φ(s3) + α2φ(s2) • φ(s3) + α3φ(s3) • φ(s3) = 1 s3’s decision

Let φ( ) be I (identity vector):

Since data in this example are linearly separable, we’re going to be able to use a linear SVM; so we’re just going to use identify function for φ

α1s1 • s1 + α2s2 • s1 + α3s3 • s1 = -1

α1s1 • s2 + α2s2 • s2 + α3s3 • s2 = 1

α1s1 • s3 + α2s2 • s3 + α3s3 • s3 = 1

Compute dot product in each equation:

2α1 + 4α2 + 4α3 = -1 e.g., s2 • s1 = (3 1 1) • (1 0 1) = (3*1) + (1*0) + (1*1) = 4

4α1 + 11α2 + 9α3 = 1

4α1 + 9α2 + 11α3 = 1

Solve for α1, α2, and α3:

α1 = -3.5, α2 = 0.75, α3 = 0.75

Define the discriminating hyperplane:

w’ = α1s1 + α2s2 + α3s3

= -3.5 (1 0 1) + 0.75 (3 1 1) + 0.75 (3 -1 1)

= (1 0 -2)

For hyperplane in 2D, (a b c) translates to ax + by + c = 0

For hyperplane in 3D, (a b c d) translates to ax + by + cz + d = 0 etc.

So in this case (1 0 -2) translates to 1x + 0y + -2 = 0 or simply x = 2 Doing Linear SVM in Python

Import Libraries

```import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVC ```

`df = pd.read_csv('svm_exampleData.csv')`

Change non numeric value into numeric value

```# Just for graphing purposes, change non-decision attr's to have numeric values
df= df.replace({'z': r'positive'}, {'z':1}, regex=True)
df= df.replace({'z': r'negative'}, {'z':-1}, regex=True)```
```X = df.iloc[:, 0:2].values
y = df.iloc[:, 2].values
r,_ = df.shape ```
```# Display original data points
plt.scatter(X[:, 0], X[:, 1], c=y, s=r, cmap='winter');
plt.show() ```

Make Linear SVM

```# Make a linear SVM
model = SVC(kernel='linear')
model.fit(X, y)
print(model.support_vectors_)         # The support vectors
print(model.coef_)                    # The weights (x and y)
print(model.intercept_)               # The intercept ```

```# Function to plot SVM boundary lines - cool!
def plot_svc_decision_function(model, ax=None, plot_support=True):
if ax is None:
ax = plt.gca()
xlim = ax.get_xlim()
ylim = ax.get_ylim()

# Create grid to evaluate model
x = np.linspace(xlim, xlim, 30)
y = np.linspace(ylim, ylim, 30)
Y, X = np.meshgrid(y, x)
xy = np.vstack([X.ravel(), Y.ravel()]).T
P = model.decision_function(xy).reshape(X.shape)

# Plot decision boundary and margins
ax.contour(X, Y, P, colors='k',
levels=[-1, 0, 1], alpha=0.5,
linestyles=['--', '-', '--'])

# Plot support vectors
if plot_support:
ax.scatter(model.support_vectors_[:, 0],
model.support_vectors_[:, 1],
s=300, linewidth=1, facecolors='none');
ax.set_xlim(xlim)
ax.set_ylim(ylim)

# Call the function to show points and SV boundaries
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='winter')
plot_svc_decision_function(model);
plt.show()

# Make predictions (will be array([-1]) or array())
model.predict([[-3, 4]])
model.predict([[7, 5]])

```