Perceptron#
The Perceptron is a simple algorithm for binary classification. It’s a type of linear classifier that makes its predictions based on a linear predictor function combining a set of weights with the feature vector.
Perceptron is used on the dataset which is linearly seperable (line)
Mathematically, the prediction of a Perceptron for an input vector x is given by:
Here:
wis the weight vectorxis the input vectorbis the bias term.denotes the dot productsignThe signum function of a real number \({\displaystyle z}\) where \(z = (w.x + b)\)
The Perceptron learning algorithm works as follows:
Initialize the weight vector
wand the biasbto zero.For each training example \((x, y)\) in our training set, perform the following steps:
a. Compute the prediction \(y\_hat = sign(w.x + b)\). Here,
signis a function that returns-1if the input is less than0, and1otherwise.b. If the prediction
y_hatis incorrect (i.e.,y_hat != y), update the weight vector and bias as follows:\(w = w + y * x\)
\(b = b + y\)
Repeat step 2 until all training examples are correctly classified, or a maximum number of iterations has been reached.
The algorithm iterates over the training examples, making these updates until all examples are correctly classified, or a maximum number of iterations has been reached. The resulting w and b define a linear decision boundary that can be used to classify new examples.
import numpy as np
# Define a simple binary classification dataset
X = np.array([[-4, 2], [-2, 1], [-1, -1], [2, 2], [1, -2]]) # feature vector
y = np.array([1, 1, -1, -1, -1]) # corresponding labels
# Initialize w and b to zero
w = np.zeros(X.shape[1])
b = 0
# Define the sign function
def sign(x):
if x > 0:
return 1
elif x == 0:
return 0
else:
return -1
# Perceptron learning algorithm
for _ in range(200): # Limit iterations to prevent infinite loop
error_count = 0
for xi, yi in zip(X, y):
prediction = sign(np.dot(w, xi) + b)
if prediction != yi:
w += yi * xi
b += yi
error_count += 1
print(f"Prediction: {prediction}, Actual: {yi}, xi: {xi}, w: {w}, b: {b}")
if error_count == 0:
break
print("Final w:", w)
print("Final b:", b)
Prediction: 0, Actual: 1, xi: [-4 2], w: [-4. 2.], b: 1
Prediction: 1, Actual: 1, xi: [-2 1], w: [-4. 2.], b: 1
Prediction: 1, Actual: -1, xi: [-1 -1], w: [-3. 3.], b: 0
Prediction: 0, Actual: -1, xi: [2 2], w: [-5. 1.], b: -1
Prediction: -1, Actual: -1, xi: [ 1 -2], w: [-5. 1.], b: -1
Prediction: 1, Actual: 1, xi: [-4 2], w: [-5. 1.], b: -1
Prediction: 1, Actual: 1, xi: [-2 1], w: [-5. 1.], b: -1
Prediction: 1, Actual: -1, xi: [-1 -1], w: [-4. 2.], b: -2
Prediction: -1, Actual: -1, xi: [2 2], w: [-4. 2.], b: -2
Prediction: -1, Actual: -1, xi: [ 1 -2], w: [-4. 2.], b: -2
Prediction: 1, Actual: 1, xi: [-4 2], w: [-4. 2.], b: -2
Prediction: 1, Actual: 1, xi: [-2 1], w: [-4. 2.], b: -2
Prediction: 0, Actual: -1, xi: [-1 -1], w: [-3. 3.], b: -3
Prediction: -1, Actual: -1, xi: [2 2], w: [-3. 3.], b: -3
Prediction: -1, Actual: -1, xi: [ 1 -2], w: [-3. 3.], b: -3
Prediction: 1, Actual: 1, xi: [-4 2], w: [-3. 3.], b: -3
Prediction: 1, Actual: 1, xi: [-2 1], w: [-3. 3.], b: -3
Prediction: -1, Actual: -1, xi: [-1 -1], w: [-3. 3.], b: -3
Prediction: -1, Actual: -1, xi: [2 2], w: [-3. 3.], b: -3
Prediction: -1, Actual: -1, xi: [ 1 -2], w: [-3. 3.], b: -3
Final w: [-3. 3.]
Final b: -3
In this code, we first define a simple binary classification dataset with four examples. The feature vectors are in X and the corresponding labels are in y. We initialize w and b to zero and define the sign function.
Then we run the Perceptron learning algorithm. For each training example, we compute the prediction using the current w, b and the sign function. If the prediction is incorrect, we update w by adding the product of the label and the feature vector, and we increment b by the label. We count the number of errors made in each iteration.
The algorithm stops when it makes no errors on the training set, or when it has performed a maximum number of iterations (in this case, 100). Finally, we print the learned w and b.
import numpy as np
# Define a simple binary classification dataset
X = np.array([[-1, -1], [1, 0], [-1, 10]]) # feature vector
y = np.array([1, -1, 1]) # corresponding labels
# Initialize w and b to zero
# w = np.zeros(X.shape[1])
w = np.array([-1, -1])
b = 0
mistakes = 0
# Define the sign function
def sign(x):
if x > 0:
return 1
elif x == 0:
return 0
else:
return -1
# Perceptron learning algorithm
for _ in range(200): # Limit iterations to prevent infinite loop
error_count = 0
for xi, yi in zip(X, y):
prediction = sign(np.dot(w, xi) + b)
if prediction != yi:
w += yi * xi
b += yi
error_count += 1
mistakes += 1
print(f"Prediction: {prediction}, Actual: {yi}, xi: {xi}, w: {w}, b: {b}")
if error_count == 0:
break
print("Final w:", w)
print("Final b:", b)
print("Number of mistakes:", mistakes)
Prediction: 1, Actual: 1, xi: [-1 -1], w: [-1 -1], b: 0
Prediction: -1, Actual: -1, xi: [1 0], w: [-1 -1], b: 0
Prediction: -1, Actual: 1, xi: [-1 10], w: [-2 9], b: 1
Prediction: -1, Actual: 1, xi: [-1 -1], w: [-3 8], b: 2
Prediction: -1, Actual: -1, xi: [1 0], w: [-3 8], b: 2
Prediction: 1, Actual: 1, xi: [-1 10], w: [-3 8], b: 2
Prediction: -1, Actual: 1, xi: [-1 -1], w: [-4 7], b: 3
Prediction: -1, Actual: -1, xi: [1 0], w: [-4 7], b: 3
Prediction: 1, Actual: 1, xi: [-1 10], w: [-4 7], b: 3
Prediction: 0, Actual: 1, xi: [-1 -1], w: [-5 6], b: 4
Prediction: -1, Actual: -1, xi: [1 0], w: [-5 6], b: 4
Prediction: 1, Actual: 1, xi: [-1 10], w: [-5 6], b: 4
Prediction: 1, Actual: 1, xi: [-1 -1], w: [-5 6], b: 4
Prediction: -1, Actual: -1, xi: [1 0], w: [-5 6], b: 4
Prediction: 1, Actual: 1, xi: [-1 10], w: [-5 6], b: 4
Final w: [-5 6]
Final b: 4
Number of mistakes: 4