projects
🔢

MNIST Neural Network from Scratch

a two-layer neural network that recognizes handwritten digits, written with nothing but numpy. forward pass, backpropagation and gradient descent all done by hand, reaching 90.5% accuracy on the mnist test set.

pythonnumpyneural networksbackpropagation
enesdemirelus/mnist-using-numpy
language
Python
stars
0
updated
Sep 22

about this project

Note: All the code has been written by me.

A small neural network that recognizes handwritten digits, written with nothing but NumPy. No PyTorch, no TensorFlow. Forward pass, backpropagation and gradient descent are all done by hand.

It gets 90.5% accuracy on the MNIST test set.

Training

How it works

The network has two layers: 784 inputs (one per pixel), 10 hidden neurons with ReLU, and 10 outputs with softmax.

Forward pass:

Z1=W1X+b1,A1=ReLU(Z1),Z2=W2A1+b2,A2=softmax(Z2)Z_1 = W_1 X + b_1, \quad A_1 = \text{ReLU}(Z_1), \quad Z_2 = W_2 A_1 + b_2, \quad A_2 = \text{softmax}(Z_2)

Backward pass (with one-hot labels YY and mm training examples):

dZ2=A2−Y,dW2=1mdZ2A1T,db2=1m∑dZ2dZ_2 = A_2 - Y, \quad dW_2 = \tfrac{1}{m} dZ_2 A_1^T, \quad db_2 = \tfrac{1}{m} \textstyle\sum dZ_2 dZ1=W2TdZ2⊙ReLU′(Z1),dW1=1mdZ1XT,db1=1m∑dZ1dZ_1 = W_2^T dZ_2 \odot \text{ReLU}'(Z_1), \quad dW_1 = \tfrac{1}{m} dZ_1 X^T, \quad db_1 = \tfrac{1}{m} \textstyle\sum dZ_1

Then every parameter is updated with W←W−α dWW \leftarrow W - \alpha \, dW.

I trained it with full-batch gradient descent on all 60,000 training images, learning rate 0.5, for 500 iterations. It takes about 10 seconds on a laptop.

Results

Predictions on test images (wrong ones in red):

Predictions

Most mistakes happen between digits that look alike, like 4 and 9 or 3 and 5:

Confusion matrix

The first-layer weights of each hidden neuron, reshaped back into 28x28 images:

Hidden weights

Running it

pip install -r requirements.txt
python mnist.py

You need the MNIST CSV files in dataset/ first. See dataset/README.md for where to get them.

GitHub