Neural network from scratch
A feedforward neural network in plain C
As a part of my journey to understand AI better, I undertook the project of building a fully functioning feedforward neural network (FFNN) from scratch, using no external libraries, all in C. Throughout this project, I learned a lot about how machine learning models and training them works under the hood. I learned about the linear algebra that goes into making a model like this, and the calculus that goes into backpropagation and training.
I was able to successfully train a simple feedforward neural network to solve the XOR problem. Along the way, I ran into many problems that took a long time to fix, such as training not working because I forgot to negate the gradients, or ReLU cells dying because their activations were too negative.
Process
To build this project, I first worked on creating functionality for implementing linear algebra. I created structures for matrices and functions for multiplying them, adding them, resizing them, and more.
Then, I implemented the structure of an FFNN. I made a Layer structure with matrices for weights and biases (and their gradients), and then strung them together into a larger FFNN structure, which consisted of multiple layers.
I then implemented the forward pass: taking an input vector and multiplying by each successive weight matrix, adding the biases at each step. So far, everything was working well and it had been very simple, but I knew that training would be much more difficult.
I implemented backpropagation by looping backward through each layer and calculating its gradient with respect to the layer before, and then multiplying this with the gradient of the layer before to get the current layer’s gradient. This is an application of the chain rule from calculus. I ran into a lot of problems along the way but eventually I got it working.
I then set up the training loop with batch training, epochs, and learning rates.
Finally, I set up a simple example to train it on - the XOR problem, which was given two numbers that are 0 or 1 as input, the program must output what you get when you XOR them. This is a bit complicated because it is nonlinear, so it requires the use of a nonlinear activation function like ReLU or sigmoid to solve. At first, my program completely failed to solve this, but then I dug deeper into the code and figured out that my ReLU cells were dying, so after replacing them with sigmoid, it worked well. This was really rewarding, seeing my program learn on its own how to solve a problem, albeit a simple one.
Example Code
Here is the code I used for training the model:
#include <stdio.h>
#include "../include/nn/ffnn.h"
int main() {
int seed = 7;
// Create a FFNN trained on the XOR problem
// Input: 2, Hidden: 5, Output: 1
int input_dim = 2;
int layer_dims[] = { 5, 1};
int num_layers = sizeof(layer_dims) / sizeof(layer_dims[0]);
// Create the feedforward neural network
FFNN ffnn = ffnn_create(input_dim, num_layers, layer_dims, seed,
approx_sigmoid_mat,
approx_sigmoid_derivative,
approx_sigmoid_mat,
approx_sigmoid_derivative);
// Print each layer and its weights and biases
printf("Feedforward Neural Network:\n");
for (int i = 0; i < ffnn.num_layers; i++) {
FFNNLayer layer = ffnn.layers[i];
printf("Layer %d:\n", i);
printf("Weights:\n");
mat_print(layer.weights);
printf("Biases:\n");
mat_print(layer.biases);
printf("\n");
}
// Create training data for the XOR problem
Matrix x[4];
x[0] = mat_create_zeros(2, 1); // [0, 0]
x[1] = mat_create_zeros(2, 1); // [0, 1]
x[2] = mat_create_zeros(2, 1); // [1, 0]
x[3] = mat_create_zeros(2, 1); // [1, 1]
*mat_idx(x[1], 1, 0) = 1.0;
*mat_idx(x[2], 0, 0) = 1.0;
*mat_idx(x[3], 1, 0) = 1.0;
*mat_idx(x[3], 0, 0) = 1.0;
Matrix y_true[4];
y_true[0] = mat_create_zeros(1, 1); // [0]
y_true[1] = mat_create_zeros(1, 1); // [1]
y_true[2] = mat_create_zeros(1, 1); // [1]
y_true[3] = mat_create_zeros(1, 1); // [0]
*mat_idx(y_true[1], 0, 0) = 1.0;
*mat_idx(y_true[2], 0, 0) = 1.0;
// Train the network
int batch_size = 1;
int num_epochs = 10000;
dtype learning_rate = 0.01;
ffnn_train(&ffnn, 4, x, y_true, num_epochs, learning_rate, batch_size, 1);
// Print each layer and its weights and biases
printf("Feedforward Neural Network:\n");
for (int i = 0; i < ffnn.num_layers; i++) {
FFNNLayer layer = ffnn.layers[i];
printf("Layer %d:\n", i);
printf("Weights:\n");
mat_print(layer.weights);
printf("Biases:\n");
mat_print(layer.biases);
printf("\n");
}
int example = 3;
ffnn_forward(&ffnn, x[example]);
printf("\nForward pass output:\n");
for (int i = 0; i < ffnn.num_layers; i++) {
FFNNLayer layer = ffnn.layers[i];
printf("Layer %d:\n", i);
printf("Outputs:\n");
mat_print(layer.outputs);
printf("Activations:\n");
mat_print(layer.activations);
printf("\n");
}
// Perform backpropagation
ffnn_zero_grad(&ffnn);
ffnn_backprop(&ffnn, x[example], y_true[example]);
printf("\nBackpropagation gradients:\n");
for (int i = 0; i < ffnn.num_layers; i++) {
FFNNLayer layer = ffnn.layers[i];
printf("Layer %d:\n", i);
printf("Weights gradient:\n");
mat_print(layer.weights_grad);
printf("Biases gradient:\n");
mat_print(layer.biases_grad);
printf("Outputs gradient:\n");
mat_print(layer.outputs_grad);
printf("Activations gradient:\n");
mat_print(layer.activations_grad);
printf("\n");
}
// Predict the output for each of the training data
for (int i = 0; i < 4; i++) {
ffnn_forward(&ffnn, x[i]);
printf("Predicted output for input %d:\n", i);
mat_print(ffnn.layers[ffnn.num_layers - 1].activations);
}
// Free the memory allocated for the neural network
ffnn_free(&ffnn);
// Free x and y_true
for (int i = 0; i < 4; i++) {
mat_free(&x[i]);
mat_free(&y_true[i]);
}
return EXIT_SUCCESS;
}