GitHub - MouslihAbdelhakim/sicrograd: Scala implementation of Micrograd. A tiny scalar-valued autograd engine and a neural net implementation.

Sicrograd

A Scala implementation of Micrograd, approached with a pure and functional style. This project was inspired by and built while following Andrej Karpathy's YouTube video Spelled-out intro to neural networks.

Inside, you'll discover:

• A domain-specific language for crafting mathematical expressions.
• A pure functional backpropagation engine.
• The foundational elements for a small neural network library.
• Utilities for visualizing expression graphs and observing neuron training.

Prerequisites

To get started, you'll need:

• SBT: For compiling and running the Scala code.
• Graphviz: For rendering the mathematical expression visualizations.

Once you have SBT and Graphviz installed, you can dive into the demos:

# To visualize a math expression
sbt "runMain com.github.mouslihabdelhakim.sicrograd.VisualizeExpressions" 

# To train a neural network to approximate the sin function
sbt "runMain com.github.mouslihabdelhakim.sicrograd.SinFunctionDemo" 

# To train a neural network on the moon dataset 
sbt "runMain com.github.mouslihabdelhakim.sicrograd.MoonDataSetDemo" 

# To visualize the training of a neural network
sbt "runMain com.github.mouslihabdelhakim.sicrograd.VisualizeNeuronTraining" 

Crafting and Visualizing Expressions

Sicrograd's DSL allows you to intuitively define variables (like weights and biases) and combine them to construct mathematical expressions. These expressions can then be visualized.

Here’s a taste:

    val expression = for {
      weight1 <- VariableScalar(1.0, "w1")
      weight2 <- VariableScalar(1.0, "w2")
      bias <- VariableScalar(1.0, "b")
    } yield weight1 * 2 + weight2 * 4 + bias

    val drawingResults = for {
      exp <- expression
      _ <- derive(exp)
      _ <- draw(exp)
    } yield ()

The output of this code is the following diagram:

Training neural networks

Sicrograd can train simple Multi-Layer Perceptrons (MLPs). The repository includes demos for tasks like function approximation and binary classification.

1. Approximating a Sine Wave

The SinFunctionDemo trains a single-layer MLP to approximate the sin function for values between [0, 2π].

The network is trained on randomly generated data points:

And here's how the network's output compares to the actual sin function after training:

2. Moon Dataset Classification

The MoonDataSetDemo provides a full example of training an MLP as a binary classifier on the classic "moon" dataset.

The data used to train the neural network are shown below:

Here is the achieved decision boundary on the moon dataset:

Visualizing Neuron Training Dynamics

For a peek into the learning process, the VisualizeNeuronTraining class demonstrates how a neuron's weights are updated by the gradient descent algorithm. While the GIF aesthetics are a work-in-progress, it effectively illustrates the iterative nature of training: