Key Terms in Deep Learning – Explained Simply

Neural Network Basics: A Hands-On Guide to MLPs, Activation Functions, and Learning Dynamics

Introduction

Neural networks have become foundational to modern machine learning, powering everything from image recognition to language models.
In this notebook, we’ll break down the basics of neural networks, focusing on perceptrons, multi-layer perceptrons (MLPs), activation functions,
and core training principles. We'll also build a simple MLP using PyTorch, explore the role of weight initialization, and visualize decision
boundaries to understand what our networks learn.

1. Perceptron: The Building Block

The perceptron is the simplest type of neural network—a single neuron. It performs a weighted sum of inputs and passes the result through an activation function.

Equation:

𝑦 = 𝜙(𝑤⊤𝑥 + 𝑏)
x: input vector
w: weight vector
b: bias
ϕ: activation function (e.g., step function)

Limitations: A single-layer perceptron can only solve linearly separable problems like AND and OR—not XOR.

2. Multi-Layer Perceptron (MLP)

An MLP is a neural network with one or more hidden layers. It allows for the modeling of complex, non-linear functions.

Structure:

• Input layer: Receives raw input.
• Hidden layers: Transform input using learned weights and non-linear activation functions.
• Output layer: Produces the final prediction.

Each layer applies:
h = ϕ(Wx + b)

MLPs are universal function approximators—with enough neurons and layers, they can model any function.

3. Activation Functions

Activation functions introduce non-linearity, allowing networks to learn complex mappings.

4. Forward Pass and Backpropagation

Forward Pass: How Neural Networks Make Predictions

Imagine you’re feeding an image of a cat into a neural network. What happens next?

In an MLP:

• Input data is multiplied by weights and passed through an activation function (like ReLU or sigmoid).
• This process happens layer by layer, from input to output — this is called the forward pass.
• The final layer gives us a prediction (e.g., “80% chance this is a cat”).

Mathematically:

z = W*x + b
a = activation(z)

Where:

• x = input
• W = weight matrix
• b = bias
• a = activation output

Backpropagation: Learning From Mistakes

Now that the network made a prediction, we compare it to the true label and calculate the loss.

Backpropagation is how the network learns:

1. Compute the gradient of the loss w.r.t. each weight using the chain rule.
2. Adjust the weights slightly in the direction that reduces the loss.

W = W - learning_rate * ∇(loss)

Error is calculated (loss), and gradients are computed using the chain rule to update weights via gradient descent.

High-level Steps:

1. Compute loss (e.g., cross-entropy)
2. Compute gradients of loss w.r.t. weights
3. Update weights:

Vanishing and Exploding Gradients

Backpropagation works well — until it doesn’t.

In deep networks, gradients can become:

• Very small (vanishing) ➜ early layers learn very slowly
• Very large (exploding) ➜ weights blow up, training becomes unstable

This is especially problematic with activation functions like sigmoid, which squash outputs into a small range.

Solution: Weight Initialization

Smart weight initialization helps keep gradients in a good range. Two popular strategies:

1. Xavier (Glorot) Initialization – best for tanh activations
2. He Initialization – ideal for ReLU

These methods set initial weights with a variance that depends on the number of inputs/outputs per layer.

5. Build an MLP in PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Generate toy data
X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)
X = torch.tensor(X, dtype=torch.float32)
y = torch.tensor(y, dtype=torch.long)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Define model
class SimpleMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(2, 16),
            nn.ReLU(),
            nn.Linear(16, 2)
        )

    def forward(self, x):
        return self.net(x)

model = SimpleMLP()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Training loop
for epoch in range(100):
    optimizer.zero_grad()
    outputs = model(X_train)
    loss = criterion(outputs, y_train)
    loss.backward()
    optimizer.step()

    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item():.4f}")
---
Thanks for Reading
---