Deep Learning Architectures
Sliding the kernel…
Initializing Sorting Algorithms...
Sorting
Trees
Graphs
Preparing interactive visualizations...
Deep Learning Architectures
Sliding the kernel…
A plain network treats every input the same way. Architectures build in what we know about the data — images are made of local patterns, sentences arrive in order — and that is what makes them learn.
Slide a 3×3 kernel over a picture and watch every multiply-add. The same nine weights are used at every position, which is what makes a convolution cheap and an edge detector work anywhere.
y(i, j) = Σₐ Σᵦ w(a, b) · x(i + a, j + b)Forward PassSliding Dot ProductMax and average pooling shrink a feature map, kernels stack into channels, and each layer down the network sees a wider patch of the image than the one before.
RF = RF₍prev₎ + (k − 1) · jumpConvolutionDownsample and StackDraw a digit and a real network trained on MNIST reads it: every feature map of every layer, then the ten probabilities. Edges first, then strokes, then digits.
28×28 → 8×14×14 → 16×7×7 → 32 → 10Pooling and Feature MapsReal ModelUnroll a network over a sequence and carry its memory as a vector. Train one live to count, then see exactly where its memory runs out.
hₜ = tanh(W hₜ₋₁ + U xₜ + b)BackpropagationShared Weights Over TimeGates are numbers between 0 and 1 that decide what to keep, forget and output. Watch trained gates hold one number while noise streams past, where a plain RNN drifts.
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜRecurrent Neural NetworksGated MemoryAn encoder squeezes a whole sentence into one vector and a decoder writes the translation from it. The longer the sentence, the more gets lost — the problem attention was invented to fix.
c = h_T, yₜ = decoder(yₜ₋₁, sₜ₋₁, c)LSTM and GRUEncode, Then DecodeTwenty plain layers fit a spiral worse than four. Train deep stacks with and without skip connections side by side, and watch adding the input back keep the gradient alive through every layer — the backbone of every transformer.
hₗ₊₁ = hₗ + F(hₗ)Vanishing and Exploding GradientsSkip ConnectionSqueeze 144 pixels through two numbers and back. Train it live, then drag a point around the two-number space and watch the decoder paint the picture it stands for.
x̂ = decode(encode(x)), L = ‖x − x̂‖²Principal Component AnalysisBottleneckA dense layer from a 28×28 image to 8 feature maps would need almost five million weights. A convolution does the same job with 80, because it assumes a pattern means the same thing wherever it appears.
Recurrent networks carry memory as one vector, gates keep it alive, and a seq2seq encoder squeezes a sentence into it — until it breaks. Attention is the fix, and this section is the problem it fixes.
Residual connections are why a 100-layer network trains at all, and an autoencoder's bottleneck is the simplest way to see what an embedding is. Both are inside every large model you have used.