ReLU: The AI That Beat a Go Legend Ran on One Rule
A neural network is a line with bends: each neuron adds one bend with one rule, if it's negative make it zero, and with enough bends the line can follow any curve.
AlphaGo, the AI that beat Go legend Lee Sedol 4–1 in 2016, ran its networks on that one rule. Inside, a network is mostly knobs (numbers it can tune) multiplying and adding, and multiplying and adding alone can only ever draw a straight line: warmer is better, or warmer is worse, never "just right". This episode builds the fix by hand on a curve everyone feels, how comfortable you are against the room temperature. The rule folds a straight line into one bend; two bent lines make the comfort mountain; enough of them follow any curve, and training slides the bends into place. Then back to AlphaGo: the same idea with a whole Go board going in and its chance of winning coming out.
Watch the Short of this episode on YouTube
From the film
Where this sits: Deep Learning 2, the Big Idea of a network as a line with bends. The Deep Dives that take it further are Deep Learning 2.1 (ReLU pieces), 2.2 (why it can fit any curve) and 2.3 (many inputs and outputs: what a bend is when a whole board goes in) (links to follow once they are published). Next Big Idea: why AI is deep, not wide (Deep Learning 3).
More on the rule: its name is ReLU, the rectified linear unit, max(0, z). It won AI's big photo contest, ImageNet: AlexNet (2012) missed 15.3 % with five guesses against the runner-up's 26.2 %, and its makers chose the rule because networks with it "train several times faster"; ResNet (2015) stacked 152 layers with the rule after each. AlphaGo's winning-chance network learned from 30 million positions, each from a separate game it played against itself. Chatbots such as Meta's LLaMA use a smoothed version of the bend (SwiGLU), and BERT and GPT another (GELU), so no chatbot is claimed here to run on this exact rule. "After every sum": every hidden layer of AlphaGo's networks ends on the rule; the last layer turns the total into probabilities or a score (the output, not a bend). AlphaGo also looked ahead with a tree search that asked the network who was winning.
Sources
- AlphaGo's networks (the rectifier after every hidden layer, the softmax / tanh outputs, k = 192 filters, about 3.9 million weights in the policy network, 30 million self-play positions, the lookahead search): Silver et al., "Mastering the game of Go with deep neural networks and tree search", Nature 529 (2016)https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf
- AlphaGo Lee defeated Lee Sedol 4–1 in March 2016 and is "similar in most regards" to the published AlphaGo; Lee Sedol, "the winner of 18 international titles": Silver et al., "Mastering the game of Go without human knowledge", Nature 550 (2017), UCL accepted manuscripthttps://discovery.ucl.ac.uk/10045895/1/agz_unformatted_nature.pdf
- The rule (ReLU), one bend per neuron, and the theorem that enough neurons get as close as you like to any continuous curve: Prince, Understanding Deep Learning, MIT Press (2023), §3.1–3.4https://github.com/udlbook/udlbook/releases/download/v1.0.4/UnderstandingDeepLearning_08_05_23_C.pdf
- ImageNet 2012 (15.3 % against 26.2 %; ReLUs "train several times faster"): Krizhevsky, Sutskever & Hinton, NIPS 2012https://papers.nips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
- ResNet (152 layers, ILSVRC 2015, ReLU): He et al. (2015)https://arxiv.org/abs/1512.03385
- LLaMA (SwiGLU instead of ReLU): Touvron et al., Meta AI (2023)https://arxiv.org/abs/2302.13971
- BERT (GELU, following OpenAI GPT): Devlin et al. (2018)https://arxiv.org/abs/1810.04805
- The comfort example is an illustration of a feeling, not data. Every number in the film (the two-neuron mountain 0, 0, 0, 5, 10, 5, 0 at 50 … 80 °F; the 3 / 6 / 12-neuron fits; the 3.9 million) is checked in the project's check script.
The drawings are made with generative AI and drawn on stroke by stroke; the graphs are plotted by code. The narration is a synthetic voice.