Gradient Descent Arena

Three ways to walk downhill on the same map. They read the same slope from the same starting dot and differ only in how much of the past they carry — and on this map that alone decides who finds the real bottom. Click anywhere to drop all three somewhere new.

A contour map of a loss landscape with three optimizer paths drawn on it.
click the map to drop them somewhere new

Race step 0

Landscape

Step size

0.050

How far each step moves per unit of slope. Too small and nobody gets anywhere. Push it to the top on the narrow valley and plain SGD overshoots and blows up.

The three rules

sgd  x ← x − lr·g
momentum  v ← 0.9·v + g
               x ← x − lr·v
adam  m ← 0.9·m + 0.1·g
       v ← 0.999·v + 0.001·g²
       x ← x − lr·m̂/(√v̂+1e−8)
g is the exact slope at the current point — no sampling noise — so the only thing separating the three paths is the update rule itself.