From Follow the Leader and continuous-time gradient flows to Mirror Descent, Multiplicative Weights, and minimax duality
Chihao Zhang · Fall 2026
In Chapter 1 we studied sequential decision-making under a benign statistical assumption: in the stochastic multi-armed bandit problem, each action draws rewards independently from a fixed, unknown probability distribution, so empirical averages concentrate around true means by Hoeffding’s inequality. What happens when we drop the probability model altogether? Whether a player is guessing numbers against an adaptive opponent, a router is sending packets through a network with fluctuating traffic, or an optimization algorithm is iteratively adjusting dual multipliers to satisfy coupling constraints, the environment is rarely independent and identically distributed—it may be non-stationary, reactive, or actively adversarial.
This chapter develops the theory of online learning from first principles across two meetings. In our first meeting, we start with a concrete online number guessing game under quadratic loss (xt−yt)2, where the greedy Follow-the-Leader (FTL) rule—playing the historical average xt=t−11∑s=1t−1ys—achieves logarithmic regret 2HT=Θ(logT). Yet as soon as the loss functions become linear, FTL oscillates between extreme points and suffers linear regret Ω(T), and in general solving the offline optimization problem argminx∈V∑s=1t−1ℓs(x) from scratch each round is already non-trivial. We therefore step back to study offline convex optimization minx∈Vf(x) and prove the convergence of offline Projected Gradient Descent via the Euclidean potential ϕ(x)=21∥x−x∗∥22, before moving to Online Projected Gradient Descent (OGD)—which achieves O(T) regret for general convex losses and O(logT) regret for μ-strongly convex losses. Applying Euclidean OGD to combinatorial problems via convex lifting then solves prediction with expert advice on the simplex Δn with O(nT) regret and online shortest paths via probability flows.
In our second meeting, we first extend Euclidean OGD to adversarial multi-armed bandits via importance-weighted Online Stochastic Gradient Descent (OSGD) with O((nT)2/3) regret, and then confront the two sharp puzzles exposed by Euclidean OGD on the probability simplex Δn: why does full-information expert prediction pay a polynomial dimension factor n, and why does bandit feedback degrade from T1/2 to (nT)2/3 because the variance ∑i=1nxt(i)ℓt(i)2 blows up near the boundary of Δn? Both bottlenecks stem from the Euclidean regularizer 2η1∥x−xt∥22, whose flat identity Hessian ignores the geometry of probability distributions. Replacing 21∥x−xt∥22 by a general Bregman divergence Bψ(x,xt) leads to Online Mirror Descent (OMD), whose master regret bound (proved via continuous-time Bregman flow discretization in §2.1) governs the second-order penalty by the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12. Requiring (∇2ψ(x))−1=diag(x1,…,xn) on Δn uniquely forces negative Shannon entropy ψ(x)=∑i=1nxilnxi—whose inverse mirror map (∇ψ)−1 derives the exponential weight update of Multiplicative Weights Update (MWU / Hedge) and EXP3 (Online Stochastic Mirror Descent) from first principles, improving experts regret to O(Tlnn) and bandit regret to O(nTlnn)! Armed with MWU, we conclude by turning online no-regret play into a universal offline solver for von Neumann’s minimax theorem in zero-sum games and the Plotkin–Shmoys–Tardos (PST) oracle framework for approximate linear feasibility.
Reading guide. Course Content is organized into two 120-minute meetings following a first-principles progression:
Meeting 1 (§§1.1–1.5): Formulates the online number guessing game and proves the 2HT=Θ(logT) regret bound of Follow the Leader via the inductive Be-The-Leader lemma (§1.1); introduces the general online learning protocol and exhibits FTL’s Ω(T) oscillation on alternating linear losses (§1.2); develops offline convex optimization and proves the convergence of offline Gradient Descent and Projected Gradient Descent (§1.3); states the O(T) regret bound of Online Gradient Descent and proves its O(logT) refinement under μ-strong convexity (§1.4); and reduces discrete expert prediction (O(nT)) and online shortest paths (via probability flows) to OGD (§1.5).
Meeting 2 (§§1.6–1.11): Extends Euclidean OGD to adversarial multi-armed bandits via importance-weighted Online Stochastic Gradient Descent with α-uniform exploration (O((nT)2/3)) (§1.6); reformulates projected gradient descent via a local quadratic regularizer, diagnoses the (ℓ2,ℓ2) versus (ℓ1,ℓ∞) mismatch and flat boundary curvature on Δn, and introduces Bregman divergences (§1.7); develops Online Mirror Descent from four complementary perspectives (primal proximal, local Riemannian metric, dual mirror space, and FTRL accumulation) and states its master regret bound in terms of the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12 (§1.8); instantiates OMD with negative entropy ψ(x)=∑i=1nxilnxi to derive Multiplicative Weights Update (MWU / Hedge) (O(Tlnn)) and EXP3 / Online Stochastic Mirror Descent (O(nTlnn)) from first principles (§1.9); and applies MWU to prove von Neumann’s minimax theorem for zero-sum matrix games (§1.10) and the Plotkin–Shmoys–Tardos oracle theorem for linear feasibility (§1.11).
Reading Contents (§§2.1–2.5) presents the unified continuous-time discretization proof of Online Mirror Descent and Online Gradient Descent (§2.1), Fenchel conjugation and the generalized Pythagorean inequality (§2.2), Pinsker’s inequality (§2.3), minimum-congestion multicommodity routing via PST (§2.4), and online-to-batch conversion (§2.5). Further Topics (§§3.1–3.4) covers lazy versus greedy projection (FTRL vs. OMD), AdaBoost, Matrix Multiplicative Weights for SDPs, and Tsallis-INF.
I. Course Content
1.1 An online number guessing game and Follow the Leader
The multi-armed bandit problem of Chapter 1 was our first example of sequential decision-making under uncertainty. To see what changes when we move from stationary probability distributions to arbitrary adversarial sequences, let us begin with a simple quantitative game.
The quadratic number guessing game: stochastic versus adversarial
The online number guessing game lasts for T rounds. In each round t=1,2,…,T:
The adversary chooses a target number yt∈[0,1];
The player picks a guess xt∈[0,1] (without knowing yt);
The target yt is revealed, and the player pays the quadratic cost (xt−yt)2.
The player’s goal is to keep the cumulative cost ∑t=1T(xt−yt)2 as small as possible. How the player should act depends on how the sequence y1,…,yT is generated. Consider two contrasting regimes:
The stochastic setting: Suppose each yt is drawn independently from a fixed distribution D supported on [0,1]. Then in round t, for any choice xt∈[0,1] fixed before yt∼D is drawn, the expected single-round cost is EY∼D[(xt−Y)2]=xt2−2E[Y]xt+E[Y2]=(xt−E[Y])2+Var(Y), which is minimized at the population mean x∗:=E[Y]∈[0,1]. If D were known in advance, the optimal strategy would simply play xt=x∗ in every round. When D is unknown, no online strategy can beat in expectation the hindsight benchmark of committing to the best fixed point x∈[0,1], so it is natural to measure performance by the regret of not playing the best fixed point in hindsight: R(T):=t=1∑T(xt−yt)2−x∈[0,1]mint=1∑T(x−yt)2.
The adversarial setting: Now drop the distribution D completely: the adversary may choose yt∈[0,1] arbitrarily in each round—and because the player must commit to xt before seeing yt, we may assume that the adversary chooses ytafter observing xt. We still evaluate the player by the hindsight regret R(T)=t=1∑T(xt−yt)2−x∈[0,1]mint=1∑T(x−yt)2.
Why benchmark against a fixed comparator x∗? In an adversarial environment, no algorithm can guarantee a small raw cost ∑t=1T(xt−yt)2 in isolation: if the adversary sets yt=1 whenever xt≤1/2 and yt=0 whenever xt>1/2, the player pays at least T/4. Subtracting minx∈[0,1]∑t=1T(x−yt)2 calibrates the player’s loss against the best static compromise xT∗ in hindsight, connecting online play directly to offline optimization.
Hannan consistency and Blackwell approachability. James Hannan (Approximation to Bayes Risk in Repeated Play, 1957) proved the first O(T) external-regret guarantee in repeated games shortly after David Blackwell (1956) established his vector-valued Approachability Theorem. In game theory, an online strategy whose worst-case regret satisfies limsupT→∞R(T)/T≤0 is called Hannan consistent (or no-regret).
The Follow-the-Leader algorithm and the Be-The-Leader lemma
How should we play the number guessing game online? Let us first solve the offline optimization problem over the full horizon: given y1,…,yT∈[0,1], which fixed point x∈[0,1] minimizes FT(x):=∑t=1T(x−yt)2? Setting the derivative FT′(x)=2∑t=1T(x−yt) to zero yields the empirical mean: xT∗:=argx∈[0,1]mint=1∑T(x−yt)2=T1t=1∑Tyt. When round t begins in the online game, the player only knows the prefix y1,…,yt−1. If the game had ended at round t−1, the offline optimum would have been the prefix average xt−1∗:=argx∈[0,1]mins=1∑t−1(x−ys)2=t−11s=1∑t−1ys. A natural online heuristic is therefore to play xt=xt−1∗—that is, Follow the Leader (FTL) from the previous round! Remarkably, on quadratic losses, this simple greedy rule achieves O(logT) regret against any adversarial sequence.
Theorem 1 (Logarithmic regret of Follow the Leader on the number guessing game) Let y1,…,yT∈[0,1] be an arbitrary sequence of numbers. Suppose the player chooses x1∈[0,1] arbitrarily in round 1 and plays the Follow-the-Leader iterate xt:=xt−1∗=t−11s=1∑t−1ysfor t=2,…,T, where xt∗=t1∑s=1tys denotes the offline optimum through round t. Then R(T)=t=1∑T(xt−yt)2−x∈[0,1]mint=1∑T(x−yt)2≤2HT=Θ(logT), where HT:=∑t=1Tt1≤1+lnT is the T-th harmonic number.
Proof. We first establish a general structural inequality—often called the Be-The-Leader (BTL) inequality—stating that the cumulative cost of the non-adaptive global optimum xT∗ is at least the cumulative cost of the sequence of adaptive optima xt∗: t=1∑T(xT∗−yt)2≥t=1∑T(xt∗−yt)2. We prove this inequality by induction on T≥1. For T=1, both sides equal (x1∗−y1)2=0. For the inductive step from T−1 to T, note that the t=T term on both sides is identical, namely (xT∗−yT)2. Subtracting (xT∗−yT)2 from both sides shows that the claim for T is equivalent to t=1∑T−1(xT∗−yt)2≥t=1∑T−1(xt∗−yt)2. By the induction hypothesis at horizon T−1, and the fact that xT−1∗ minimizes ∑t=1T−1(x−yt)2 over all x∈[0,1] (in particular, it beats the candidate x=xT∗), we have t=1∑T−1(xt∗−yt)2≤t=1∑T−1(xT−1∗−yt)2≤t=1∑T−1(xT∗−yt)2, completing the induction.
Using the Be-The-Leader inequality, we can upper-bound the regret by comparing xt=xt−1∗ against xt∗ term by term: R(T)=t=1∑T(xt−yt)2−t=1∑T(xT∗−yt)2≤t=1∑T[(xt−yt)2−(xt∗−yt)2]. For t=1, since x1,y1∈[0,1] and x1∗=y1, the first term is (x1−y1)2≤1<12. For each round t=2,…,T, factoring the difference of squares with xt=xt−1∗ gives (xt−yt)2−(xt∗−yt)2=(xt−1∗+xt∗−2yt)(xt−1∗−xt∗)≤2xt−1∗−xt∗=2t−11s=1∑t−1ys−t1s=1∑tys=2t(t−1)1s=1∑t−1ys−t1yt=t2xt−1∗−yt≤t2, where we used xt−1∗,xt∗,yt∈[0,1] to bound ∣xt−1∗+xt∗−2yt∣≤2 and ∣xt−1∗−yt∣≤1. Summing from t=1 to T yields R(T)≤∑t=1Tt2=2HT.
1.2 General online learning and why Follow the Leader fails
We can immediately generalize the number guessing game to an arbitrary decision space V⊆Rd and general loss functions ℓt:V→R, obtaining the general framework of online learning (or online optimization).
Definition 1 (General online learning protocol and regret) Let V⊆Rd be a decision set and fix a horizon T≥1. In each round t=1,2,…,T:
The player chooses an action xt∈V (possibly at random from a distribution over V);
The adversary chooses and reveals a loss function ℓt:V→R;
The player pays the cost ℓt(xt).
The player’s regret over T rounds with respect to a fixed comparator x∗∈V and its worst-case regret over V are RegT(x∗):=t=1∑Tℓt(xt)−t=1∑Tℓt(x∗),RegT:=R(T)=t=1∑Tℓt(xt)−x∈Vmint=1∑Tℓt(x). When the player uses a randomized strategy, we measure expected regret E[R(T)].
The Follow-the-Leader (FTL) strategy applies verbatim to Definition 1: at round t, play an offline minimizer of the cumulative losses observed so far, xt:=xt−1∗∈argx∈Vmins=1∑t−1ℓs(x),wherext∗∈argx∈Vmins=1∑tℓs(x). Moreover, the inductive proof of the Be-The-Leader inequality in Theorem 1 holds for any sequence of functions ℓ1,…,ℓT:V→R: because xT−1∗ minimizes ∑t=1T−1ℓt(x) over V, induction on T gives ∑t=1Tℓt(xT∗)≥∑t=1Tℓt(xt∗), and therefore R(T)=t=1∑Tℓt(xt−1∗)−t=1∑Tℓt(xT∗)≤t=1∑T(ℓt(xt−1∗)−ℓt(xt∗)).
Look closely at the right-hand side: the regret of Follow the Leader is controlled by the sensitivity of the intermediate optimal solutions, ℓt(xt−1∗)−ℓt(xt∗)—how much the round-t loss changes when we replace the round-t hindsight leader xt∗ by the round-(t−1) leader xt−1∗. In the quadratic number guessing game, the curvature of (x−ys)2 forced the empirical means to stabilize at rate ∣xt−1∗−xt∗∣≤t1. What happens when the loss functions ℓt are linear and have no curvature to anchor xt∗?
For a linear objective (∑s=1t−1as)x on [−1,1], the minimizer xt−1∗=−sgn(∑s=1t−1as) is a step function that jumps across the entire interval whenever the cumulative slope crosses 0.
Example 1 (Failure of Follow the Leader on alternating linear losses) Consider the interval V=[−1,1] with linear loss functions ℓt(x)=atx, where a1=−21 and, for t>1, at={+1,−1,if t is even,if t is odd. Then the cumulative slope At−1:=∑s=1t−1as alternates sign at every single round: A1=−21, A2=+21, A3=−21, A4=+21,…. Because minimizing At−1x over x∈[−1,1] sets xt=+1 when At−1<0 (even t) and xt=−1 when At−1>0 (odd t>1), FTL plays xt=at and pays ℓt(xt)=at2=1 on every round t≥2, whereas the fixed point x∗=0∈[−1,1] pays 0 on every round!
The exact same oscillation occurs on the two-expert probability simplex Δ2={(p(1),p(2))∈R≥02:p(1)+p(2)=1} under ℓ1=(1/2,0) and alternating losses ℓt=(0,1) (even t), ℓt=(1,0) (odd t≥3):
Round t
Historical cumulative loss Lt−1
FTL distribution pt
Revealed loss ℓt
Learner’s loss ⟨pt,ℓt⟩
Updated Lt
t=1
(0,0)
(1/2,1/2)
(1/2,0)
1/4
(1/2,0)
t=2
(1/2,0)
(0,1)
(0,1)
1
(1/2,1)
t=3
(1/2,1)
(1,0)
(1,0)
1
(3/2,1)
t=4
(3/2,1)
(0,1)
(0,1)
1
(3/2,2)
t=5
(3/2,2)
(1,0)
(1,0)
1
(5/2,2)
Over an even horizon T, FTL suffers total loss 41+(T−1)=T−43, whereas the two fixed experts suffer LT(1)=2T−1 and LT(2)=2T, giving linear regret: RegT=(T−43)−2T−1=2T−41=Ω(T).
Caution (Why Follow the Leader cannot be our general online engine).
Follow the Leader suffers from two fundamental defects:
Boundary instability on linear losses: When the cumulative objective ∑s=1t−1ℓs(x) is linear (or weakly curved), its minimizer xt−1∗ jumps discontinuously between extreme points of V whenever two cumulative slopes cross, allowing the adversary to inflict Ω(1) sensitivity ℓt(xt−1∗)−ℓt(xt∗) on every round.
Offline computational overhead: Even when ∑s=1t−1ℓs(x) is curved, FTL requires solving the full offline optimization problem argminx∈V∑s=1t−1ℓs(x) from scratch at every step t, which can be non-trivial or computationally expensive.
Both defects point in the same direction: instead of jumping all the way to the global minimum of ∑s=1t−1ℓs(x) at round t, we should update our previous state xtincrementally using only local first-order information from the most recent loss ℓt.
1.3 Convex optimization and the offline Gradient Descent algorithm
Because solving the offline optimization problem argminx∈V∑s=1t−1ℓs(x) from scratch at each round is already non-trivial, let us first step back from the online game and study the offline convex optimization problem: x∈Vminf(x),where V⊆Rn is a convex set and f:V→R is a convex function. Recall that a subset V⊆Rn is convex if for all x,y∈V and λ∈[0,1], the line segment point λx+(1−λ)y lies in V. A function f:V→R is convex if its epigraphepi(f):={(x,y)∈V×R:y≥f(x)} is a convex subset of Rn+1.
Throughout this chapter, we rely on four fundamental properties of convex functions:
Jensen’s inequality: For any x,y∈V and λ∈[0,1], f(λx+(1−λ)y)≤λf(x)+(1−λ)f(y). More generally, for any random variable X∈V, f(E[X])≤E[f(X)].
First-order tangent-plane inequality: If f is differentiable at x∈V, then its graph lies everywhere above its tangent hyperplane at x: f(y)≥f(x)+⟨∇f(x),y−x⟩for all y∈V.
Second-order Taylor expansion with Lagrange remainder: If f∈C2(V), then for every x,y∈V there exists a point ξ on the segment [x,y] such that f(y)=f(x)+⟨∇f(x),y−x⟩+21(y−x)⊤∇2f(ξ)(y−x). In particular, f is convex on V if and only if its Hessian is positive semidefinite (∇2f(z)⪰0) for all z∈V.
First-order optimality condition: For a differentiable convex function f on a convex set V, x∗∈argx∈Rnminf(x)⟺∇f(x∗)=0,andx∗∈argx∈Vminf(x)⟺∀y∈V,⟨∇f(x∗),y−x∗⟩≥0.
The offline Gradient Descent algorithm
How can we find an approximate minimizer of f using only local first-order information? Let us first consider the unconstrained caseV=Rn. Starting from an initial point x0∈V, Gradient Descent iteratively updates the current point xt (t=0,1,…,T−1) by stepping in the direction of the negative gradient with step size η>0: xt+1:=xt−η∇f(xt).
−∇f(xₜ)x − xₜ
Because f is convex, −∇f(xt) always forms an acute angle with x∗−xt: ⟨−∇f(xt),x∗−xt⟩≥f(xt)−f(x∗)≥0.
Why is −∇f(xt) a good direction to move? Let x∗∈argminx∈Vf(x) be an optimal solution. By the first-order tangent-plane inequality (Property 2), f(x∗)≥f(xt)+⟨∇f(xt),x∗−xt⟩⟺⟨−∇f(xt),x∗−xt⟩≥f(xt)−f(x∗)≥0. Geometrically, the step direction −∇f(xt) always forms an acute angle with the displacement vector x∗−xt pointing from xt toward the optimum x∗! Consequently, moving a sufficiently small distance η>0 along −∇f(xt) decreases the Euclidean distance to x∗.
To make this quantitative, define the Euclidean potential functionϕ(x):=21∥x−x∗∥22. Tracking how ϕ changes in a single gradient step yields the fundamental convergence bound of offline Gradient Descent.
Theorem 2 (Convergence bound for offline Gradient Descent) Let f:Rn→R be a differentiable convex function, let x∗∈Rn be any comparator point with ϕ(x):=21∥x−x∗∥22, and fix a step size η>0. Then the Gradient Descent iterates xt+1=xt−η∇f(xt) satisfy, for every step t=0,1,…,T−1, f(xt)−f(x∗)≤⟨∇f(xt),xt−x∗⟩=ηϕ(xt)−ϕ(xt+1)+2η∇f(xt)22, and summing over t=0,1,…,T−1 yields t=0∑T−1(f(xt)−f(x∗))≤t=0∑T−1⟨∇f(xt),xt−x∗⟩≤ηϕ(x0)−ϕ(xT)+2ηt=0∑T−1∇f(xt)22.
Proof. Expanding the potential difference ϕ(xt+1)−ϕ(xt) for xt+1=xt−η∇f(xt) gives ϕ(xt+1)−ϕ(xt)=21xt−η∇f(xt)−x∗22−21∥xt−x∗∥22=21⟨−2η∇f(xt),xt−x∗⟩+21η2∇f(xt)22=−η⟨∇f(xt),xt−x∗⟩+21η2∇f(xt)22. Dividing by η>0 and rearranging gives the exact identity ⟨∇f(xt),xt−x∗⟩=ηϕ(xt)−ϕ(xt+1)+2η∥∇f(xt)∥22. Combining this with the tangent-plane inequality f(xt)−f(x∗)≤⟨∇f(xt),xt−x∗⟩ and telescoping the sum over t=0,…,T−1 completes the proof.
Averaging, Lipschitzness, and Projected Gradient Descent
Three remarks turn Theorem 2 into a complete offline optimization guarantee:
Averaging: Suppose the right-hand side of Theorem 2 is bounded by R(T), so that ∑t=0T−1(f(xt)−f(x∗))≤R(T). By Jensen’s inequality (Property 1), the averaged iterate xˉT:=T1∑t=0T−1xt∈V satisfies f(xˉT)−f(x∗)=f(T1t=0∑T−1xt)−f(x∗)≤T1t=0∑T−1(f(xt)−f(x∗))≤TR(T). Whenever R(T)=o(T), the optimality gap f(xˉT)−f(x∗) converges to 0 as T→∞.
Lipschitzness of f: To bound the second-order sum ∑t=0T−1∥∇f(xt)∥22, assume f is L-Lipschitz in the sense that ∥∇f(xt)∥2≤L for all t. Since ϕ(xT)≥0 and ϕ(x0)=21∥x0−x∗∥22, Theorem 2 yields t=0∑T−1(f(xt)−f(x∗))≤2η∥x0−x∗∥22+2ηL2T=∥x0−x∗∥2⋅LT upon choosing the optimal step size η=LT∥x0−x∗∥2, which implies f(xˉT)−f(x∗)≤T∥x0−x∗∥2⋅L.
Constrained case: Projected Gradient Descent (PGD): When V⊊Rn is a constrained closed convex set, the raw gradient step yt+1:=xt−η∇f(xt) may step outside V. To restore feasibility, we apply the Euclidean projection operatorΠV(y):=argminx∈V∥x−y∥2, obtaining Projected Gradient Descent: xt+1:=ΠV(xt−η∇f(xt)). Why does projection not hurt our potential analysis? Because projecting onto a convex set V only moves the point closer to every point x∗∈V:
Lemma 1 (Pythagorean contraction of Euclidean projection) Let V⊆Rn be a closed convex set and let xt+1=ΠV(y) be the Euclidean projection of y∈Rn onto V. Then for every x∗∈V, ⟨y−xt+1,x∗−xt+1⟩≤0and consequentlyϕ(ΠV(y))=21ΠV(y)−x∗22≤21∥y−x∗∥22=ϕ(y).
Proof. By definition of projection, xt+1=argminx∈Vg(x) for the convex function g(x):=∥x−y∥22 with gradient ∇g(x)=2(x−y). The first-order optimality condition on V (Property 4) gives ⟨∇g(xt+1),x∗−xt+1⟩=2⟨xt+1−y,x∗−xt+1⟩≥0, or ⟨y−xt+1,x∗−xt+1⟩≤0. Expanding ∥y−x∗∥22 around xt+1 then gives ∥y−x∗∥22=(y−xt+1)+(xt+1−x∗)22=∥y−xt+1∥22+∥xt+1−x∗∥22−2⟨y−xt+1,x∗−xt+1⟩≥∥xt+1−x∗∥22.
Applying Lemma 1 to y=xt−η∇f(xt) gives ϕ(xt+1)−ϕ(xt)≤ϕ(xt−η∇f(xt))−ϕ(xt), so the convergence bound of Theorem 2 holds for constrained Projected Gradient Descent (with the single-step equality in Theorem 2 replaced by ≤)!
1.4 Online Gradient Descent and strong convexity
Now let us return to the online setting! When the decision set V⊆Rn is a closed convex set and each round’s loss function ℓs:V→R (s=0,1,…,T−1) is convex and differentiable, the online learning game (Definition 1) is called Online Convex Optimization (OCO).
Linearization reduction and the Online Projected Gradient Descent algorithm
Applying the first-order tangent-plane inequality ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩ at each round s immediately reduces general convex losses to linear losses:
Lemma 2 (Linearization reduction) Let V⊆Rn be a convex set and let ℓ0,…,ℓT−1:V→R be differentiable convex loss functions. Then for any sequence of iterates x0,…,xT−1∈V and every comparator x∗∈V, R(T)=s=0∑T−1(ℓs(xs)−ℓs(x∗))≤s=0∑T−1⟨∇ℓs(xs),xs−x∗⟩.
This motivates the online counterpart of Projected Gradient Descent, called Online Projected Gradient Descent (OGD): starting from an initial point x0∈V, at each round s=0,1,…,T−1 the player plays xs∈V, observes the gradient ∇ℓs(xs) of the revealed convex loss ℓs, and updates with step size ηs>0: xs+1:=ΠV(xs−ηs∇ℓs(xs)).
Look back at the single-step proof of Theorem 2 and Lemma 1 in §1.3: at step s, the inequality ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩≤ηsϕ(xs)−ϕ(xs+1)+2ηs∥∇ℓs(xs)∥22 only used the convexity of the single function ℓs at xs—it never required the functions across different rounds to be identical! Consequently, Online Projected Gradient Descent satisfies the exact same regret bound against any adversarial sequence of convex losses ℓ0,…,ℓT−1.
Theorem 3 (Regret bound for Online Projected Gradient Descent) Let V⊆Rn be a closed convex set and let ℓ0,…,ℓT−1:V→R be differentiable convex loss functions. For any initial point x0∈V, any comparator x∗∈V with ϕ(x)=21∥x−x∗∥22, and any step sizes ηs>0, the Online Projected Gradient Descent iterates xs+1=ΠV(xs−ηs∇ℓs(xs)) satisfy, for every round s=0,…,T−1, ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩≤ηsϕ(xs)−ϕ(xs+1)+2ηs∇ℓs(xs)22. In particular, if V has Euclidean diameter diam(V):=maxx,y∈V∥x−y∥2, each ℓs is L-Lipschitz (∥∇ℓs(xs)∥2≤L), and we take constant step size ηs=η=LTdiam(V), then R(T)=s=0∑T−1(ℓs(xs)−ℓs(x∗))≤ηϕ(x0)−ϕ(xT)+2ηs=0∑T−1∇ℓs(xs)22≤2ηdiam(V)2+2ηL2T=diam(V)⋅LT.
Continuous-time interpretation of Theorem 3 (see §2.1). Where does the second-order penalty 2ηs∥∇ℓs(xs)∥22 come from? In Reading Contents §2.1, we prove a general continuous-time discretization theorem for Online Mirror Descent (which includes Theorem 3 as the special case ψ(x)=21∥x∥22): a continuous gradient flow dtdyt=−gt pays only the potential drop ϕ(x0)−ϕ(xT) with zero second-order loss, while 2ηs∥∇ℓs(xs)∥22 is the exact discretization lag incurred because the discrete player freezes at xs during an interval of length ηs.
Strongly convex losses: recovering O(logT) regret on curved objectives
Notice an intriguing comparison: when we apply constant-step OGD (Theorem 3) to the online number guessing game ℓs(x)=(x−ys)2 on V=[0,1], we get O(T) regret—which is much worse than the 2HT=Θ(logT) bound achieved by Follow the Leader (Theorem 1)! Why did Theorem 3 lose so much on (x−ys)2?
Look at the first inequality in Theorem 3: we used the tangent-plane bound ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩, which is tight only when ℓs is linear. When ℓs is strictly curved, the tangent hyperplane lies strictly below ℓs(x∗) by a quadratic margin!
Definition 2 (Euclidean strong convexity) A differentiable function f:V→R on a convex set V⊆Rn is μ-strongly convex (for a parameter μ>0) if for all x,y∈V, f(y)≥f(x)+⟨∇f(x),y−x⟩+2μ∥y−x∥22.
Retaining this −2μ∥xs−x∗∥22 curvature bonus and decaying the step size as ηs=μ(s+1)1 allows adjacent potential terms to telescope completely, recovering O(logT) regret.
Theorem 4 (Logarithmic regret of OGD for strongly convex losses) Suppose each loss function ℓs:V→R (s=0,1,…,T−1) is μ-strongly convex and L-Lipschitz (∥∇ℓs(xs)∥2≤L). Then Online Projected Gradient Descent with decreasing step sizes ηs:=μ(s+1)1(s=0,1,…,T−1) satisfies, for every x∗∈V, R(T)=s=0∑T−1(ℓs(xs)−ℓs(x∗))≤2μL2HT=O(logT).
Proof. By μ-strong convexity of ℓs and the single-round OGD inequality from Theorem 3, for each s=0,…,T−1, ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩−2μ∥xs−x∗∥22≤ηsϕ(xs)−ϕ(xs+1)+2ηs∇ℓs(xs)22−2μ∥xs−x∗∥22=(2ηs1−2μ)∥xs−x∗∥22−2ηs1∥xs+1−x∗∥22+2ηs∇ℓs(xs)22. Summing this inequality over s=0,1,…,T−1 and grouping the coefficients of ∥xs−x∗∥22 gives s=0∑T−1(ℓs(xs)−ℓs(x∗))≤(2η01−2μ)∥x0−x∗∥22−2ηT−11∥xT−x∗∥22+s=1∑T−1(2ηs1−2ηs−11−2μ)∥xs−x∗∥22+s=0∑T−12ηs∇ℓs(xs)22. Substituting ηs1=μ(s+1) makes 2η01−2μ=0 and 2ηs1−2ηs−11−2μ=0 for every s≥1! Dropping the nonpositive terminal term −2ηT−11∥xT−x∗∥22 and using ∥∇ℓs(xs)∥2≤L yields s=0∑T−1(ℓs(xs)−ℓs(x∗))≤s=0∑T−12μ(s+1)L2=2μL2HT.
Matching FTL on the number guessing game. On ℓs(x)=(x−ys)2 over V=[0,1] with ys∈[0,1], we have ℓs′′(x)=2 (μ=2), ∣ℓs′(x)∣=2∣x−ys∣≤2 (L=2), and step size ηs=2(s+1)1, so xs+1=xs−s+11(xs−ys)=s+1sxs+s+11ys=s+11∑i=0syi is literally the running average of y0,…,ys, and Theorem 4 gives R(T)≤2⋅222HT=HT, matching Follow the Leader!
1.5 Reducing discrete problems to OGD: experts and online shortest paths
At first glance, Online Projected Gradient Descent (Theorem 3) applies only when the action space V⊆Rn is a continuous convex body. Remarkably, many discrete combinatorial online problems can be transformed into convex optimization problems by randomization—lifting a discrete set of choices into a convex space of probability distributions. Let us examine two canonical reductions.
1. Learning with expert advice lifted to the probability simplex
In the learning with expert advice problem, the discrete action set is V=[n]={1,…,n}: we treat each i∈[n] as an expert, and in each round t=1,…,T, the player picks one expert it∈[n] to follow while the adversary reveals a loss vector ℓt∈[0,1]n. Clearly V=[n] is finite and not convex, so gradient descent cannot be applied directly on [n]. Worse still, no deterministic strategy can achieve sublinear regret!
Contrast: the Halving algorithm when a perfect expert exists. If each expert announces a binary prediction y^t(i)∈{0,1} and at least one expert i∗∈[n] makes zero mistakes (ℓt(i∗)=0∀t), predicting the majority vote among surviving zero-mistake experts eliminates at least half of the survivors on every mistake, making at most ⌊log2n⌋ mistakes in total. Once every expert can make mistakes, however, eliminating erring experts empties the pool, and Example 2 forces randomization.
Example 2 (Failure of every deterministic strategy) Suppose n=2 (or general n≥2) and the player chooses an expert it∈[n] deterministically in each round t. Because the adversary knows the player’s deterministic rule, it can set ℓt(j)={1,0,if j=it,otherwise. After T rounds, the player suffers cumulative loss T, whereas the sum of cumulative losses across all n experts is ∑j=1n∑t=1Tℓt(j)=T, so the best expert in hindsight suffers at most T/n (at most T/2 when n=2). Thus every deterministic strategy suffers linear regret RegT≥(1−1/n)T=Ω(T).
Therefore, a randomized strategy is mandatory: in each round t, instead of committing to a single expert, the player chooses a probability distribution xt in the (n−1)-dimensional probability simplex Δn:={x=(x1,…,xn)∈[0,1]n:i=1∑nxi=1}, draws an action At∼xt, and pays expected loss EAt∼xt[ℓt(At)]=∑i=1nxt(i)ℓt(i)=⟨ℓt,xt⟩.
Definition 3 (Prediction with expert advice on Δn) Fix n≥2 experts and horizon T≥1. For each round t=1,…,T:
The player selects a probability distribution xt∈Δn;
The adversary chooses a loss vector ℓt∈[0,1]n;
The player samples At∼xt, incurs loss ℓt(At), and observes the full vector ℓt.
The expected regret with respect to a fixed expert j∈[n] and the worst-case expected regret are E[t=1∑T(ℓt(At)−ℓt(j))]=E[t=1∑T⟨ℓt,xt−ej⟩],RegT:=j∈[n]maxE[t=1∑T⟨ℓt,xt−ej⟩], where ej∈Δn is the j-th standard basis vector.
Because Δn⊆Rn is compact and convex and the expected loss x↦⟨ℓt,x⟩ is linear in x with gradient ∇⟨ℓt,x⟩=ℓt, learning with expert advice is now an instance of Online Convex Optimization on Δn! Running Online Projected Gradient Descent from the uniform distribution x1=(n1,…,n1) with update xt+1=ΠΔn(xt−ηℓt) immediately yields a sublinear regret bound via Theorem 3: since ∥x1−ej∥22=(1−n1)2+(n−1)n21=1−n1≤1 (and diam(Δn)=2) while ∥ℓt∥2=∑i=1nℓt(i)2≤n for ℓt∈[0,1]n, Theorem 3 gives E[t=1∑T(ℓt(At)−ℓt(j))]≤2η1+2ηnT=nT(choosing η=nT1).
2. Online shortest paths via probability flows
What if the number of discrete experts N is itself exponentially large in the natural problem size?
From edge flow p to path distribution. A probability flow p:E→[0,1] on a DAG G=(V,E) satisfies y:(x,y)∈E∑p(x,y)=y:(y,x)∈E∑p(y,x)(∀x∈V∖{u,v}) and ∑y:(u,y)∈Ep(u,y)=∑y:(y,v)∈Ep(y,v)=1. At each vertex x with positive outflow, p induces a transition distribution over out-neighbors: p^(x,y):=∑z:(x,z)∈Ep(x,z)p(x,y). Sampling a random walk from u to v according to p^ generates a path P∈Pu,v with Pr[P]=∏e∈Pp^(e) whose marginal probability of traversing any edge e∈E is exactly Pr[e∈P]=p(e)!
Example 3 (Online shortest paths via the probability flow polytope) Let G=(V,E) be a directed acyclic graph and fix two vertices u,v∈V. In each round t=1,…,T:
The player picks a directed path Pt∈Pu,v, where Pu,v is the set of all simple paths from u to v;
The adversary reveals edge weights wt:E→[0,1];
The player pays the path length ℓt(Pt):=∑e∈Ptwt(e).
We want to minimize regret against the shortest fixed path in hindsight, P∗∈argminP∈Pu,v∑t=1Tℓt(P).
If we treat each path P∈Pu,v as an independent expert in Δ∣Pu,v∣, the vector dimension ∣Pu,v∣ can be 2Ω(∣V∣), making it computationally infeasible to even write down the distribution vector! However, because the loss of a path ∑e∈Pwt(e) is additive over edges, the expected loss of any distribution over Pu,v depends only on the marginal probability p(e):=Pr[e∈P]∈[0,1] that each edge e∈E is traversed: EP[ℓt(P)]=e∈E∑Pr[e∈P]wt(e)=e∈E∑p(e)wt(e)=⟨wt,p⟩. As shown in the margin note, the set of all valid edge-marginal vectors p∈[0,1]∣E∣ is precisely the unit u–v probability flow polytopeP⊆R∣E∣, defined by ∣V∣ linear flow-conservation equations and ∣E∣ box inequalities 0≤p(e)≤1. We can therefore run OGD directly on the ∣E∣-dimensional convex polytope P and sample Pt in O(∣V∣+∣E∣) time per round!
1.6 Adversarial multi-armed bandits via Online Stochastic Gradient Descent
We open our second meeting by returning to the multi-armed bandit problem from Chapter 1! Compare the stochastic bandit model of Chapter 1 with the full-information expert model of §1.5:
In §1.5, the adversary chooses ℓt∈[0,1]n arbitrarily, and after pulling arm At∼xt, the player observes the entire vectorℓt=(ℓt(1),…,ℓt(n)).
In a bandit problem, after pulling arm At∼xt, the player observes only the single numberℓt(At)∈[0,1] of the pulled arm, and never sees ℓt(i) for i=At.
When we drop Chapter 1’s i.i.d. assumption and allow the adversary to choose the loss vectors ℓ1,…,ℓT∈[0,1]n arbitrarily while restricting observation to the single coordinate ℓt(At), we obtain the adversarial multi-armed bandit problem.
Definition 4 (Adversarial multi-armed bandit protocol) Fix n≥2 arms and a horizon T≥1. For each round t=1,…,T:
Based on the past history Ft−1=(A1,ℓ1(A1),…,At−1,ℓt−1(At−1)), the player chooses a distribution xt∈Δn and samples an arm At∼xt;
The adversary chooses ℓt∈[0,1]n (measurable with respect to Ft−1), and the player pays and observes only the scalar lossℓt(At)∈[0,1].
The player’s expected bandit regret (often called pseudo-regret when maximizing over fixed arms j∈[n] outside the expectation, which coincides with E[R(T)] against an oblivious adversary) is RegTbandit:=j∈[n]maxE[t=1∑T(ℓt(At)−ℓt(j))]=j∈[n]maxt=1∑TE[⟨ℓt,xt−ej⟩].
Guessing the unseen loss vector via importance weighting
How can we apply gradient-based online learning when n−1 coordinates of the gradient ℓt are hidden? The key idea is to construct a random vector ℓ^t∈R≥0n from the single observation (At,ℓt(At)) that is an unbiased estimator of ℓt (E[ℓ^t∣Ft−1]=ℓt), and feed ℓ^t into our online optimization algorithm in place of ℓt!
Dividing by xt(i) keeps the expected rectangle area xt(i)⋅(ℓt(i)/xt(i))=ℓt(i) unbiased, but a tiny sampling probability xt(i) creates a spike of height ℓt(i)/xt(i) whose second moment scales as ℓt(i)2/xt(i).
Specifically, for every arm i∈[n] with xt(i)>0, define the importance-weighted (Horvitz–Thompson) estimator: ℓ^t(i):=xt(i)1{At=i}ℓt(i)=⎩⎨⎧xt(At)ℓt(At),0,if i=At,if i=At. Notice that ℓ^t is completely computable from the bandit observation (At,ℓt(At)) and the known distribution xt. Let us record its conditional first and second moments given Ft−1.
Lemma 3 (Conditional mean and second moments of importance-weighted losses) Let Ft−1 denote the σ-algebra of the first t−1 rounds, so that xt∈int(Δn) and ℓt∈[0,1]n are Ft−1-measurable. Under the random draw At∼xt:
Conditional unbiasedness: For every i∈[n], E[ℓ^t(i)∣Ft−1]=ℓt(i), and therefore for every comparator x∗∈Δn, E[⟨ℓt,xt−x∗⟩]=E[⟨E[ℓ^t∣Ft−1],xt−x∗⟩]=E[E[⟨ℓ^t,xt−x∗⟩Ft−1]]=E[⟨ℓ^t,xt−x∗⟩].
Euclidean second moment versus xt-weighted second moment: For every i∈[n], E[ℓ^t(i)2∣Ft−1]=xt(i)ℓt(i)2, so E[∥ℓ^t∥22Ft−1]=i=1∑nxt(i)(xt(i)ℓt(i))2=i=1∑nxt(i)ℓt(i)2≤i=1∑nxt(i)1, whereas the xt-weighted sum of squares satisfies E[i=1∑nxt(i)ℓ^t(i)2Ft−1]=i=1∑nxt(i)⋅xt(i)ℓt(i)2=i=1∑nℓt(i)2≤n.
Proof: tower property and indicator moments
Conditional on Ft−1, the indicator 1{At=i} takes value 1 with probability xt(i) and 0 with probability 1−xt(i). Hence E[ℓ^t(i)∣Ft−1]=xt(i)⋅xt(i)ℓt(i)+(1−xt(i))⋅0=ℓt(i) and E[ℓ^t(i)2∣Ft−1]=xt(i)⋅(xt(i)ℓt(i))2+(1−xt(i))⋅0=xt(i)ℓt(i)2. Because xt−x∗ is Ft−1-measurable, linearity of conditional expectation and the law of total expectation give E[⟨ℓt,xt−x∗⟩]=E[⟨ℓ^t,xt−x∗⟩]. Summing E[ℓ^t(i)2∣Ft−1] with and without the factor xt(i) yields the two second-moment formulas.
Online Stochastic Gradient Descent (OSGD) and the O((nT)2/3) barrier
By Part 1 of Lemma 3, we can analyze the gradient descent update xt+1=ΠΔn(xt−ηℓ^t)—called Online Stochastic Gradient Descent (OSGD)—as if the random estimator ℓ^t were the true loss vector! Taking expectations in the Euclidean OGD bound (Theorem 3) with x1=(n1,…,n1) gives, for every arm j∈[n], E[t=1∑T⟨ℓt,xt−ej⟩]=E[t=1∑T⟨ℓ^t,xt−ej⟩]≤2η1+2ηt=1∑TE[∥ℓ^t∥22]≤2η1+2ηt=1∑TE[i=1∑nxt(i)1].
Caution (Variance blow-up of Euclidean OSGD at the simplex boundary).
Look at the second term ∑i=1nxt(i)1: under Euclidean projection ΠΔn, we have no lower bound on xt(i)—if an arm i looks bad early on, ΠΔn can push xt(i) arbitrarily close to 0 (or to 0 itself), causing the variance E[∥ℓ^t∥22∣Ft−1]=∑i=1nxt(i)ℓt(i)2 to explode!
To prevent 1/x~t(i) from blowing up under Euclidean OSGD, we are forced to mix xt with the uniform distribution u:=(n1,…,n1) using an explicit exploration parameter α∈(0,1]:
Initialize x1=u=(n1,…,n1).
In each round t=1,…,T:
Mix x~t:=(1−α)xt+αu, and sample At∼x~t;
Observe ℓt(At) and compute ℓ^t(i):=x~t(i)1{At=i}ℓt(i) for i∈[n];
Update xt+1:=ΠΔn(xt−ηℓ^t).
Theorem 5 (Regret bound for Euclidean OSGD with uniform exploration) In the adversarial multi-armed bandit problem (Definition 4), running Euclidean OSGD with exploration parameter α=(4Tn2)1/3 and step size η=241/3(nT)−2/3 (for T≥n2/4) satisfies RegTbandit≤2η1+2αηn2T+αT=3⋅4−1/3(nT)2/3=O((nT)2/3).
Proof. Fix any comparator arm j∈[n]. Since the player pulls At∼x~t=(1−α)xt+αu and E[ℓ^t∣Ft−1]=ℓt, the expected regret decomposes into the OGD regret of xt plus the exploration bias of αu: E[t=1∑T⟨ℓt,x~t−ej⟩]=E[t=1∑T⟨ℓ^t,(1−α)xt+αu−ej⟩]=E[t=1∑T⟨ℓ^t,xt−ej⟩]+αE[t=1∑T⟨ℓ^t,u−xt⟩]≤2η1+2ηt=1∑TE[i=1∑nx~t(i)(x~t(i)ℓt(i))2]+αE[t=1∑T⟨ℓt,u⟩]. Because x~t(i)=(1−α)xt(i)+nα≥nα and ℓt(i)∈[0,1], each inner sum is bounded by ∑i=1nx~t(i)1≤n⋅αn=αn2, while ⟨ℓt,u⟩≤1. Therefore, RegTbandit≤2η1+2αηn2T+αT. Balancing the three terms by setting 2η1=2αηn2T=αT gives η=241/3(nT)−2/3, α=(4Tn2)1/3, and total regret 3⋅4−1/3(nT)2/3.
1.7 From quadratic regularizer to Bregman divergence and the simplex mismatch
Stepping back from §§1.5–1.6, we now face two sharp puzzles left behind by Euclidean Online Gradient Descent on the probability simplex Δn:
Full-information experts (§1.5): Why did Euclidean OGD pay O(nT)—growing polynomially with the number of experts n—when we will soon see that O(Tlnn) is achievable?
Adversarial bandits (§1.6): Why did Euclidean OSGD degrade from T1/2 to O((nT)2/3), forced to inject α-uniform exploration because E[∥ℓ^t∥22∣Ft−1]=∑i=1nxt(i)ℓt(i)2 blew up near the boundary?
To find the common culprit behind both inefficiencies, let us rewrite the projected gradient descent step xt+1=ΠV(xt−η∇f(xt)) as a local optimization problem.
Proposition 1 (Proximal quadratic regularizer view of Projected Gradient Descent) For any closed convex set V⊆Rn, any point xt∈V, and any step size η>0, the projected gradient descent update xt+1=ΠV(xt−η∇f(xt)) is equivalent to xt+1=argx∈Vmin{⟨∇f(xt),x−xt⟩+2η1∥x−xt∥22}.
Proof. Expanding the squared Euclidean distance in the definition of ΠV and dropping terms that do not depend on the optimization variable x∈V, we obtain ΠV(xt−η∇f(xt))=argx∈Vminxt−η∇f(xt)−x22=argx∈Vmin{⟨x,x⟩−2⟨x,xt−η∇f(xt)⟩}=argx∈Vmin{2η⟨∇f(xt),x−xt⟩+∥x−xt∥22}, and dividing the objective by 2η>0 gives the claimed formula.
Look at the intuition revealed by Proposition 1: at step t, the only information we use about f is its gradient ∇f(xt), which defines the local linear approximation L(x):=⟨∇f(xt),x−xt⟩. Because minimizing L(x) alone is unbounded (minL(x)=−∞ when V=Rn, or jumps to a corner when V=Δn) and L(x)≈f(x) holds only for x near xt, we add a proximity regularizer2η1∥x−xt∥22 to keep xt+1 close to xt.
In Rn, the unit ℓ∞ cube [−1,1]n has ℓ∞ radius 1, but its corners (1,…,1) lie at Euclidean distance n from the origin. Measuring ℓt∈[0,1]n in ∥⋅∥2 pays this n diagonal factor.
Example 4 (Why the quadratic regularizer 21∥x−xt∥22 is mismatched to Δn) Choosing the quadratic function 21∥x−xt∥22 in Proposition 1 was purely for algebraic convenience, and on the simplex Δn it fails in two ways:
The (ℓ2,ℓ2) versus (ℓ1,ℓ∞) norm mismatch and the dual role of the distance function: Recall how the single-step OGD regret decomposition (Theorem 3) uses the same distance function 21∥y−x∥22 in two places simultaneously: ⟨ℓt,xt−x∗⟩≤telescopes over T rounds to 2η∥x1−x∗∥22η21∥xt−x∗∥22−21∥xt+1−x∗∥22+non-telescoping per-round remainder ≤2η∥ℓt∥22(⟨ℓt,xt−xt+1⟩−2η1∥xt−xt+1∥22).
In the non-telescoping remainder on the right, because ℓt∈[0,1]n is bounded coordinatewise in ∥ℓt∥∞≤1, measuring ℓt in ∥⋅∥2 via Cauchy–Schwarz pays ∥ℓt∥22≤n∥ℓt∥∞2≤n. To replace ∥ℓt∥22 by ∥ℓt∥∞2≤1 via Hölder’s inequality ⟨ℓt,xt−xt+1⟩≤∥ℓt∥∞∥xt−xt+1∥1, the regularizer divergence must lower-bound 21∥xt+1−xt∥12.
In the telescoping potential drop on the left, requiring the divergence to lower-bound 21∥x∗−x1∥12 would blow up by Θ(n) if V were the hypercube [−1,1]n—except that on the probability simplex Δn, every point satisfies ∥x∥1=∑i=1nxi=1, so the primal ℓ1 diameter ∥x∗−x1∥1≤2 is already O(1)! Because Δn is ℓ1-bounded in the primal space while ℓt∈[0,1]n is ℓ∞-bounded in the dual space, switching to an ℓ1-compatible regularizer wins on both ends at once.
Flat boundary curvature (ψ′′(xi)≡1) and local spring stiffness: How does a regularizer ψ control the local step size Δx:=xt+1−xt? In one dimension, the first-order Taylor remainder Bψ(x,xt):=ψ(x)−ψ(xt)−ψ′(xt)(x−xt) has second-order Taylor approximation Bψ(x,xt)≈21ψ′′(xt)(Δx)2 near xt. Minimizing gtΔx+2η1ψ′′(xt)(Δx)2 over Δx yields Δx=xt+1−xt≈−ψ′′(xt)ηgt,gt(xt−xt+1)≈ηψ′′(xt)gt2. Geometrically, the second derivative ψ′′(xt) acts as the local spring stiffness (damping coefficient) anchoring xt. The Euclidean regularizer ψ(x)=21∑i=1nxi2 has constant stiffness ψ′′(xi)≡1 everywhere on Δn, penalizing a step near the cliff xi≈0 no more than at xi=1/n; when bandit importance sampling produces a spike ℓ^t(i)=ℓt(i)/xt(i), the flat Euclidean bowl cannot brake the step and suffers E[ℓ^t(i)2/ψ′′(xt(i))]=ℓt(i)2/xt(i)→∞. Designing a regularizer whose stiffness diverges as ψ′′(xi)=xi1→+∞ as xi→0+ automatically scales steps as Δxi≈−ηxt(i)gt(i) (preventing boundary crashes) and cancels the 1/xt(i) bandit variance blow-up: E[ℓ^t(i)2/ψ′′(xt(i))]=xt(i)ℓt(i)2⋅xt(i)=ℓt(i)2≤1!
Bregman divergence and the two canonical geometries
How do we generalize 21∥x−xt∥22 to an arbitrary strictly convex differentiable regularizer ψ:D→R? Notice that 21∥y−x∥22=21∥y∥22−[21∥x∥22+⟨x,y−x⟩] is simply the difference between ψ(y) and its first-order Taylor approximation at x.
Definition 5 (Bregman divergence and strong convexity) Let ψ:D→R be continuously differentiable and strictly convex on a convex domain D⊆Rn containing V.
For any y∈D and x∈int(D), the Bregman divergence of ψ is defined by Bψ(y,x)≡Dψ(y,x):=ψ(y)−[ψ(x)+⟨∇ψ(x),y−x⟩]≥0.
Given a norm ∥⋅∥ on Rn with dual norm ∥g∥∗=sup∥v∥≤1⟨g,v⟩, we say that ψ is μ-strongly convex with respect to ∥⋅∥ on V if Bψ(y,x)≥2μ∥y−x∥2 for all x,y∈V.
Let us compute Bψ(y,x) and the Hessian ∇2ψ(x) for our two flagship regularizers:
Squared Euclidean norm ψ(x)=21∥x∥22 on Rn: Here ∇ψ(x)=x, ∇2ψ(x)=I, and Bψ(y,x)=21∥y∥22−[21∥x∥22+⟨x,y−x⟩]=21∥y−x∥22.
Negative Shannon entropy ψ(x)=∑i=1nxilnxi on R≥0n: Recall that for a distribution p∈Δn, its Shannon entropy is H(p):=−∑i=1npilnpi. For ψ(x)=∑i=1nxilnxi, differentiating gives (∇ψ(x))i=1+lnxi and the diagonal Hessian ∇2ψ(x)i,j=⎩⎨⎧xi1,0,if i=j,otherwise. Its Bregman divergence on R>0n is Bψ(y,x)=i=1∑nyilnyi−i=1∑nxilnxi−i=1∑n(yi−xi)(1+lnxi)=i=1∑nyilnxiyi−i=1∑nyi+i=1∑nxi. When restricted to probability distributions x,y∈Δn (so ∑i=1nyi=∑i=1nxi=1), the linear terms cancel and Bψ(y,x) becomes the Kullback–Leibler (KL) divergence (relative entropy): Bψ(y,x)=i=1∑nyilnxiyi=KL(y∥x).
Although Bψ(y,x) is in general asymmetric (Bψ(y,x)=Bψ(x,y)), it obeys an exact three-point identity that generalizes the Euclidean law of cosines.
Lemma 4 (Three-point identity for Bregman divergences) For any x∈D and y,z∈int(D), ⟨∇ψ(z)−∇ψ(y),x−y⟩=Bψ(x,y)−Bψ(x,z)+Bψ(y,z).
Proof: expanding the three Bregman divergences
Expanding the right-hand side directly from Definition 5, Bψ(x,y)−Bψ(x,z)+Bψ(y,z)=(ψ(x)−ψ(y)−⟨∇ψ(y),x−y⟩)−(ψ(x)−ψ(z)−⟨∇ψ(z),x−z⟩)+(ψ(y)−ψ(z)−⟨∇ψ(z),y−z⟩)=−⟨∇ψ(y),x−y⟩+⟨∇ψ(z),(x−z)−(y−z)⟩=⟨∇ψ(z)−∇ψ(y),x−y⟩.
1.8 Online Mirror Descent and its master regret bound
Now replace the quadratic proximity regularizer 2η1∥x−xt∥22 in Proposition 1 by the Bregman divergence η1Bψ(x,xt) of a strictly convex regularizer ψ. The resulting updating rule is Online Mirror Descent (OMD): xt+1:=argx∈Vmin{⟨∇f(xt),x−xt⟩+η1Bψ(x,xt)}.
Four complementary perspectives on Online Mirror Descent
Why is this algorithm called Mirror Descent, and how does it cure the simplex bottlenecks of §1.7? We can understand the update above from four complementary viewpoints:
Primal proximal view: Directly replacing the flat Euclidean bowl 21∥x−xt∥22 in Proposition 1 by the asymmetric Bregman potential well Bψ(x,xt) penalizes candidate moves x−xt according to the geometry of ψ.
Local Riemannian metric view: By a second-order Taylor expansion at xt, there exists ξ∈[xt,x] such that Bψ(x,xt)=21(x−xt)⊤∇2ψ(ξ)(x−xt)≈21∥x−xt∥∇2ψ(xt)2. Thus ψ equips the primal space V with a position-dependent Riemannian metric tensor G(xt)=∇2ψ(xt). When ∂xi2∂2ψ→+∞ and ∂xi∂ψ→−∞ near the boundary xi→0+, the diverging curvature strongly penalizes local coordinate steps toward 0 while the diverging mirror coordinate maps finite dual steps back to strictly positive coordinates, preventing iterates from crashing into 0 without artificial α-truncation.
Dual mirror-space view (why Mirror Descent?): Mathematically, primal states xt∈V and gradients ∇f(xt)∈V∗ (linear functionals on V acting via ⟨∇f(xt),x⟩) live in dual vector spaces. Writing xt−η∇f(xt) in Euclidean OGD only made sense because the identity gradient ∇(21∥x∥22)=x tacitly identified V with V∗. In non-Euclidean geometries (such as ℓ1 probabilities versus ℓ∞ loss prices), Mirror Descent first maps xt∈V to its dual representation Zt=∇ψ(xt)∈V∗ via the mirror map∇ψ:V→V∗, takes the linear gradient step Zt+1=Zt−η∇f(xt) inside V∗, maps back via the inverse mirror map xt+1′=(∇ψ)−1(Zt+1)=∇ψ∗(Zt+1), and projects onto V under Bψ.
FTRL historical-accumulation view: When V=Rn is unconstrained (or on Δn under negative entropy, where projection shifts dual coordinates by a uniform constant), unfolding the dual recursion Zt+1=Z0−η∑s=0t∇ℓs(xs) from x0∈argminx∈Vψ(x) (∇ψ(x0)=0) shows that OMD coincides with Follow the Regularized Leader (FTRL): ∇ψ(xt+1)=−ηs=0∑t∇ℓs(xs)⟺xt+1=argx∈Vmin{s=0∑t⟨∇ℓs(xs),x⟩+η1ψ(x)}, which directly cures the step-function oscillation of unregularized Follow the Leader (§1.2) by adding the strictly convex stabilizer η1ψ(x)!
The mirror map ∇ψ:V→V∗ sends xt to Zt=∇ψ(xt), takes a linear gradient step Zt+1=Zt−η∇f(xt) in the dual space, maps back via xt+1′=(∇ψ)−1(Zt+1), and projects onto V with respect to Bψ.
Origin of Mirror Descent and Legendre duality. Arkadi Nemirovski and David Yudin (Problem Complexity and Method Efficiency in Optimization, 1983) invented Mirror Descent after observing that xt∈V and ∇f(xt)∈V∗ live in dual spaces. On the unconstrained interior (or on Δn under negative entropy), accumulating dual gradients Zt+1=Z0−η∑s=0t∇ℓs(xs) initialized at x0∈argminx∈Vψ(x) is also equivalent to Follow the Regularized Leader (FTRL): xt+1=argminx∈V{η∑s=0t⟨∇ℓs(xs),x⟩+ψ(x)}.
Proposition 2 (Mirror-map equivalence and Bregman projection) Let ψ:D→R be a Legendre convex function with dual image D∗=∇ψ(int(D)) containing Zt+1=Zt−η∇f(xt).
Unconstrained mirror step: When V=Rn, the proximal update xt+1=argminx∈Rn{⟨∇f(xt),x−xt⟩+η1Bψ(x,xt)} is equivalent to ∇ψ(xt+1)=∇ψ(xt)−η∇f(xt).
Constrained Bregman projection: When V⊆D is a closed convex subset, the constrained proximal update xt+1=argminx∈V{⟨∇f(xt),x−xt⟩+η1Bψ(x,xt)} is equivalent to the four-step procedure:
Map to dual space: Zt=∇ψ(xt);
Take a linear gradient step in dual space: Zt+1=Zt−η∇f(xt);
Map back to primal space: xt+1′=(∇ψ)−1(Zt+1);
Project onto V under the Bregman divergence: xt+1=ΠVψ(xt+1′):=argx∈VminBψ(x,xt+1′), which satisfies the generalized Pythagorean contractionBψ(x∗,xt+1)+Bψ(xt+1,xt+1′)≤Bψ(x∗,xt+1′) for all x∗∈V.
Proof. Define G(x):=⟨∇f(xt),x−xt⟩+η1Bψ(x,xt)=⟨∇f(xt),x−xt⟩+η1[ψ(x)−ψ(xt)−⟨∇ψ(xt),x−xt⟩]. Differentiating with respect to x gives ∇G(x)=∇f(xt)+η1(∇ψ(x)−∇ψ(xt)). In the unconstrained case, ∇G(x)=0⟺∇ψ(x)=∇ψ(xt)−η∇f(xt), proving (1). In the constrained case, let xt+1′ satisfy ∇ψ(xt+1′)=∇ψ(xt)−η∇f(xt); then ∇xBψ(x,xt+1′)=∇ψ(x)−∇ψ(xt+1′)=η∇G(x), so x↦Bψ(x,xt+1′) and ηG(x) differ by a constant independent of x and share the exact same minimizer xt+1∈V (see Proposition 5 in §2.2 for the generalized Pythagorean inequality).
The master regret bound and the local inverse-Hessian norm
How does replacing the Euclidean regularizer 21∥x∥22 by a general regularizer ψ change the regret bound of Theorem 3? For any positive definite matrix M≻0, write ∥v∥M:=v⊤Mv for the induced Mahalanobis norm. In Reading Contents §2.1, we prove the following master regret theorem for Online Mirror Descent via a unified continuous-time discretization analysis:
Theorem 6 (Master regret bound for Online Mirror Descent (local inverse-Hessian and global dual-norm forms)) Let V⊆D⊆Rn be a closed convex set, let ψ be a twice continuously differentiable Legendre regularizer on D with ∇2ψ≻0 on int(D) and dual steps Zt−ηtgt∈∇ψ(int(D)), and let ℓ0,…,ℓT−1:V→R be convex loss functions with gradients gt=∇ℓt(xt). Setting ϕ(x):=Bψ(x∗,x), the Online Mirror Descent iterates xt+1=argminx∈V{⟨gt,x−xt⟩+ηt1Bψ(x,xt)} satisfy:
Local inverse-Hessian norm bound: For every comparator x∗∈V, R(T)=t=0∑T−1(ℓt(xt)−ℓt(x∗))≤t=0∑T−1[ηtϕ(xt)−ϕ(xt+1)+2ηtξt∈[xt,xt+1′]sup∇ℓt(xt)(∇2ψ(ξt))−12]. In particular, with constant step size ηt=η, R(T)≤ηBψ(x∗,x0)+2ηt=0∑T−1ξt∈[xt,xt+1′]sup∇ℓt(xt)(∇2ψ(ξt))−12.
Global dual-norm bound: If ψ is μ-strongly convex with respect to a norm ∥⋅∥ on V(Bψ(y,x)≥2μ∥y−x∥2), then for constant step size η>0, R(T)≤ηBψ(x∗,x0)+2μηt=0∑T−1∇ℓt(xt)∗2.
Compare Theorem 6 side by side with the Euclidean OGD bound (Theorem 3) to see the exact structural Δ induced by replacing 21∥x∥22 with a general Legendre regularizer ψ:
Structural component
Euclidean OGD (ψ(x)=21∥x∥22)
Online Mirror Descent (general Legendre ψ)
First-principles origin of the Δ
Local proximity regularizer
21∥x−xt∥22
Bψ(x,xt)≈21∥x−xt∥∇2ψ(xt)2
Primal displacements x−xt are measured in the local Hessian metric Ht=∇2ψ(xt)
Telescoping potential term
η21∥x∗−x0∥22
ηBψ(x∗,x0)
Euclidean potential ϕ(x)=21∥x∗−x∥22 upgrades to the Bregman potential ϕ(x)=Bψ(x∗,x)
Measuring primal steps in H=∇2ψ induces the dual quadratic norm with inverse matrix H−1=(∇2ψ)−1: maxd{η⟨gt,d⟩−21d⊤Hd}=2η2gt⊤H−1gt
In particular, when ψ(x)=21∥x∥22, we have ∇2ψ(x)=I, so ∥gt∥(∇2ψ(ξt))−12=∥gt∥22 and Theorem 6 immediately recovers Theorem 3 as a special case! (See Reading Contents §2.1 for the complete continuous-time discretization proof and the three-line discrete verification via the Three-Point Identity.)
1.9 Understanding Multiplicative Weights (Hedge) and EXP3 via Online Mirror Descent
With the general Online Mirror Descent theorem (Theorem 6) in hand, we can now resolve both simplex puzzles (§§1.5–1.6) at a single stroke—and answer a foundational first-principles question: Why on earth would an online algorithm update probability distributions using an exponential function e−ηℓt(i)?
1. First-principles derivation of Multiplicative Weights from negative-entropy OMD
Return to the learning with expert advice problem (§1.5) on the probability simplex V=Δn, where the round-t loss is ⟨ℓt,xt⟩ with gradient ∇ℓt(xt)=ℓt∈[0,1]n. By Part 1 of Theorem 6, running Online Mirror Descent with any regularizer ψ yields R(T)≤ηBψ(x∗,x0)+2ηt=0∑T−1ξt∈[xt,xt+1′]sup∥ℓt∥(∇2ψ(ξt))−12. Look at the local inverse-Hessian norm ∥ℓt∥(∇2ψ(ξt))−12=ℓt⊤(∇2ψ(ξt))−1ℓt:
When we chose the Euclidean regularizer ψ(x)=21∥x∥22, its inverse Hessian was (∇2ψ(ξt))−1=I, which summed all n coordinates equally: ∥ℓt∥I2=∑i=1nℓt(i)2≤n, costing a factor of n!
Because ξt lies near the probability vector xt∈Δn (whose coordinates satisfy ∑i=1nxt(i)=1), how can we turn the sum of n bounded terms ∑i=1nℓt(i)2 into a weighted sum bounded by 1? By choosing a coordinatewise separable regularizer ψ(x)=∑i=1nh(xi) so that its inverse Hessian is diagonal with entries ξt(i): (∇2ψ(ξ))−1=diag(ξ(1),…,ξ(n))⟺h′′(xi)=∂xi2∂2ψ(x)=xi1and∂xi∂xj∂2ψ(x)=0(i=j)! Integrating h′′(xi)=xi1 twice with respect to xi uniquely produces negative Shannon entropyψ(x)=∑i=1nxilnxi (up to irrelevant affine terms)!
Now let us plug ψ(x)=∑i=1nxilnxi—with mirror map ∇ψ(x)=(1+lnx(1),…,1+lnx(n))—into the four-step Online Mirror Descent procedure of Proposition 2:
Map to dual space:Zt(i)=1+lnxt(i) for each i∈[n].
Linear gradient step in dual space:Zt+1(i)=1+lnxt(i)−ηℓt(i).
Invert the mirror map (1+lnxt+1′(i)=Zt+1(i)):xt+1′(i)=xt(i)e−ηℓt(i)!
Bregman projection onto Δn: We solve xt+1=ΠΔnψ(xt+1′)=argx∈Δnmin{i=1∑nx(i)lnxt+1′(i)x(i)−i=1∑nx(i)+i=1∑nxt+1′(i)}. Introducing a Lagrange multiplier λ∈R for the simplex constraint ∑i=1nx(i)=1 and setting the derivative with respect to x(i) to zero gives lnxt+1′(i)xt+1(i)+λ=0, so xt+1(i)=e−λxt+1′(i). Enforcing ∑i=1nxt+1(i)=1 simply normalizes the vector xt+1′: xt+1(i)=∑j=1nxt+1′(j)xt+1′(i)=∑j=1nxt(j)e−ηℓt(j)xt(i)e−ηℓt(i).
Why exponential weights? We do not multiply probabilities by e−ηℓt(i) as an ad-hoc algebraic trick; rather, exp(⋅) is the inverse mirror map (∇ψ)−1 of negative entropy ψ(x)=∑i=1nxilnxi, which is the unique separable regularizer on Δn whose curvature ∇2ψ(x)=diag(1/xi) matches the geometry of probability distributions!
Definition 6 (Multiplicative Weights Update (MWU / Hedge / Entropic OMD)) Fix a learning rate η>0. Initialize weights w1(i):=1 (so p1(i):=x0(i)=n1) for all i∈[n]. At each round t=1,…,T:
Normalize the current weights into a probability distribution on Δn: Φt:=i=1∑nwt(i),pt(i):=Φtwt(i)(i∈[n]).
Observe the loss vector ℓt, incur expected loss ⟨pt,ℓt⟩, and update the weights multiplicatively: wt+1(i):=wt(i)e−ηℓt(i)=exp(−ηLt(i)),pt+1(i)=∑j=1npt(j)e−ηℓt(j)pt(i)e−ηℓt(i).
Independent discoveries of Multiplicative Weights. The multiplicative-weights method was discovered independently across learning theory, optimization, and game theory: Nick Littlestone and Manfred Warmuth (The Weighted Majority Algorithm, Information and Computation 1994) introduced exponential updates for mistake bounds; Yoav Freund and Robert Schapire (A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting, JCSS 1997) formulated Hedge; and Sanjeev Arora, Elad Hazan, and Satyen Kale (The Multiplicative Weights Update Method: a Meta-Algorithm and Applications, Theory of Computing 2012) unified its algorithmic applications across TCS.
Example 5 (How MWU cures FTL’s alternating-loss oscillation) On the n=2 alternating sequence of Example 1 (ℓ1=(1/2,0), ℓt=(0,1) for even t, ℓt=(1,0) for odd t≥3), before any round t≥2 the cumulative loss gap between the two experts is ∣Lt−1(1)−Lt−1(2)∣=1/2. Instead of jumping to (0,1) or (1,0) like FTL, MWU plays the current leader with probability 1+e−η/21=21+8η+O(η3)≤21+8η, so its excess loss above T/2 over T rounds is at most 8ηT=O(T) when η=Θ(1/T).
2. Regret bounds for Multiplicative Weights: OMD proof and potential refinements
Let us now evaluate the two terms of our local inverse-Hessian OMD bound (Theorem 6) for ψ(x)=∑i=1nxilnxi starting from the uniform distribution x0=u=(n1,…,n1):
Bregman diameter of Δn under ψ: For any comparator distribution x∗∈Δn, Bψ(x∗,x0)=KL(x∗∥u)=i=1∑nx∗(i)ln(nx∗(i))=lnn−H(x∗)≤lnn, so the first term ηBψ(x∗,x0) is at most ηlnn.
Local inverse-Hessian norm term (and two subtle points on why ξt(i)≤xt(i)): Because ∇2ψ(ξt)=diag(1/ξt(1),…,1/ξt(n)), we have ∥ℓt∥(∇2ψ(ξt))−12=∑i=1nℓt(i)2ξt(i). To bound ξt(i) by xt(i), notice two crucial structural facts that hold for both full-information losses ℓt∈[0,1]n and importance-weighted bandit estimators ℓ^t(i)≥0:
Why does ξt lie on the segment [xt+1′,xt] rather than [xt+1,xt]? After normalization, xt+1∈Δn still sums to 1, so xt+1(i)≤xt(i) cannot hold for all i. However, in the four-step decomposition of Proposition 2, Step 4 (the Bregman projection xt+1′↦xt+1=ΠΔnψ(xt+1′)) only contracts the potential Bψ(x∗,xt+1)≤Bψ(x∗,xt+1′) and introduces zero second-order error; the entire discretization lag arises during the unconstrained dual step xt→xt+1′, so the Taylor mean-value point ξt lies on the segment [xt+1′,xt] connecting xt to the unnormalized point xt+1′!
Why does nonnegativity of the losses guarantee ξt(i)≤xt(i)? Whenever the input loss vector gt has nonnegative coordinatesgt(i)≥0 (even if a bandit estimate ℓ^t(At)=ℓt(At)/xt(At) is huge when xt(At) is small!), the factor e−ηgt(i) lies in (0,1], so every unnormalized coordinate shrinks: xt+1′(i)=xt(i)e−ηgt(i)≤xt(i). Consequently, every convex combination ξt∈[xt+1′,xt] satisfies ξt(i)≤xt(i) for all i∈[n]! (By contrast, updating with positive exponential rewards e+ηr^t(i) would cause xt+1′(At) to explode exponentially as xt(At)→0, breaking ξt(i)≤xt(i).)
Using ξt(i)≤xt(i) for ℓt∈[0,1]n immediately yields 2ηt=0∑T−1ξt∈[xt+1′,xt]sup∥ℓt∥(∇2ψ(ξt))−12=2ηt=0∑T−1ξtsupi=1∑nℓt(i)2ξt(i)≤2ηt=0∑T−1i=1∑nℓt(i)2xt(i)≤2ηt=0∑T−1i=1∑nxt(i)=2ηT.
Combining the two pieces gives R(T)≤ηlnn+2ηt=0∑T−1i=1∑nxt(i)ℓt(i)2≤ηlnn+2ηT=2Tlnn(at η=T2lnn). Notice that the exact same O(Tlnn) bound also follows from Part 2 of Theorem 6 via Pinsker’s inequality (KL(y∥x)≥21∥y−x∥12, proved in §2.3), which shows that ψ is 1-strongly convex with respect to ∥⋅∥1, paired with the dual norm ∥ℓt∥∞≤1!
Recall (Hoeffding’s Lemma, Chapter 1). For any random variable Z∈[a,b] and λ∈R, lnE[eλZ]≤λE[Z]+8λ2(b−a)2. Applying this directly to ΦtΦt+1=∑i=1npt(i)e−ηℓt(i)=EI∼pt[e−ηℓt(I)] improves the variance constant from 1/2 to 1/8 when ℓt∈[0,1]n.
For our upcoming applications to zero-sum games (§1.10) and linear programming feasibility (§1.11), it is useful to record both the sharp Hoeffding constant for ℓt∈[0,1]n and the extension of the local second-moment bound to signed payoffsℓt∈[−ρ,ρ]n.
Theorem 7 (Regret bound for Multiplicative Weights on [0,1]n) For any sequence of loss vectors ℓ1,…,ℓT∈[0,1]n and any learning rate η>0, the Multiplicative Weights Update algorithm (Definition 6) satisfies, for every expert i∗∈[n], RegT(i∗)=t=1∑T⟨pt,ℓt⟩−t=1∑Tℓt(i∗)≤ηlnn+8ηT. In particular, setting η=2T2lnn yields RegT≤2Tlnn.
Proof. Since pt(i)=wt(i)/Φt, at each round t we have ΦtΦt+1=∑i=1npt(i)e−ηℓt(i)=EIt∼pt[e−ηℓt(It)]. Because ℓt(It)∈[0,1] has mean ⟨pt,ℓt⟩, Hoeffding’s Lemma gives lnΦtΦt+1≤−η⟨pt,ℓt⟩+8η2. Telescoping over t=1,…,T (with Φ1=n) and comparing against the lower bound lnΦT+1≥lnwT+1(i∗)=−η∑t=1Tℓt(i∗) yields −η∑t=1Tℓt(i∗)−lnn≤−η∑t=1T⟨pt,ℓt⟩+8η2T. Dividing by η>0 completes the proof.
Theorem 8 (Second-order Multiplicative Weights bound for signed and nonnegative losses) Suppose at each round t=1,…,T the vector ℓt∈Rn and learning rate η>0 satisfy ηℓt(i)≥−1 for all i∈[n]. Then Multiplicative Weights (Definition 6) satisfies, for every i∗∈[n], t=1∑T⟨pt,ℓt⟩−t=1∑Tℓt(i∗)≤ηlnn+ηt=1∑Ti=1∑npt(i)ℓt(i)2. If ℓt(i)≥0 for all t,i (with no upper bound on ηℓt(i)), the second term improves by a factor of two (matching the OMD local inverse-Hessian bound above): t=1∑T⟨pt,ℓt⟩−t=1∑Tℓt(i∗)≤ηlnn+2ηt=1∑Ti=1∑npt(i)ℓt(i)2. In particular, if ℓt∈[−ρ,ρ]n for all t and 0<η≤1/ρ, then RegT(i∗)≤ηlnn+ηρ2T.
Proof: quadratic Taylor bound on the partition function
Using e−u≤1−u+u2 (valid for all u≥−1, or e−u≤1−u+21u2 for u≥0) and ln(1+z)≤z, we have lnΦtΦt+1=lni=1∑npt(i)e−ηℓt(i)≤ln(1−η⟨pt,ℓt⟩+η2i=1∑npt(i)ℓt(i)2)≤−η⟨pt,ℓt⟩+η2i=1∑npt(i)ℓt(i)2. Telescoping against lnΦT+1≥−η∑t=1Tℓt(i∗) and dividing by η gives both claims.
Polynomial weight update. One often writes the linearized variant wt+1(i)=wt(i)(1−ηℓt(i)) for ℓt∈[−1,1]n and η≤1/2, which satisfies RegT(i∗)≤ηlnn+η∑t=1T∣ℓt(i∗)∣ with the same O(Tlnn) scaling.
Now return to the adversarial multi-armed bandit problem (§1.6). In §1.6, feeding the importance-weighted estimator ℓ^t(i)=xt(i)1{At=i}ℓt(i) into Euclidean OGD produced the variance E[∥ℓ^t∥22∣Ft−1]=∑i=1nxt(i)ℓt(i)2, which blew up near xt(i)=0 and forced O((nT)2/3) regret.
What happens if we instead run Online Stochastic Mirror Descent (OSMD) with negative entropy ψ(x)=∑i=1nxilnxi on the exact same estimator ℓ^t? In primal coordinates, this update xt+1(i)=∑j=1nxt(j)exp(−ηℓ^t(j))xt(i)exp(−ηℓ^t(i)) is the celebrated EXP3 algorithm (Exponential-weight algorithm for Exploration and Exploitation, Auer et al. 2002). Look at what happens when we plug ℓ^t(i)≥0 into our local inverse-Hessian OMD bound (Theorem 6)!
Caution (How the inverse Hessian of negative entropy cancels 1/xt(i)).
Because ℓ^t(i)≥0, every point ξt∈[xt+1′,xt] satisfies ξt(i)≤xt(i). In the local inverse-Hessian norm ∥ℓ^t∥(∇2ψ(ξt))−12=∑i=1nℓ^t(i)2ξt(i)≤∑i=1nℓ^t(i)2xt(i), the inverse Hessian (∇2ψ(xt))−1=diag(xt(1),…,xt(n)) supplies an extra factor of xt(i) that cancels the remaining 1/xt(i) denominator in E[ℓ^t(i)2∣Ft−1]=xt(i)ℓt(i)2, without any artificial α-uniform exploration!
Theorem 9 (Regret bound for Online Stochastic Mirror Descent (EXP3) in adversarial bandits) In the adversarial multi-armed bandit problem (Definition 4) with losses ℓt∈[0,1]n, running Online Stochastic Mirror Descent with negative entropy ψ(x)=∑i=1nxilnxi from x0=u=(n1,…,n1) on the importance-weighted estimator ℓ^t(i)=xt(i)1{At=i}ℓt(i) satisfies, for every fixed comparator x∗∈Δn, E[t=0∑T−1⟨ℓt,xt−x∗⟩]≤ηlnn+2ηnT. In particular, maximizing over x∗=ej (j∈[n]) and choosing η=nT2lnn yields RegTbandit≤2nTlnn=O(nTlnn).
Proof. Fix any deterministic comparator x∗∈Δn. By conditional unbiasedness E[ℓ^t∣Ft−1]=ℓt, Lemma 3, and Part 1 of Theorem 6 applied to the realized sequence ℓ^0,…,ℓ^T−1 (using ξt(i)≤xt(i) since ℓ^t(i)≥0), taking expectations over A0,…,AT−1 gives E[t=0∑T−1⟨ℓt,xt−x∗⟩]=E[t=0∑T−1⟨ℓ^t,xt−x∗⟩]≤ηBψ(x∗,x0)+2ηE[t=0∑T−1ξt∈[xt+1′,xt]sup∥ℓ^t∥(∇2ψ(ξt))−12]≤ηlnn+2ηt=0∑T−1E[i=1∑n(xt(i)1{At=i}ℓt(i))2xt(i)]≤ηlnn+2ηt=0∑T−1E[i=1∑nxt(i)1xt(i)]=ηlnn+2ηnT=2nTlnn(at η=nT2lnn). Taking the maximum over x∗=ej (j∈[n]) gives RegTbandit≤2nTlnn.
1.10 Playing a zero-sum game repeatedly: von Neumann’s minimax theorem
Now that we have derived the Multiplicative Weights Update (MWU) algorithm from first principles via Online Mirror Descent, let us see why this online learning primitive is one of the most versatile tools in algorithm design. The central paradigm is self-play: if we pit a no-regret MWU learner against a dual opponent or constraint oracle that repeatedly exposes the worst-case violation, the time-averaged trajectory of the learner must converge to an equilibrium or feasible solution! Our first payoff is a constructive proof of von Neumann’s minimax theorem for two-player zero-sum games.
Zero-sum matrix games and weak duality
Let A∈[−1,1]m×n be a payoff matrix with m rows and n columns. Two players choose mixed strategies:
The row player (minimizer) chooses a distribution p∈Δm over the m rows;
The column player (maximizer) chooses a distribution q∈Δn over the n columns.
When they play (p,q), the row player pays the expected amount p⊤Aq=∑i=1m∑j=1np(i)Ai,jq(j) to the column player. If the row player must commit to p∈Δm first, the column player responds to maximize p⊤Aq, yielding the minimax (upper) valueV:=p∈Δmminq∈Δnmaxp⊤Aq=p∈Δmminj∈[n]max(p⊤A)j. If the column player commits to q∈Δn first, the row player responds to minimize p⊤Aq, yielding the maximin (lower) valueV:=q∈Δnmaxp∈Δmminp⊤Aq=q∈Δnmaxi∈[m]min(Aq)i.
Recall (Weierstrass extreme value theorem). Because p↦maxj∈[n](p⊤A)j and q↦mini∈[m](Aq)i are continuous on the compact simplices Δm and Δn, both extrema V and V are attained.
Lemma 5 (Weak minimax inequality) For any matrix A∈Rm×n, moving second can only help: V=q∈Δnmaxp∈Δmminp⊤Aq≤p∈Δmminq∈Δnmaxp⊤Aq=V.
Proof. For every pair (p~,q~)∈Δm×Δn, we have minp∈Δmp⊤Aq~≤p~⊤Aq~≤maxq∈Δnp~⊤Aq. Taking maxq~∈Δn on the left and minp~∈Δm on the right gives V≤V.
Closing the duality gap via no-regret self-play
To prove that V=V over mixed strategies—and to compute an ε-approximate Nash equilibrium—we let the row player run Multiplicative Weights over T rounds against a best-responding column player:
At round t=1,…,T, the row player outputs pt∈Δm via MWU (Definition 6);
The column player picks a best response qt∈argmaxq∈Δnpt⊤Aq;
The row player observes the loss vector ℓt:=Aqt∈[−1,1]m and pays ⟨pt,ℓt⟩=pt⊤Aqt.
From fixed-point topology to online self-play. John von Neumann (Zur Theorie der Gesellschaftsspiele, 1928) originally proved Theorem 10 via a Brouwer-type topological fixed-point argument, before George Dantzig and von Neumann linked zero-sum games to linear programming duality in 1947 (Chapter 6). Replacing Brown (1949) and Robinson’s (1951) fictitious play (FTL) with no-regret MWU accelerates convergence from exponential to O(ε−2lnm) rounds.
Theorem 10 (von Neumann’s minimax theorem and approximate equilibrium via MWU) For every payoff matrix A∈[−1,1]m×n, the minimax and maximin values coincide: p∈Δmminq∈Δnmaxp⊤Aq=q∈Δnmaxp∈Δmminp⊤Aq=:V∗. Moreover, if the row player runs Multiplicative Weights (Definition 6) with η=(2lnm)/T on ℓt=Aqt∈[−1,1]m against best responses qt∈argmaxq∈Δnpt⊤Aq for T=⌈ε22lnm⌉ rounds, then the time-averaged strategies pˉT:=T1∑t=1Tpt∈Δm and qˉT:=T1∑t=1Tqt∈Δn form an ε-approximate Nash equilibrium: q∈ΔnmaxpˉT⊤Aq−ε≤V∗≤p∈Δmminp⊤AqˉT+ε.
Proof. Because ℓt=Aqt∈[−1,1]m takes values in an interval of length 2, shifting and scaling ℓt to (ℓt+1)/2∈[0,1]m with learning rate 2η leaves pt invariant and scales Theorem 7 by 2, giving T1t=1∑Tpt⊤Aqt−i∗∈[m]minT1t=1∑Tei∗⊤Aqt=TRegT≤ηTlnm+2η=T2lnm≤ε. Now sandwich the time-averaged game value T1∑t=1Tpt⊤Aqt between V and V: V≤V=p∈Δmminq∈Δnmaxp⊤Aq≤q∈ΔnmaxpˉT⊤Aq=q∈Δnmax(T1t=1∑Tpt⊤Aq)≤T1t=1∑Tq∈Δnmaxpt⊤Aq=T1t=1∑Tpt⊤Aqt≤i∗∈[m]minei∗⊤A(T1t=1∑Tqt)+T2lnm=p∈Δmminp⊤AqˉT+T2lnm≤V+T2lnm. Letting T→∞ forces V=V=V∗, and setting T=⌈2ε−2lnm⌉ gives the ε-equilibrium guarantee.
Example 6 (Asymmetric penalty-kick game and why time-averaging is essential) Consider the 2×2 game A=(1002), whose unique minimax equilibrium is p∗=q∗=(2/3,1/3) with value V∗=2/3. Notice why Theorem 10 outputs the time-averaged strategies (pˉT,qˉT) rather than the last iterate (pT,qT): each best response qt∈argmaxqpt⊤Aq is a pure column (1,0) or (0,1), which by itself only secures minpp⊤Aqt=0<2/3. However, to keep pt near (2/3,1/3), the column player must pick column 1 on 2/3 of the rounds and column 2 on 1/3 of the rounds, so the empirical frequency qˉT=T1∑t=1Tqt converges to q∗=(2/3,1/3)!
1.11 Solving linear feasibility via an oracle: the Plotkin–Shmoys–Tardos framework
Our second algorithmic application turns Multiplicative Weights into a general-purpose solver for linear programming feasibility (and fractional packing/covering optimization). Suppose we want to decide whether there exists a point x∈Rd satisfying x∈PandAix≥bifor all i=1,…,m, where P⊆Rd is a “simple” convex set (such as a box or decoupled single-commodity flows) and Ax≥b is a system of m “hard” coupling linear inequalities.
Collapsing m constraints into one weighted average
Suppose that although checking all m constraints Aix≥bi simultaneously is hard, for any probability distribution p=(p(1),…,p(m))∈Δm over the m constraints we can efficiently check the single collapsed linear inequality∃?x∈P:p⊤Ax≥p⊤b⟺i=1∑mp(i)(Aix−bi)≥0. Note first that if for somep∈Δm no such x∈P exists, then {x∈P:Ax≥b} is certainly infeasible, and p is a compact infeasibility certificate.
Definition 7 (ρ-bounded average-constraint oracle) Given a convex set P⊆Rd, linear constraints Aix≥bi (i∈[m]), and a width parameter ρ>0, a ρ-bounded average-constraint oracle takes any p∈Δm and either:
returns a point x∈P satisfying the average constraint and width bound i=1∑mp(i)(Aix−bi)≥0andAix−bi≤ρfor all i∈[m];
or correctly declares that no x∈P satisfies p⊤Ax≥p⊤b (certifying infeasibility).
Why is positive slack treated as a positive loss? In the PST game, each constraint i∈[m] is an “expert.” If the oracle’s point xt satisfies constraint i with a large margin Aixt−bi≫0, constraint i is “easy,” so assigning it a positive loss ℓt(i)=Aixt−bi>0 shrinks its weight wt+1(i)=wt(i)e−ηℓt(i). Conversely, violated constraints have ℓt(i)<0, so their weights grow exponentially, forcing subsequent oracle calls to satisfy them!
To find a single point xˉT∈P satisfying allm constraints within ε, we treat the m constraints as experts in MWU and define the round-t loss of constraint i as its slack ℓt(i):=Aixt−bi∈[−ρ,ρ].
Theorem 11 (Plotkin–Shmoys–Tardos approximate feasibility theorem) Let P⊆Rd be a convex set and let Aix≥bi (i∈[m]) be equipped with a ρ-bounded average-constraint oracle (Definition 7). Given 0<ε≤2ρ, run Multiplicative Weights (Definition 6) over the m constraints with learning rate η=2ρ2ε≤ρ1 and losses ℓt(i)=Aixt−bi for T:=⌈ε24ρ2lnm⌉ rounds. Then either some oracle call certifies that {x∈P:Ax≥b} is infeasible, or the average point xˉT:=T1∑t=1Txt∈P satisfies every constraint within ε: AixˉT≥bi−εfor all i∈[m].
Proof. Suppose all T oracle calls return x1,…,xT∈P, so xˉT=T1∑t=1Txt∈P by convexity. At every round t, the oracle guarantees ⟨pt,ℓt⟩=∑i=1mpt(i)(Aixt−bi)≥0 and ∣ℓt(i)∣≤ρ. Since η=2ρ2ε≤ρ1, the signed second-order MWU bound (Theorem 8) gives, for every constraint i∗∈[m], 0≤T1t=1∑T⟨pt,ℓt⟩≤T1t=1∑T(Ai∗xt−bi∗)+ηTlnm+ηρ2≤(Ai∗xˉT−bi∗)+2ε+2ε, which rearranges to Ai∗xˉT≥bi∗−ε for all i∗∈[m].
Example 7 (Balancing traffic across two parallel links via PST) To route 1 unit of flow across m=2 parallel links with capacity 1/2 each, set P={x=(x1,x2)∈R≥02:x1+x2=1} and constraints 21−xi≥0 (i=1,2), so ℓt(i)=21−xt,i∈[−1/2,1/2] has width ρ=1/2. Given weights pt=(pt(1),pt(2))∈Δ2, the oracle maximizes 21−⟨pt,x⟩ over x∈Δ2 by sending the entire unit of flow along whichever link currently has the smaller weight pt(i). Although each individual response xt∈{(1,0),(0,1)} overloads one link, MWU exponentially boosts the overloaded link’s weight, forcing the oracle to alternate between the two links so that xˉT=T1∑t=1Txt converges to (1/2,1/2) in O(ε−2) rounds. (See §2.4 for general multicommodity network routing.)
Where we stand, and the question for Chapter 5
Look back at the arc we have traversed: starting from Follow the Leader on the number guessing game, we developed offline and online Projected Gradient Descent, exposed the n and (nT)2/3 bottlenecks of the Euclidean quadratic regularizer on Δn, generalized to Online Mirror Descent via the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12, and saw how negative entropy ψ(x)=∑i=1nxilnxi derives both MWU / Hedge (O(Tlnn) under full information) and EXP3 / OSMD (O(nTlnn) under bandit feedback) from first principles.
This leaves a fundamental question for Chapter 5 (Information Budgets and Lower Bounds): Is the extra n factor in EXP3 (O(nTlnn) versus O(Tlnn)) merely an artifact of importance weighting, or is Ω(nT) an unbreakable information-theoretic barrier for every bandit algorithm? In Chapter 5 we will see that the exact same relative entropy KL(p∥q) that appeared in §1.7 as the Bregman divergence of negative entropy on the simplex also governs how much statistical evidence a bandit transcript can accumulate—proving that no bandit algorithm can beat Ω(nT).
II. Reading Contents
The sections below provide self-contained derivations and algorithmic extensions for both classroom meetings: the unified continuous-time discretization proof of Online Mirror Descent and Online Gradient Descent (§2.1), the convex-analytic foundation of Mirror Descent via Fenchel conjugation and the generalized Pythagorean inequality (§2.2), a complete proof of Pinsker’s inequality and its connection to normed strong convexity (§2.3), a concrete instantiation of the Plotkin–Shmoys–Tardos framework for multicommodity network routing (§2.4), and the online-to-batch conversion linking online regret to stochastic convex optimization (§2.5).
2.1 Continuous-time discretization proof of Online Mirror Descent and OGD
In §1.4 and §1.8 we stated the regret bounds of Online Projected Gradient Descent (Theorem 3) and Online Mirror Descent (Theorem 6). While one can verify both bounds algebraically in discrete time, a continuous-time approach gives much deeper insight into why the regret bound has the exact form “potential drop plus a second-order local norm penalty”—and proves Theorem 6 (and therefore Theorem 3 as the special case ψ(x)=21∥x∥22) in a single unified calculation.
1. Why continuous time? Gradient flow and the origin of the second-order penalty
To see the intuition before introducing general regularizers, return first to the unconstrained offline Gradient Descent recurrence xt+1−xt=−η∇f(xt) from §1.3. Dividing both sides by η and letting the step size η→0 turns the discrete difference equation into a continuous-time ordinary differential equation—the gradient flow: dtdx(t)=−∇f(x(t))(t≥0). Let us track the Euclidean potential ϕ(t):=21∥x(t)−x∗∥22 along this continuous trajectory. By the chain rule and convexity of f, dtdϕ(t)=⟨x(t)−x∗,dtdx(t)⟩=−⟨∇f(x(t)),x(t)−x∗⟩≤f(x∗)−f(x(t)). Integrating from t=0 to t=T yields the continuous-time convergence law ∫0T(f(x(t))−f(x∗))dt≤∫0T⟨∇f(x(t)),x(t)−x∗⟩dt=ϕ(0)−ϕ(T)! Compare this continuous formula with the discrete bound ∑t=0T−1(f(xt)−f(x∗))≤ηϕ(x0)−ϕ(xT)+2η∑t=0T−1∥∇f(xt)∥22 from Theorem 2: a continuous learner moving instantaneously along −∇f(x(t)) pays only the potential difference ϕ(0)−ϕ(T), with no second-order gradient term at all!
Why, then, does a discrete online algorithm pay a second-order penalty? Because during a discrete round s of step size ηs, the discrete player remains frozen at xs while the continuous trajectory yt has already started moving in direction −gs=−∇ℓs(xs). Let us now formalize this discretization lag directly for Online Mirror Descent with an arbitrary Legendre regularizer ψ, proving Theorem 6 and Theorem 3 simultaneously.
2. Continuous-time Bregman flow and proof of the Master OMD Theorem (Theorem 6)
Recall from §1.8 that at each round s=0,1,…,T−1, Online Mirror Descent observes gs:=∇ℓs(xs) and updates via Proposition 2: ∇ψ(xs+1′)=∇ψ(xs)−ηsgs,xs+1=ΠVψ(xs+1′)=argx∈VminBψ(x,xs+1′). We embed the discrete rounds s=0,1,…,T−1 into continuous time t∈[0,TT] by defining the cumulative time grid T0:=0,Ts:=i=0∑s−1ηi(s=1,…,T), and the piecewise-constant gradient signal gt:=gs=∇ℓs(xs) for t∈[Ts,Ts+1). On each interval [Ts,Ts+1] of length Ts+1−Ts=ηs, define the continuous-time mirror trajectoryyt∈int(D) starting from yTs:=xs by the dual differential equation dtd∇ψ(yt)=−gs(t∈[Ts,Ts+1]). Integrating from Ts to t∈[Ts,Ts+1] gives ∇ψ(yt)=∇ψ(xs)−(t−Ts)gs, which immediately implies:
Proposition 3 (Grid interpolation of the continuous mirror trajectory) On each round s=0,1,…,T−1, the continuous trajectory yt starts at yTs=xs and reaches the unprojected mirror step at t=Ts+1: yTs+1=xs+1′=(∇ψ)−1(∇ψ(xs)−ηsgs). In particular, when V=Rn is unconstrained, yTs=xs at every grid point s=0,1,…,T.
Now measure the distance from any comparator x∗∈V to yt by the Bregman potentialϕ(y):=Bψ(x∗,y)=ψ(x∗)−ψ(y)−⟨∇ψ(y),x∗−y⟩. Differentiating ϕ(yt) with respect to t∈(Ts,Ts+1) via the product and chain rules produces an exact cancellation: dtdϕ(yt)=−⟨∇ψ(yt),dtdyt⟩−⟨dtd∇ψ(yt),x∗−yt⟩+⟨∇ψ(yt),dtdyt⟩=⟨dtd∇ψ(yt),yt−x∗⟩=−⟨gs,yt−x∗⟩! Integrating this exact derivative from t=Ts to t=Ts+1 and applying the generalized Pythagorean contraction ϕ(xs+1)=Bψ(x∗,xs+1)≤Bψ(x∗,xs+1′)=ϕ(yTs+1) (Proposition 5 in §2.2) yields the continuous-time conservation inequality on round s: ∫TsTs+1⟨gs,yt−x∗⟩dt=ϕ(yTs)−ϕ(yTs+1)=ϕ(xs)−ϕ(xs+1′)≤ϕ(xs)−ϕ(xs+1).
Now compare the discrete learner’s linearized loss ηs⟨gs,xs−x∗⟩ on round s with the continuous integral ∫TsTs+1⟨gs,yt−x∗⟩dt. Because ∫TsTs+1⟨gs,x∗⟩dt=ηs⟨gs,x∗⟩, their difference is the discretization lag between the frozen point xs and the moving point yt: ηs⟨gs,xs−x∗⟩−∫TsTs+1⟨gs,yt−x∗⟩dt=∫TsTs+1⟨gs,xs−yt⟩dt. Let us evaluate ⟨gs,xs−yt⟩ for t∈[Ts,Ts+1]:
Special case 1: Euclidean Online Gradient Descent (ψ(x)=21∥x∥22, proving Theorem 3). When ψ(x)=21∥x∥22, the mirror map is the identity ∇ψ(y)=y, so xs−yt=(t−Ts)gs is linear in t. The discretization lag integral evaluates in closed form: ∫TsTs+1⟨gs,xs−yt⟩dt=∥gs∥22∫TsTs+1(t−Ts)dt=2ηs2∥gs∥22. Adding ∫TsTs+1⟨gs,yt−x∗⟩dt≤ϕ(xs)−ϕ(xs+1) and dividing by ηs gives ℓs(xs)−ℓs(x∗)≤⟨gs,xs−x∗⟩≤ηsϕ(xs)−ϕ(xs+1)+2ηs∥gs∥22, completing the proof of Theorem 3!
General case: Online Mirror Descent for arbitrary Legendre ψ (proving Theorem 6). In dual coordinates, ∇ψ(yt)=∇ψ(xs)−(t−Ts)gs. Because yt=(∇ψ)−1(∇ψ(xs)−(t−Ts)gs) and the inverse function theorem gives the Jacobian d(∇ψ)−1(θ)=(∇2ψ(y))−1 at θ=∇ψ(y), applying the Mean Value Theorem to the scalar function τ⟼⟨gs,(∇ψ)−1(∇ψ(xs)−τgs)⟩on τ∈[0,t−Ts] produces a point ξt=yτt along the mirror trajectory Γs:={yτ:τ∈[Ts,Ts+1]} connecting xs to xs+1′ (whose coordinatewise range coincides with the box spanned by xs and xs+1′ whenever ψ is coordinatewise separable, while the discrete Taylor argument below directly places ξs on the straight line segment [xs,xs+1′] for arbitrary ψ) such that ⟨gs,xs−yt⟩=(t−Ts)gs⊤(∇2ψ(ξt))−1gs=(t−Ts)∥gs∥(∇2ψ(ξt))−12. Integrating over t∈[Ts,Ts+1] gives ∫TsTs+1⟨gs,xs−yt⟩dt≤ξs∈Γssupgs(∇2ψ(ξs))−12∫TsTs+1(t−Ts)dt=2ηs2ξs∈Γssupgs(∇2ψ(ξs))−12. Combining this with ∫TsTs+1⟨gs,yt−x∗⟩dt≤ϕ(xs)−ϕ(xs+1) and dividing by ηs yields ℓs(xs)−ℓs(x∗)≤⟨gs,xs−x∗⟩≤ηsϕ(xs)−ϕ(xs+1)+2ηsξs∈Γssupgs(∇2ψ(ξs))−12, giving the continuous-trajectory form of Theorem 6 (and see the folded verification below for the exact segment ξs∈[xs,xs+1′])!
Alternative discrete verification via the Three-Point Identity and Taylor’s theorem
Both parts of Theorem 6 also follow in three lines from the discrete optimality condition of xt+1. By first-order optimality of xt+1=argminx∈V{ηt⟨gt,x⟩+Bψ(x,xt)} and the Three-Point Identity (Lemma 4), ηt⟨gt,xt−x∗⟩=ηt⟨gt,xt+1−x∗⟩+ηt⟨gt,xt−xt+1⟩≤⟨∇ψ(xt+1)−∇ψ(xt),x∗−xt+1⟩+ηt⟨gt,xt−xt+1⟩=Bψ(x∗,xt)−Bψ(x∗,xt+1)−Bψ(xt+1,xt)+ηt⟨gt,xt−xt+1⟩. For Part 2, Bψ(xt+1,xt)≥2μ∥xt−xt+1∥2 and Fenchel–Young gives ηt⟨gt,xt−xt+1⟩≤2μηt2∥gt∥∗2+2μ∥xt−xt+1∥2. For Part 1, by the generalized Pythagorean inequality Bψ(x∗,xt+1)≤Bψ(x∗,xt+1′) and the Three-Point Identity with ∇ψ(xt)−∇ψ(xt+1′)=ηtgt and dt:=xt−xt+1′, there is ξt∈[xt,xt+1′] with Bψ(xt+1′,xt)=21dt⊤∇2ψ(ξt)dt, so ηt⟨gt,xt−x∗⟩−Bψ(x∗,xt)+Bψ(x∗,xt+1′)=Bψ(xt,xt+1′)=ηt⟨gt,dt⟩−Bψ(xt+1′,xt)=ηt⟨gt,dt⟩−21dt⊤∇2ψ(ξt)dt≤2ηt2∥gt∥(∇2ψ(ξt))−12 for some ξt∈[xt,xt+1′]. Dividing by ηt and summing over t=0,…,T−1 completes both proofs.
2.2 Fenchel conjugation, the Young–Fenchel gap, and constrained Bregman projections
In §§1.7–1.8 we introduced the mirror map ∇ψ:V→V∗, its inverse (∇ψ)−1=∇ψ∗, and the Bregman divergence Bψ(x,y). Here we develop the convex duality behind these objects from first principles and prove that the one-step proximal update of Online Mirror Descent is equivalent to an unconstrained dual gradient step followed by a Bregman projection.
Paired spaces and the Fenchel conjugate
Let V=Rn be the primal vector space (where decision points x∈K live) and let V∗=Rn be the dual space of linear functionals (where gradients y=gt live), paired by the bilinear evaluation map ⟨y,x⟩=∑i=1ny(i)x(i). Although V and V∗ are both Rn as sets of coordinates, keeping their roles distinct prevents us from confusing primal displacements x−x′∈V with dual covectors y∈V∗.
Given a convex function ψ:D→R on a convex domain D⊆V, every affine function x↦⟨y,x⟩−c with slope y∈V∗ that lies below ψ on D must satisfy ⟨y,x⟩−c≤ψ(x) for all x∈D, or equivalently c≥supx∈D{⟨y,x⟩−ψ(x)}. The tightest such intercept c as a function of the slope y∈V∗ defines the Fenchel conjugate.
Definition 8 (Fenchel conjugate) For a function ψ:D→R on D⊆V, its Fenchel conjugateψ∗:V∗→R∪{+∞} is defined by ψ∗(y):=x∈Dsup{⟨y,x⟩−ψ(x)}(y∈V∗).
Because ψ∗(y) is the pointwise supremum of a family of affine functions y↦⟨y,x⟩−ψ(x), ψ∗ is always a convex function on V∗. Moreover, directly from the definition of supremum, for every x∈D and every y∈V∗ we have the Young–Fenchel inequality: ψ(x)+ψ∗(y)≥⟨y,x⟩.
When does equality hold in the Young–Fenchel inequality, and how does ψ∗ relate to the Bregman divergence Bψ(x,z)?
Proposition 4 (Inverse mirror map and Bregman divergence as the Young–Fenchel gap) Suppose ψ:D→R is strictly convex and differentiable on int(D) with ∥∇ψ(x)∥→∞ as x approaches the boundary of D (a Legendre function). Let D∗:=∇ψ(int(D))⊆V∗ denote the open dual image of ∇ψ. Then:
For every z∈int(D) and y∈D∗, equality ψ(z)+ψ∗(y)=⟨y,z⟩ holds if and only if y=∇ψ(z), which in turn holds if and only if z=∇ψ∗(y). In particular, the gradient of the Fenchel conjugate is the inverse mirror map on D∗: ∇ψ∗=(∇ψ)−1.
For any x∈D and z∈int(D) with dual image y=∇ψ(z), the Bregman divergence Bψ(x,z) equals the Young–Fenchel gap between x∈V and y=∇ψ(z)∈V∗: Bψ(x,z)=ψ(x)+ψ∗(∇ψ(z))−⟨∇ψ(z),x⟩≥0.
Proof: Legendre stationarity and Young–Fenchel substitution
For (1), fix y∈D∗. Since ψ is strictly convex and y∈∇ψ(int(D)), the concave function x↦⟨y,x⟩−ψ(x) attains its unique maximum on int(D) at the point z where its gradient vanishes: y−∇ψ(z)=0. At this maximizer z, the supremum in Definition 8 is achieved: ψ∗(∇ψ(z))=⟨∇ψ(z),z⟩−ψ(z). By the envelope theorem for the unique maximizer z(y) of x↦⟨y,x⟩−ψ(x) (or by the chain rule when ∇2ψ(z)≻0, which gives ∇ψ∗(y)=z(y)+(Dz(y))⊤(y−∇ψ(z(y)))=z(y)), we have ∇ψ∗(y)=z(y), so ∇ψ∗=(∇ψ)−1.
For (2), substitute the identity −ψ(z)=ψ∗(∇ψ(z))−⟨∇ψ(z),z⟩ from (1) into the definition of Bψ(x,z): Bψ(x,z)=ψ(x)−ψ(z)−⟨∇ψ(z),x−z⟩=ψ(x)+ψ∗(∇ψ(z))−⟨∇ψ(z),x⟩, which is nonnegative by the Young–Fenchel inequality.
Example 8 (Fenchel conjugates of the Euclidean norm and negative entropy) Let us compute ψ∗(y) and ∇ψ∗(y) explicitly for our two canonical regularizers:
Squared Euclidean norm ψ(x)=21∥x∥22 on V=Rn:ψ∗(y)=x∈Rnsup{⟨y,x⟩−21∥x∥22}. Setting the gradient y−x=0 gives the maximizer x=y, so ψ∗(y)=21∥y∥22 and ∇ψ∗(y)=y. The Young–Fenchel gap is 21∥x∥22+21∥y∥22−⟨y,x⟩=21∥x−y∥22.
Negative entropy ψ(p)=∑i=1np(i)lnp(i) restricted to the simplex D=Δn:ψ∗(y)=p∈Δnmax{i=1∑ny(i)p(i)−i=1∑np(i)lnp(i)}. Solving via a Lagrange multiplier for ∑i=1np(i)=1, the maximizer is the Gibbs (softmax) distribution p∗(i)=∑j=1ney(j)ey(i)=(∇ψ∗(y))i. Substituting p∗ back into the objective shows that the Fenchel conjugate of negative entropy on Δn is the log-sum-exp (log-partition) function: ψ∗(y)=i=1∑np∗(i)(y(i)−y(i)+lnj=1∑ney(j))=lnj=1∑ney(j).
Equivalence of proximal OMD and two-step Bregman projection
In §1.8 (Proposition 2) we proved that the proximal update xs+1=argminx∈K{ηs⟨∇ℓs(xs),x⟩+Bψ(x,xs)} coincides with the two-step Bregman projection ΠKψ(xs+1′)=argminx∈KBψ(x,xs+1′). Let us record the generalized Pythagorean inequality that holds for any Bregman projection onto a closed convex set K⊆D.
Proposition 5 (Generalized Pythagorean inequality for Bregman projections) Let ψ:D→R be a Legendre function and let K⊆D be closed and convex. For any xs+1′∈int(D), its Bregman projection xs+1=ΠKψ(xs+1′)=argminx∈KBψ(x,xs+1′) satisfies, for every comparator x∗∈K, ⟨∇ψ(xs+1′)−∇ψ(xs+1),x∗−xs+1⟩≤0 and consequently Bψ(x∗,xs+1)+Bψ(xs+1,xs+1′)≤Bψ(x∗,xs+1′).
Proof. First-order optimality of xs+1=argminx∈KBψ(x,xs+1′) gives ⟨∇xBψ(xs+1,xs+1′),x∗−xs+1⟩=⟨∇ψ(xs+1)−∇ψ(xs+1′),x∗−xs+1⟩≥0. Applying the Three-Point Identity (Lemma 4) with (x,y,z)=(x∗,xs+1,xs+1′) yields Bψ(x∗,xs+1′)−Bψ(x∗,xs+1)−Bψ(xs+1,xs+1′)=⟨∇ψ(xs+1)−∇ψ(xs+1′),x∗−xs+1⟩≥0, which rearranges to Bψ(x∗,xs+1)+Bψ(xs+1,xs+1′)≤Bψ(x∗,xs+1′).
2.3 Proof of Pinsker’s inequality: strong convexity of negative entropy in ℓ1
In §§1.8–1.9 we analyzed Online Mirror Descent using the local inverse-Hessian norm ∥g∥(∇2ψ(ξ))−12=g⊤(∇2ψ(ξ))−1g. A complementary global viewpoint bounds the Hessian from below with respect to a fixed norm ∥⋅∥ on V: if ψ is 1-strongly convex with respect to ∥⋅∥ in the sense that h⊤∇2ψ(ξ)h≥∥h∥2 for all h, then its inverse Hessian is bounded by the squared dual norm, ∥g∥(∇2ψ(ξ))−12≤∥g∥∗2:=∥h∥≤1sup⟨g,h⟩2. For negative Shannon entropy ψ(p)=∑i=1np(i)lnp(i) on Δn, the natural primal norm is the ℓ1 norm ∥p−q∥1=∑i=1n∣p(i)−q(i)∣ (whose dual norm is ∥g∥∞=maxi∣g(i)∣). The fact that ψ is 1-strongly convex on Δn with respect to ∥⋅∥1 is Pinsker’s inequality, which we will use again in Chapter 5 to prove information-theoretic lower bounds.
History of Pinsker’s inequality and its sharp constant. Mark Pinsker (Information and Information Stability of Random Variables and Processes, 1960) originally proved that total variation distance is bounded by a multiple of KL(p∥q) with a suboptimal constant. The sharp inequality KL(p∥q)≥21∥p−q∥12=2dTV(p,q)2 was established independently by Imre Csiszár (1967), Solomon Kullback (1967), and Johannes Kemperman (1969).
Theorem 12 (Pinsker’s inequality) For any two probability distributions p,q∈Δn, Bψ(p,q)=KL(p∥q)=i=1∑np(i)lnq(i)p(i)≥21∥p−q∥12=2dTV(p,q)2. In particular, ψ(p)=∑i=1np(i)lnp(i) is 1-strongly convex on Δn with respect to ∥⋅∥1.
Proof. We proceed in two steps: first we reduce the n-state inequality to a two-state Bernoulli distribution, and then we prove the scalar Bernoulli case by calculus.
Step 1 (Coarse-graining to two states via the log-sum inequality): Define the subset of coordinates where p exceeds q: A:={i∈[n]:p(i)≥q(i)},Ac=[n]∖A={i∈[n]:p(i)<q(i)}. Let u:=p(A)=∑i∈Ap(i)∈[0,1] and v:=q(A)=∑i∈Aq(i)∈[0,1]. As we saw in Chapter 2, the ℓ1 distance satisfies ∥p−q∥1=i∈A∑(p(i)−q(i))+i∈Ac∑(q(i)−p(i))=(u−v)+((1−v)−(1−u))=2(u−v). Meanwhile, since t↦tlnt is convex on (0,∞), Jensen’s inequality applied to the weights q(i)/q(A) on A gives i∈A∑p(i)lnq(i)p(i)=q(A)i∈A∑q(A)q(i)(q(i)p(i)lnq(i)p(i))≥q(A)(q(A)p(A)lnq(A)p(A))=ulnvu. Applying the same inequality on Ac and adding the two parts shows that coarse-graining [n] into {A,Ac} can only decrease KL divergence: KL(p∥q)≥ulnvu+(1−u)ln1−v1−u=:dKL(u∥v).
Step 2 (Scalar calculus for binary KL divergence): Since ∥p−q∥1=2(u−v), it remains to show that for all 0≤v≤u≤1, g(u,v):=ulnvu+(1−u)ln1−v1−u−2(u−v)2≥0. Fix u∈(0,1) and view h(v):=g(u,v) as a function of v∈(0,u]. At v=u, we have h(u)=0. Differentiating with respect to v gives h′(v)=−vu+1−v1−u+4(u−v)=(u−v)(4−v(1−v)1). Since v(1−v)≤41 for all v∈(0,1) by AM–GM, we have v(1−v)1≥4, and since v≤u, it follows that h′(v)≤0 for all v∈(0,u]. Therefore h(v) is nonincreasing on (0,u], which implies g(u,v)=h(v)≥h(u)=0.
2.4 Concrete instantiation of Plotkin–Shmoys–Tardos: minimum-congestion routing
To see how the Plotkin–Shmoys–Tardos oracle framework (§1.11) turns a coupled linear program into a sequence of fast combinatorial subproblems, let us instantiate it on the fractional minimum-congestion multicommodity flow problem.
Problem setup and decomposing the constraints
Let G=(V,E) be a directed graph with m=∣E∣ edges, where each edge e∈E has capacity ce>0. We are given k commodities (traffic demands) (s1,t1,d1),…,(sk,tk,dk), where commodity r∈[k] must route dr>0 units of flow from source sr∈V to sink tr∈V. Given a target congestion factor λ>0, we want to decide whether there exist flows f1,…,fk such that:
Individual flow conservation (P): For each commodity r∈[k], fr is a valid sr–tr flow of value dr in which no single commodity sends more than λce along any edge e. Setting P:=P1×⋯×Pk, where Pr:={fr∈R≥0E:fr routes dr units from sr to tr and fr(e)≤λce∀e∈E}, linear optimization over P decomposes into k independent single-commodity flow problems.
Coupling edge-capacity constraints (Ax≥b): Across all k commodities combined, the total flow on each edge e∈E does not exceed λce. Dividing by λce to normalize all right-hand sides to 1, we write these m coupling constraints in the standard form Aef≥be: 1−r=1∑kλcefr(e)≥0for every e∈E.
What does the average-constraint oracle do?
For any probability distribution p=(p(e))e∈E∈ΔE over the m edges, the collapsed average constraint asks whether there exists f=(f1,…,fk)∈P such that e∈E∑p(e)(1−r=1∑kλcefr(e))≥0⟺r=1∑ke∈E∑(λcep(e))fr(e)≤1. Look at the left-hand side of the second inequality: the k commodities are now completely decoupled! Assigning each edge e∈E the nonnegative length ℓp(e):=λcep(e), minimizing ∑r=1k∑e∈Eℓp(e)fr(e) over f=(f1,…,fk)∈P1×⋯×Pk simply requires solving kindependent single-commodity minimum-cost flow problems—one for each commodity r∈[k]!
What is the width ρ of this oracle? For any f∈P, each commodity satisfies 0≤fr(e)≤λce, so 0≤∑r=1kλcefr(e)≤k. Thus the normalized slack ℓt(e)=1−∑r=1kλceft,r(e) lies in [−(k−1),1]⊆[−k,k], giving width ρ≤k. By Theorem 11, T=⌈4k2ε−2lnm⌉ iterations of Multiplicative Weights—each solving k independent single-commodity flow problems under edge lengths ℓpt(e)=pt(e)/(λce)—either certify that congestion λ is impossible or produce a multicommodity flow fˉT=T1∑t=1Tft∈P whose total load on every edge e∈E is at most (1+ε)λce.
2.5 Online-to-batch conversion for stochastic convex optimization
In §§1.4–1.9 we designed Online Convex Optimization algorithms that guarantee ∑t=1T(ℓt(xt)−ℓt(x∗))≤R(T) against arbitrary sequences ℓ1,…,ℓT. As an immediate dividend, any such online algorithm can be used directly as a stochastic convex optimization solver when the functions ℓt are drawn i.i.d. from an unknown population distribution.
Theorem 13 (Online-to-batch conversion) Let K⊆Rn be a compact convex set, and suppose we wish to minimize a population risk F(x):=Eζ∼D[f(x;ζ)](x∈K), where x↦f(x;ζ) is convex for every ζ. Given T i.i.d. samples ζ1,…,ζT∼D, feed the random loss functions ℓt(x):=f(x;ζt) sequentially into any Online Convex Optimization algorithm producing iterates x1,…,xT∈K (where xt depends only on ζ1,…,ζt−1). Then the averaged output xˉT:=T1∑t=1Txt∈K satisfies, for every x∗∈K, E[F(xˉT)]−F(x∗)≤TE[R(T)]. In particular, when ψ is 1-strongly convex w.r.t. ∥⋅∥ and E[∥∇ℓt(xt)∥∗2]≤G2, running Online Mirror Descent (Theorem 6) with step size η=2Bψ(x∗,x1)/(TG2) achieves E[F(xˉT)]−F(x∗)≤GT2Bψ(x∗,x1) for x∗∈argminx∈KF(x).
Proof. Because F is convex on K, Jensen’s inequality gives F(xˉT)=F(T1t=1∑Txt)≤T1t=1∑TF(xt). Now fix any round t. Conditional on the past samples Ft−1=(ζ1,…,ζt−1), the iterate xt is deterministic while ζt∼D is drawn fresh and independently of Ft−1. Therefore, E[ℓt(xt)∣Ft−1]=Eζt∼D[f(xt;ζt)∣xt]=F(xt), and similarly E[ℓt(x∗)]=F(x∗) for any fixed x∗∈K. Taking full expectations and applying RegT(x∗)=∑t=1T(ℓt(xt)−ℓt(x∗))≤R(T) yields E[F(xˉT)]−F(x∗)≤T1t=1∑TE[F(xt)−F(x∗)]=T1E[t=1∑T(ℓt(xt)−ℓt(x∗))]=TE[RegT(x∗)]≤TE[R(T)].
III. Further Topics
The following topics connect the online learning and regularization toolkit to broader themes in optimization, machine learning, and semidefinite relaxations. Each entry supplies background, main results, and references for independent exploration.
3.1 Lazy versus greedy projection: when FTRL and Mirror Descent diverge
Background. In §1.8 we saw that Online Mirror Descent applies a greedy Bregman projection xs+1=ΠKψ(xs+1′) at every round. On an unconstrained space V=Rn (or on the simplex Δn with negative entropy, whose steep gradient at the boundary keeps iterates strictly in the interior), Online Mirror Descent initialized at x0∈argminx∈Kψ(x) with constant step size η produces the exact same sequence of iterates as Follow the Regularized Leader (FTRL): xs+1FTRL:=argx∈Kmin{ητ=0∑s⟨∇ℓτ(xτ),x⟩+ψ(x)}. What happens on a general constrained convex body K (such as the box [−1,1]n or a clipped simplex) when the unconstrained mirror step ∇ψ(xs+1′)=∇ψ(xs)−η∇ℓs(xs) leaves K and hits the boundary?
Main distinction. On a general constrained set K, the two algorithms handle boundary projections differently:
FTRL (Lazy projection / Dual Averaging): Accumulates all historical gradients in the dual space without projecting intermediate points, ys+1=−η∑τ=0s∇ℓτ(xτ), and maps the cumulative sum back to K once at round s: xs+1FTRL=argx∈Kmin{⟨−ys+1,x⟩+ψ(x)}.
OMD (Greedy projection): Projects back onto K at every round, overwriting the dual state with ∇ψ(xs) before subtracting η∇ℓs(xs): xs+1OMD=argx∈Kmin{η⟨∇ℓs(xs),x⟩+Bψ(x,xsOMD)}.
With a fixed learning rate η, both algorithms satisfy the same O(T) regret bound. Under time-varying step sizes ηs=1/s, however, greedy OMD can suffer linear regret on specific adversarial sequences whenever the Bregman diameter supx,y∈KBψ(x,y) is unbounded (even on a compact domain like Δn), whereas FTRL / Nesterov Dual Averaging maintains O(T) regret under bounded gradients and bounded regularizer range because its dual state ys+1 never forgets past gradients clipped by boundary projections.
3.2 Boosting weak learners via Multiplicative Weights: AdaBoost
Background. Given m labeled training examples (z1,y1),…,(zm,ym)∈Z×{−1,+1}, suppose we have access to a weak learning oracle that, for any probability distribution p∈Δm over the m training examples, can find a hypothesis h:Z→{−1,+1} with classification accuracy at least 21+γ (where γ>0 is a small edge over random guessing). Can we combine weak hypotheses into a single composite classifier H(z)=sgn(∑t=1Tαtht(z)) that achieves zero training error on all m examples?
Main result. The fixed-rate Multiplicative Weights boosting prototype underlying Freund and Schapire’s AdaBoost algorithm (which refines the vote weights to αt=21lnεt1−εt) is a direct instance of the MWU game (§§1.9–1.11):
The m training examples act as the m “experts” (or constraints), weighted by pt∈Δm.
At round t, the weak learner returns a hypothesis ht whose pt-weighted accuracy is at least 21+γ.
The booster assigns each example i∈[m] a loss ℓt(i)=1{ht(zi)=yi} with learning rate η=8(lnm)/T (so correctly classified examples receive loss 1 and have their unnormalized weights decreased, forcing the next weak learner ht+1 to focus on the currently misclassified examples!).
By the Multiplicative Weights guarantee (Theorem 7), if an unweighted majority vote (αt=1) of h1,…,hT still errs on any training example i∗∈[m], that example has cumulative loss LT(i∗)≤T/2, whereas the weak learner guarantees ⟨pt,ℓt⟩≥21+γ at every round. The regret bound γT≤(Tlnm)/2 immediately implies that the training error of the combined classifier drops to 0 as soon as T>2γ2lnm.
3.3 Matrix Multiplicative Weights and semidefinite optimization
Background. In §§1.9–1.11 our decision variable was a probability vector xt∈Δn, which can be viewed as a nonnegative diagonal matrix diag(xt) of trace 1. In Chapter 6 we will study semidefinite programs (SDPs), where the optimization variable is a symmetric positive semidefinite matrix X⪰0. Can we generalize Multiplicative Weights and negative-entropy Mirror Descent from the probability simplex Δn to the spectraplexSn:={X∈Rn×n:X=X⊤,X⪰0,Tr(X)=1} of density matrices?
Main result. Replacing the coordinatewise negative entropy ∑i=1nxilnxi by the negative von Neumann quantum entropyψ(X):=Tr(XlnX) (where XlnX=∑i=1nλi(X)lnλi(X)vivi⊤ in the eigenbasis of X) yields the Matrix Multiplicative Weights (MMW) algorithm: given symmetric loss matrices M1,…,MT∈Rn×n with −I⪯Mt⪯I and learning rate 0<η≤1, update Wt+1:=exp(−ηs=1∑tMs),Xt+1:=Tr(Wt+1)Wt+1∈Sn. Because matrices do not commute (eA+B=eAeB), the one-step potential analysis uses the Golden–Thompson inequalityTr(eA+B)≤Tr(eAeB) to establish the matrix regret bound t=1∑TTr(XtMt)−λmin(t=1∑TMt)≤ηlnn+ηt=1∑TTr(XtMt2). Just as scalar MWU solves packing/covering LPs in §1.11, Matrix Multiplicative Weights solves semidefinite programs (such as Max-Cut, balanced separator, and spectral sparsification) in O(ε−2lnn) oracle iterations.
Background. In §1.9 we proved that running Online Stochastic Mirror Descent with negative entropy ψ(x)=∑i=1nxilnxi on importance-weighted bandit losses (EXP3) achieves R(T)=O(nTlnn). Notice two open questions raised by this bound:
Can we remove the extra lnn factor and achieve Θ(nT) minimax regret in adversarial bandits?
In Chapter 1 we saw that when losses are drawn i.i.d. with suboptimality gaps Δi=μi−μi∗>0, UCB achieves logarithmic regret O(∑i:Δi>0ΔilnT). Can a single algorithm achieve O(nT) against an adversary while automatically adapting to achieve O(∑iΔilnT) whenever the environment happens to be stochastic—without knowing in advance which world it is in?
Main result. Both questions are answered simultaneously by Tsallis-INF, which runs FTRL on importance-weighted losses with a scheduled learning rate ηt and replaces negative Shannon entropy by the negative 1/2-Tsallis entropyψ1/2(x):=−2i=1∑nxi(x∈Δn). Because ∇2ψ1/2(x)=diag(21x1−3/2,…,21xn−3/2) has curvature xi−3/2 instead of negative entropy’s xi−1, the local inverse-Hessian norm 2∑i=1nxt(i)3/2ℓ^t(i)2 has conditional expectation 2∑i=1nxt(i)ℓt(i)2≤2n (by Cauchy–Schwarz ∑ixt(i)≤n), while the Bregman radius maxx∗∈ΔnBψ1/2(x∗,1/n)≤2n also scales as n. Balancing these two n factors in Tsallis-INF (Audibert–Bubeck 2009; Zimmert–Seldin 2021) yields:
R(T)≤4nT in the adversarial regime (eliminating the lnn factor completely); and
R(T)=O(∑i:Δi>0ΔilnT) in the stochastic i.i.d. regime via a self-bounding regret analysis.
References and source acknowledgments. The progression from the Online Number Guessing Game and Follow the Leader (§§1.1–1.2) to offline and online Projected Gradient Descent (§§1.3–1.4), reductions of Experts, Online Shortest Paths, and Euclidean OSGD Bandits (§§1.5–1.6), Online Mirror Descent (§§1.7–1.8) and its continuous-time Bregman discretization proof (§2.1), and the first-principles derivation of Multiplicative Weights (Hedge) and EXP3 (§1.9) is adapted directly from Chihao Zhang’s CS3958 Advanced Algorithms lecture notes [1], which in turn draw on Francesco Orabona [2] and Yin Tat Lee and Santosh S. Vempala’s notes on Convex Optimization [3]. The game-theoretic and algorithmic applications of Multiplicative Weights to zero-sum minimax duality (§1.10) and the Plotkin–Shmoys–Tardos LP feasibility framework (§1.11, §2.4) follow Sanjeev Arora, Elad Hazan, and Satyen Kale [4] and Plotkin, Shmoys, and Tardos [9]. Further developments of Mirror Descent, Fenchel conjugation, and bandit optimization draw on Nemirovski and Yudin [5], Hazan [6], Bubeck [7], Auer et al. [8], and Lattimore and Szepesvári [10].
Yin Tat Lee and Santosh S. Vempala, Theory and Algorithms for Continuous Optimization (Lecture notes on Mirror Descent and continuous-time potentials).