Topics in Advanced AlgorithmsFall 2026

Chapter 4

Online Learning, Regularization, and Optimization

From Follow the Leader and continuous-time gradient flows to Mirror Descent, Multiplicative Weights, and minimax duality

In Chapter 1 we studied sequential decision-making under a benign statistical assumption: in the stochastic multi-armed bandit problem, each action draws rewards independently from a fixed, unknown probability distribution, so empirical averages concentrate around true means by Hoeffding’s inequality. What happens when we drop the probability model altogether? Whether a player is guessing numbers against an adaptive opponent, a router is sending packets through a network with fluctuating traffic, or an optimization algorithm is iteratively adjusting dual multipliers to satisfy coupling constraints, the environment is rarely independent and identically distributed—it may be non-stationary, reactive, or actively adversarial.

This chapter develops the theory of online learning from first principles across two meetings. In our first meeting, we start with a concrete online number guessing game under quadratic loss (xt−yt)2(x_t-y_t)^2, where the greedy Follow-the-Leader (FTL) rule—playing the historical average xt=1t−1∑s=1t−1ysx_t=\frac{1}{t-1}\sum_{s=1}^{t-1}y_s—achieves logarithmic regret 2HT=Θ(log⁡T)2H_T=\Theta(\log T). Yet as soon as the loss functions become linear, FTL oscillates between extreme points and suffers linear regret Ω(T)\Omega(T), and in general solving the offline optimization problem arg⁡min⁡x∈V∑s=1t−1ℓs(x)\arg\min_{x\in V}\sum_{s=1}^{t-1}\ell_s(x) from scratch each round is already non-trivial. We therefore step back to study offline convex optimization min⁡x∈Vf(x)\min_{x\in V}f(x) and prove the convergence of offline Projected Gradient Descent via the Euclidean potential ϕ(x)=12∥x−x∗∥22\phi(x)=\frac12\|x-x^*\|_2^2, before moving to Online Projected Gradient Descent (OGD)—which achieves O(T)O(\sqrt T) regret for general convex losses and O(log⁡T)O(\log T) regret for μ\mu-strongly convex losses. Applying Euclidean OGD to combinatorial problems via convex lifting then solves prediction with expert advice on the simplex Δn\Delta_n with O(nT)O(\sqrt{nT}) regret and online shortest paths via probability flows.

In our second meeting, we first extend Euclidean OGD to adversarial multi-armed bandits via importance-weighted Online Stochastic Gradient Descent (OSGD) with O((nT)2/3)O((nT)^{2/3}) regret, and then confront the two sharp puzzles exposed by Euclidean OGD on the probability simplex Δn\Delta_n: why does full-information expert prediction pay a polynomial dimension factor n\sqrt n, and why does bandit feedback degrade from T1/2T^{1/2} to (nT)2/3(nT)^{2/3} because the variance ∑i=1nℓt(i)2xt(i)\sum_{i=1}^n\frac{\ell_t(i)^2}{x_t(i)} blows up near the boundary of Δn\Delta_n? Both bottlenecks stem from the Euclidean regularizer 12η∥x−xt∥22\frac{1}{2\eta}\|x-x_t\|_2^2, whose flat identity Hessian ignores the geometry of probability distributions. Replacing 12∥x−xt∥22\frac12\|x-x_t\|_2^2 by a general Bregman divergence Bψ(x,xt)B_\psi(x,x_t) leads to Online Mirror Descent (OMD), whose master regret bound (proved via continuous-time Bregman flow discretization in §2.1) governs the second-order penalty by the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12\|\nabla\ell_t(x_t)\|_{(\nabla^2\psi(\xi_t))^{-1}}^2. Requiring (∇2ψ(x))−1=diag⁡(x1,…,xn)(\nabla^2\psi(x))^{-1}=\operatorname{diag}(x_1,\ldots,x_n) on Δn\Delta_n uniquely forces negative Shannon entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i—whose inverse mirror map (∇ψ)−1(\nabla\psi)^{-1} derives the exponential weight update of Multiplicative Weights Update (MWU / Hedge) and EXP3 (Online Stochastic Mirror Descent) from first principles, improving experts regret to O(Tln⁡n)O(\sqrt{T\ln n}) and bandit regret to O(nTln⁡n)O(\sqrt{nT\ln n})! Armed with MWU, we conclude by turning online no-regret play into a universal offline solver for von Neumann’s minimax theorem in zero-sum games and the Plotkin–Shmoys–Tardos (PST) oracle framework for approximate linear feasibility.

Reading guide. Course Content is organized into two 120-minute meetings following a first-principles progression:

  • Meeting 1 (§§1.1–1.5): Formulates the online number guessing game and proves the 2HT=Θ(log⁡T)2H_T=\Theta(\log T) regret bound of Follow the Leader via the inductive Be-The-Leader lemma (§1.1); introduces the general online learning protocol and exhibits FTL’s Ω(T)\Omega(T) oscillation on alternating linear losses (§1.2); develops offline convex optimization and proves the convergence of offline Gradient Descent and Projected Gradient Descent (§1.3); states the O(T)O(\sqrt T) regret bound of Online Gradient Descent and proves its O(log⁡T)O(\log T) refinement under μ\mu-strong convexity (§1.4); and reduces discrete expert prediction (O(nT))\bigl(O(\sqrt{nT})\bigr) and online shortest paths (via probability flows) to OGD (§1.5).
  • Meeting 2 (§§1.6–1.11): Extends Euclidean OGD to adversarial multi-armed bandits via importance-weighted Online Stochastic Gradient Descent with α\alpha-uniform exploration (O((nT)2/3))\bigl(O((nT)^{2/3})\bigr) (§1.6); reformulates projected gradient descent via a local quadratic regularizer, diagnoses the (ℓ2,ℓ2)(\ell_2,\ell_2) versus (ℓ1,ℓ∞)(\ell_1,\ell_\infty) mismatch and flat boundary curvature on Δn\Delta_n, and introduces Bregman divergences (§1.7); develops Online Mirror Descent from four complementary perspectives (primal proximal, local Riemannian metric, dual mirror space, and FTRL accumulation) and states its master regret bound in terms of the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12\|\nabla\ell_t(x_t)\|_{(\nabla^2\psi(\xi_t))^{-1}}^2 (§1.8); instantiates OMD with negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i to derive Multiplicative Weights Update (MWU / Hedge) (O(Tln⁡n))\bigl(O(\sqrt{T\ln n})\bigr) and EXP3 / Online Stochastic Mirror Descent (O(nTln⁡n))\bigl(O(\sqrt{nT\ln n})\bigr) from first principles (§1.9); and applies MWU to prove von Neumann’s minimax theorem for zero-sum matrix games (§1.10) and the Plotkin–Shmoys–Tardos oracle theorem for linear feasibility (§1.11).

Reading Contents (§§2.1–2.5) presents the unified continuous-time discretization proof of Online Mirror Descent and Online Gradient Descent (§2.1), Fenchel conjugation and the generalized Pythagorean inequality (§2.2), Pinsker’s inequality (§2.3), minimum-congestion multicommodity routing via PST (§2.4), and online-to-batch conversion (§2.5). Further Topics (§§3.1–3.4) covers lazy versus greedy projection (FTRL vs. OMD), AdaBoost, Matrix Multiplicative Weights for SDPs, and Tsallis-INF.

I. Course Content

1.1 An online number guessing game and Follow the Leader

The multi-armed bandit problem of Chapter 1 was our first example of sequential decision-making under uncertainty. To see what changes when we move from stationary probability distributions to arbitrary adversarial sequences, let us begin with a simple quantitative game.

The quadratic number guessing game: stochastic versus adversarial

The online number guessing game lasts for TT rounds. In each round t=1,2,…,Tt=1,2,\ldots,T:

  1. The adversary chooses a target number yt∈[0,1]y_t\in[0,1];
  2. The player picks a guess xt∈[0,1]x_t\in[0,1] (without knowing yty_t);
  3. The target yty_t is revealed, and the player pays the quadratic cost (xt−yt)2(x_t-y_t)^2.

The player’s goal is to keep the cumulative cost ∑t=1T(xt−yt)2\sum_{t=1}^T(x_t-y_t)^2 as small as possible. How the player should act depends on how the sequence y1,…,yTy_1,\ldots,y_T is generated. Consider two contrasting regimes:

Why benchmark against a fixed comparator x∗x^*? In an adversarial environment, no algorithm can guarantee a small raw cost ∑t=1T(xt−yt)2\sum_{t=1}^T(x_t-y_t)^2 in isolation: if the adversary sets yt=1y_t=1 whenever xt≤1/2x_t\le 1/2 and yt=0y_t=0 whenever xt>1/2x_t>1/2, the player pays at least T/4T/4. Subtracting min⁡x∈[0,1]∑t=1T(x−yt)2\min_{x\in[0,1]}\sum_{t=1}^T(x-y_t)^2 calibrates the player’s loss against the best static compromise xT∗x^*_T in hindsight, connecting online play directly to offline optimization.

Hannan consistency and Blackwell approachability. James Hannan (Approximation to Bayes Risk in Repeated Play, 1957) proved the first O(T)O(\sqrt T) external-regret guarantee in repeated games shortly after David Blackwell (1956) established his vector-valued Approachability Theorem. In game theory, an online strategy whose worst-case regret satisfies lim sup⁡T→∞R(T)/T≤0\limsup_{T\to\infty} R(T)/T\le 0 is called Hannan consistent (or no-regret).

The Follow-the-Leader algorithm and the Be-The-Leader lemma

How should we play the number guessing game online? Let us first solve the offline optimization problem over the full horizon: given y1,…,yT∈[0,1]y_1,\ldots,y_T\in[0,1], which fixed point x∈[0,1]x\in[0,1] minimizes FT(x):=∑t=1T(x−yt)2F_T(x):=\sum_{t=1}^T(x-y_t)^2? Setting the derivative FT′(x)=2∑t=1T(x−yt)F_T'(x)=2\sum_{t=1}^T(x-y_t) to zero yields the empirical mean: xT∗:=arg⁡min⁡x∈[0,1]∑t=1T(x−yt)2=1T∑t=1Tyt. x^*_T:=\arg\min_{x\in[0,1]}\sum_{t=1}^T(x-y_t)^2=\frac{1}{T}\sum_{t=1}^T y_t. When round tt begins in the online game, the player only knows the prefix y1,…,yt−1y_1,\ldots,y_{t-1}. If the game had ended at round t−1t-1, the offline optimum would have been the prefix average xt−1∗:=arg⁡min⁡x∈[0,1]∑s=1t−1(x−ys)2=1t−1∑s=1t−1ys. x^*_{t-1}:=\arg\min_{x\in[0,1]}\sum_{s=1}^{t-1}(x-y_s)^2=\frac{1}{t-1}\sum_{s=1}^{t-1}y_s. A natural online heuristic is therefore to play xt=xt−1∗x_t=x^*_{t-1}—that is, Follow the Leader (FTL) from the previous round! Remarkably, on quadratic losses, this simple greedy rule achieves O(log⁡T)O(\log T) regret against any adversarial sequence.

Theorem 1 (Logarithmic regret of Follow the Leader on the number guessing game) Let y1,…,yT∈[0,1]y_1,\ldots,y_T\in[0,1] be an arbitrary sequence of numbers. Suppose the player chooses x1∈[0,1]x_1\in[0,1] arbitrarily in round 11 and plays the Follow-the-Leader iterate xt:=xt−1∗=1t−1∑s=1t−1ysfor t=2,…,T, x_t:=x^*_{t-1}=\frac{1}{t-1}\sum_{s=1}^{t-1}y_s\qquad\text{for }t=2,\ldots,T, where xt∗=1t∑s=1tysx^*_t=\frac{1}{t}\sum_{s=1}^t y_s denotes the offline optimum through round tt. Then R(T)=∑t=1T(xt−yt)2−min⁡x∈[0,1]∑t=1T(x−yt)2≤2HT=Θ(log⁡T), R(T)=\sum_{t=1}^T(x_t-y_t)^2-\min_{x\in[0,1]}\sum_{t=1}^T(x-y_t)^2\le 2 H_T=\Theta(\log T), where HT:=∑t=1T1t≤1+ln⁡TH_T:=\sum_{t=1}^T\frac{1}{t}\le 1+\ln T is the TT-th harmonic number.

Proof. We first establish a general structural inequality—often called the Be-The-Leader (BTL) inequality—stating that the cumulative cost of the non-adaptive global optimum xT∗x^*_T is at least the cumulative cost of the sequence of adaptive optima xt∗x^*_t: ∑t=1T(xT∗−yt)2≥∑t=1T(xt∗−yt)2. \sum_{t=1}^T(x^*_T-y_t)^2\ge\sum_{t=1}^T(x^*_t-y_t)^2. We prove this inequality by induction on T≥1T\ge 1. For T=1T=1, both sides equal (x1∗−y1)2=0(x^*_1-y_1)^2=0. For the inductive step from T−1T-1 to TT, note that the t=Tt=T term on both sides is identical, namely (xT∗−yT)2(x^*_T-y_T)^2. Subtracting (xT∗−yT)2(x^*_T-y_T)^2 from both sides shows that the claim for TT is equivalent to ∑t=1T−1(xT∗−yt)2≥∑t=1T−1(xt∗−yt)2. \sum_{t=1}^{T-1}(x^*_T-y_t)^2\ge\sum_{t=1}^{T-1}(x^*_t-y_t)^2. By the induction hypothesis at horizon T−1T-1, and the fact that xT−1∗x^*_{T-1} minimizes ∑t=1T−1(x−yt)2\sum_{t=1}^{T-1}(x-y_t)^2 over all x∈[0,1]x\in[0,1] (in particular, it beats the candidate x=xT∗x=x^*_T), we have ∑t=1T−1(xt∗−yt)2≤∑t=1T−1(xT−1∗−yt)2≤∑t=1T−1(xT∗−yt)2, \sum_{t=1}^{T-1}(x^*_t-y_t)^2\le\sum_{t=1}^{T-1}(x^*_{T-1}-y_t)^2\le\sum_{t=1}^{T-1}(x^*_T-y_t)^2, completing the induction.

Using the Be-The-Leader inequality, we can upper-bound the regret by comparing xt=xt−1∗x_t=x^*_{t-1} against xt∗x^*_t term by term: R(T)=∑t=1T(xt−yt)2−∑t=1T(xT∗−yt)2≤∑t=1T[(xt−yt)2−(xt∗−yt)2]. R(T)=\sum_{t=1}^T(x_t-y_t)^2-\sum_{t=1}^T(x^*_T-y_t)^2 \le\sum_{t=1}^T\Bigl[(x_t-y_t)^2-(x^*_t-y_t)^2\Bigr]. For t=1t=1, since x1,y1∈[0,1]x_1,y_1\in[0,1] and x1∗=y1x^*_1=y_1, the first term is (x1−y1)2≤1<21(x_1-y_1)^2\le 1<\frac{2}{1}. For each round t=2,…,Tt=2,\ldots,T, factoring the difference of squares with xt=xt−1∗x_t=x^*_{t-1} gives (xt−yt)2−(xt∗−yt)2=(xt−1∗+xt∗−2yt)(xt−1∗−xt∗)≤2 ∣xt−1∗−xt∗∣=2 ∣1t−1∑s=1t−1ys−1t∑s=1tys∣=2 ∣1t(t−1)∑s=1t−1ys−1tyt∣=2t ∣xt−1∗−yt∣≤2t, \begin{aligned} (x_t-y_t)^2-(x^*_t-y_t)^2 &=\bigl(x^*_{t-1}+x^*_t-2y_t\bigr)\bigl(x^*_{t-1}-x^*_t\bigr)\\ &\le 2\,\bigl|x^*_{t-1}-x^*_t\bigr|\\ &=2\,\left|\frac{1}{t-1}\sum_{s=1}^{t-1}y_s-\frac{1}{t}\sum_{s=1}^t y_s\right|\\ &=2\,\left|\frac{1}{t(t-1)}\sum_{s=1}^{t-1}y_s-\frac{1}{t}y_t\right| =\frac{2}{t}\,\bigl|x^*_{t-1}-y_t\bigr|\le\frac{2}{t}, \end{aligned} where we used xt−1∗,xt∗,yt∈[0,1]x^*_{t-1},x^*_t,y_t\in[0,1] to bound ∣xt−1∗+xt∗−2yt∣≤2|x^*_{t-1}+x^*_t-2y_t|\le 2 and ∣xt−1∗−yt∣≤1|x^*_{t-1}-y_t|\le 1. Summing from t=1t=1 to TT yields R(T)≤∑t=1T2t=2HTR(T)\le\sum_{t=1}^T\frac{2}{t}=2H_T.

1.2 General online learning and why Follow the Leader fails

We can immediately generalize the number guessing game to an arbitrary decision space V⊆RdV\subseteq\mathbb R^d and general loss functions ℓt:V→R\ell_t:V\to\mathbb R, obtaining the general framework of online learning (or online optimization).

Definition 1 (General online learning protocol and regret) Let V⊆RdV\subseteq\mathbb R^d be a decision set and fix a horizon T≥1T\ge 1. In each round t=1,2,…,Tt=1,2,\ldots,T:

  1. The player chooses an action xt∈Vx_t\in V (possibly at random from a distribution over VV);
  2. The adversary chooses and reveals a loss function ℓt:V→R\ell_t:V\to\mathbb R;
  3. The player pays the cost ℓt(xt)\ell_t(x_t).

The player’s regret over TT rounds with respect to a fixed comparator x∗∈Vx^*\in V and its worst-case regret over VV are Reg⁡T(x∗):=∑t=1Tℓt(xt)−∑t=1Tℓt(x∗),Reg⁡T:=R(T)=∑t=1Tℓt(xt)−min⁡x∈V∑t=1Tℓt(x). \operatorname{Reg}_T(x^*):=\sum_{t=1}^T\ell_t(x_t)-\sum_{t=1}^T\ell_t(x^*), \qquad \operatorname{Reg}_T:=R(T)=\sum_{t=1}^T\ell_t(x_t)-\min_{x\in V}\sum_{t=1}^T\ell_t(x). When the player uses a randomized strategy, we measure expected regret E[R(T)]\mathbb E[R(T)].

The Follow-the-Leader (FTL) strategy applies verbatim to Definition 1: at round tt, play an offline minimizer of the cumulative losses observed so far, xt:=xt−1∗∈arg⁡min⁡x∈V∑s=1t−1ℓs(x),wherext∗∈arg⁡min⁡x∈V∑s=1tℓs(x). x_t:=x^*_{t-1}\in\arg\min_{x\in V}\sum_{s=1}^{t-1}\ell_s(x), \qquad\text{where}\qquad x^*_t\in\arg\min_{x\in V}\sum_{s=1}^t\ell_s(x). Moreover, the inductive proof of the Be-The-Leader inequality in Theorem 1 holds for any sequence of functions ℓ1,…,ℓT:V→R\ell_1,\ldots,\ell_T:V\to\mathbb R: because xT−1∗x^*_{T-1} minimizes ∑t=1T−1ℓt(x)\sum_{t=1}^{T-1}\ell_t(x) over VV, induction on TT gives ∑t=1Tℓt(xT∗)≥∑t=1Tℓt(xt∗)\sum_{t=1}^T\ell_t(x^*_T)\ge\sum_{t=1}^T\ell_t(x^*_t), and therefore R(T)=∑t=1Tℓt(xt−1∗)−∑t=1Tℓt(xT∗)≤∑t=1T(ℓt(xt−1∗)−ℓt(xt∗)). R(T)=\sum_{t=1}^T\ell_t(x^*_{t-1})-\sum_{t=1}^T\ell_t(x^*_T) \le\sum_{t=1}^T\Bigl(\ell_t(x^*_{t-1})-\ell_t(x^*_t)\Bigr).

Look closely at the right-hand side: the regret of Follow the Leader is controlled by the sensitivity of the intermediate optimal solutions, ℓt(xt−1∗)−ℓt(xt∗)\ell_t(x^*_{t-1})-\ell_t(x^*_t)—how much the round-tt loss changes when we replace the round-tt hindsight leader xt∗x^*_t by the round-(t−1)(t-1) leader xt−1∗x^*_{t-1}. In the quadratic number guessing game, the curvature of (x−ys)2(x-y_s)^2 forced the empirical means to stabilize at rate ∣xt−1∗−xt∗∣≤1t|x^*_{t-1}-x^*_t|\le\frac1t. What happens when the loss functions ℓt\ell_t are linear and have no curvature to anchor xt∗x^*_t?

Cumulative slope ∑ aₛ = 0 +1 0 −1 FTL (jump) Regularized

For a linear objective (∑s=1t−1as)x\bigl(\sum_{s=1}^{t-1}a_s\bigr)x on [−1,1][-1,1], the minimizer xt−1∗=−sgn⁡(∑s=1t−1as)x^*_{t-1}=-\operatorname{sgn}\bigl(\sum_{s=1}^{t-1}a_s\bigr) is a step function that jumps across the entire interval whenever the cumulative slope crosses 00.

Example 1 (Failure of Follow the Leader on alternating linear losses) Consider the interval V=[−1,1]V=[-1,1] with linear loss functions ℓt(x)=atx\ell_t(x)=a_t x, where a1=−12a_1=-\frac12 and, for t>1t>1, at={+1,if t is even,−1,if t is odd. a_t=\begin{cases} +1, & \text{if }t\text{ is even},\\[2pt] -1, & \text{if }t\text{ is odd}. \end{cases} Then the cumulative slope At−1:=∑s=1t−1asA_{t-1}:=\sum_{s=1}^{t-1}a_s alternates sign at every single round: A1=−12A_1=-\frac12, A2=+12A_2=+\frac12, A3=−12A_3=-\frac12, A4=+12,…A_4=+\frac12,\ldots. Because minimizing At−1xA_{t-1}x over x∈[−1,1]x\in[-1,1] sets xt=+1x_t=+1 when At−1<0A_{t-1}<0 (even tt) and xt=−1x_t=-1 when At−1>0A_{t-1}>0 (odd t>1t>1), FTL plays xt=atx_t=a_t and pays ℓt(xt)=at2=1\ell_t(x_t)=a_t^2=1 on every round t≥2t\ge 2, whereas the fixed point x∗=0∈[−1,1]x^*=0\in[-1,1] pays 00 on every round!

The exact same oscillation occurs on the two-expert probability simplex Δ2={(p(1),p(2))∈R≥02:p(1)+p(2)=1}\Delta_2=\{(p(1),p(2))\in\mathbb R_{\ge0}^2:p(1)+p(2)=1\} under ℓ1=(1/2,0)\ell_1=(1/2,0) and alternating losses ℓt=(0,1)\ell_t=(0,1) (even tt), ℓt=(1,0)\ell_t=(1,0) (odd t≥3t\ge 3):

Round tt Historical cumulative loss Lt−1L_{t-1} FTL distribution ptp_t Revealed loss ℓt\ell_t Learner’s loss ⟨pt,ℓt⟩\langle p_t,\ell_t\rangle Updated LtL_t
t=1t=1 (0,0)(0,0) (1/2, 1/2)(1/2,\,1/2) (1/2, 0)(1/2,\,0) 1/41/4 (1/2, 0)(1/2,\,0)
t=2t=2 (1/2, 0)(1/2,\,0) (0, 1)(0,\,1) (0, 1)(0,\,1) 11 (1/2, 1)(1/2,\,1)
t=3t=3 (1/2, 1)(1/2,\,1) (1, 0)(1,\,0) (1, 0)(1,\,0) 11 (3/2, 1)(3/2,\,1)
t=4t=4 (3/2, 1)(3/2,\,1) (0, 1)(0,\,1) (0, 1)(0,\,1) 11 (3/2, 2)(3/2,\,2)
t=5t=5 (3/2, 2)(3/2,\,2) (1, 0)(1,\,0) (1, 0)(1,\,0) 11 (5/2, 2)(5/2,\,2)

Over an even horizon TT, FTL suffers total loss 14+(T−1)=T−34\frac14+(T-1)=T-\frac34, whereas the two fixed experts suffer LT(1)=T−12L_T(1)=\frac{T-1}{2} and LT(2)=T2L_T(2)=\frac{T}{2}, giving linear regret: Reg⁡T=(T−34)−T−12=T2−14=Ω(T). \operatorname{Reg}_T=\Bigl(T-\frac34\Bigr)-\frac{T-1}{2}=\frac{T}{2}-\frac14=\Omega(T).

Caution (Why Follow the Leader cannot be our general online engine).

Follow the Leader suffers from two fundamental defects:

  1. Boundary instability on linear losses: When the cumulative objective ∑s=1t−1ℓs(x)\sum_{s=1}^{t-1}\ell_s(x) is linear (or weakly curved), its minimizer xt−1∗x^*_{t-1} jumps discontinuously between extreme points of VV whenever two cumulative slopes cross, allowing the adversary to inflict Ω(1)\Omega(1) sensitivity ℓt(xt−1∗)−ℓt(xt∗)\ell_t(x^*_{t-1})-\ell_t(x^*_t) on every round.
  2. Offline computational overhead: Even when ∑s=1t−1ℓs(x)\sum_{s=1}^{t-1}\ell_s(x) is curved, FTL requires solving the full offline optimization problem arg⁡min⁡x∈V∑s=1t−1ℓs(x)\arg\min_{x\in V}\sum_{s=1}^{t-1}\ell_s(x) from scratch at every step tt, which can be non-trivial or computationally expensive.

Both defects point in the same direction: instead of jumping all the way to the global minimum of ∑s=1t−1ℓs(x)\sum_{s=1}^{t-1}\ell_s(x) at round tt, we should update our previous state xtx_t incrementally using only local first-order information from the most recent loss ℓt\ell_t.

1.3 Convex optimization and the offline Gradient Descent algorithm

Because solving the offline optimization problem arg⁡min⁡x∈V∑s=1t−1ℓs(x)\arg\min_{x\in V}\sum_{s=1}^{t-1}\ell_s(x) from scratch at each round is already non-trivial, let us first step back from the online game and study the offline convex optimization problem: min⁡x∈Vf(x),where V⊆Rn is a convex set and f:V→R is a convex function. \min_{x\in V} f(x), \qquad\text{where }V\subseteq\mathbb R^n\text{ is a convex set and }f:V\to\mathbb R\text{ is a convex function.} Recall that a subset V⊆RnV\subseteq\mathbb R^n is convex if for all x,y∈Vx,y\in V and λ∈[0,1]\lambda\in[0,1], the line segment point λx+(1−λ)y\lambda x+(1-\lambda)y lies in VV. A function f:V→Rf:V\to\mathbb R is convex if its epigraph epi⁡(f):={(x,y)∈V×R:y≥f(x)}\operatorname{epi}(f):=\{(x,y)\in V\times\mathbb R:y\ge f(x)\} is a convex subset of Rn+1\mathbb R^{n+1}.

Throughout this chapter, we rely on four fundamental properties of convex functions:

  1. Jensen’s inequality: For any x,y∈Vx,y\in V and λ∈[0,1]\lambda\in[0,1], f(λx+(1−λ)y)≤λf(x)+(1−λ)f(y)f\bigl(\lambda x+(1-\lambda)y\bigr)\le\lambda f(x)+(1-\lambda)f(y). More generally, for any random variable X∈VX\in V, f(E[X])≤E[f(X)]f(\mathbb E[X])\le\mathbb E[f(X)].
  2. First-order tangent-plane inequality: If ff is differentiable at x∈Vx\in V, then its graph lies everywhere above its tangent hyperplane at xx: f(y)≥f(x)+⟨∇f(x), y−x⟩for all y∈V. f(y)\ge f(x)+\langle\nabla f(x),\,y-x\rangle\qquad\text{for all }y\in V.
  3. Second-order Taylor expansion with Lagrange remainder: If f∈C2(V)f\in C^2(V), then for every x,y∈Vx,y\in V there exists a point ξ\xi on the segment [x,y][x,y] such that f(y)=f(x)+⟨∇f(x), y−x⟩+12(y−x)⊤∇2f(ξ)(y−x). f(y)=f(x)+\langle\nabla f(x),\,y-x\rangle+\frac12(y-x)^\top\nabla^2 f(\xi)(y-x). In particular, ff is convex on VV if and only if its Hessian is positive semidefinite (∇2f(z)⪰0\nabla^2 f(z)\succeq 0) for all z∈Vz\in V.
  4. First-order optimality condition: For a differentiable convex function ff on a convex set VV, x∗∈arg⁡min⁡x∈Rnf(x)  ⟺  ∇f(x∗)=0,andx∗∈arg⁡min⁡x∈Vf(x)  ⟺  ∀y∈V, ⟨∇f(x∗), y−x∗⟩≥0. x^*\in\arg\min_{x\in\mathbb R^n}f(x)\iff\nabla f(x^*)=0, \qquad\text{and}\qquad x^*\in\arg\min_{x\in V}f(x)\iff\forall y\in V,\ \langle\nabla f(x^*),\,y-x^*\rangle\ge 0.

The offline Gradient Descent algorithm

How can we find an approximate minimizer of ff using only local first-order information? Let us first consider the unconstrained case V=RnV=\mathbb R^n. Starting from an initial point x0∈Vx_0\in V, Gradient Descent iteratively updates the current point xtx_t (t=0,1,…,T−1t=0,1,\ldots,T-1) by stepping in the direction of the negative gradient with step size η>0\eta>0: xt+1:=xt−η∇f(xt). x_{t+1}:=x_t-\eta\nabla f(x_t).

xₜ x −∇f(xₜ) x − xₜ

Because ff is convex, −∇f(xt)-\nabla f(x_t) always forms an acute angle with x∗−xtx^*-x_t: ⟨−∇f(xt),x∗−xt⟩≥f(xt)−f(x∗)≥0\langle -\nabla f(x_t),x^*-x_t\rangle\ge f(x_t)-f(x^*)\ge 0.

Why is −∇f(xt)-\nabla f(x_t) a good direction to move? Let x∗∈arg⁡min⁡x∈Vf(x)x^*\in\arg\min_{x\in V}f(x) be an optimal solution. By the first-order tangent-plane inequality (Property 2), f(x∗)≥f(xt)+⟨∇f(xt), x∗−xt⟩⟺⟨−∇f(xt), x∗−xt⟩≥f(xt)−f(x∗)≥0. f(x^*)\ge f(x_t)+\langle\nabla f(x_t),\,x^*-x_t\rangle \quad\Longleftrightarrow\quad \langle-\nabla f(x_t),\,x^*-x_t\rangle\ge f(x_t)-f(x^*)\ge 0. Geometrically, the step direction −∇f(xt)-\nabla f(x_t) always forms an acute angle with the displacement vector x∗−xtx^*-x_t pointing from xtx_t toward the optimum x∗x^*! Consequently, moving a sufficiently small distance η>0\eta>0 along −∇f(xt)-\nabla f(x_t) decreases the Euclidean distance to x∗x^*.

To make this quantitative, define the Euclidean potential function ϕ(x):=12∥x−x∗∥22\phi(x):=\frac12\|x-x^*\|_2^2. Tracking how ϕ\phi changes in a single gradient step yields the fundamental convergence bound of offline Gradient Descent.

Theorem 2 (Convergence bound for offline Gradient Descent) Let f:Rn→Rf:\mathbb R^n\to\mathbb R be a differentiable convex function, let x∗∈Rnx^*\in\mathbb R^n be any comparator point with ϕ(x):=12∥x−x∗∥22\phi(x):=\frac12\|x-x^*\|_2^2, and fix a step size η>0\eta>0. Then the Gradient Descent iterates xt+1=xt−η∇f(xt)x_{t+1}=x_t-\eta\nabla f(x_t) satisfy, for every step t=0,1,…,T−1t=0,1,\ldots,T-1, f(xt)−f(x∗)≤⟨∇f(xt), xt−x∗⟩=ϕ(xt)−ϕ(xt+1)η+η2∥∇f(xt)∥22, f(x_t)-f(x^*)\le\bigl\langle\nabla f(x_t),\,x_t-x^*\bigr\rangle =\frac{\phi(x_t)-\phi(x_{t+1})}{\eta}+\frac{\eta}{2}\bigl\|\nabla f(x_t)\bigr\|_2^2, and summing over t=0,1,…,T−1t=0,1,\ldots,T-1 yields ∑t=0T−1(f(xt)−f(x∗))≤∑t=0T−1⟨∇f(xt), xt−x∗⟩≤ϕ(x0)−ϕ(xT)η+η2∑t=0T−1∥∇f(xt)∥22. \sum_{t=0}^{T-1}\bigl(f(x_t)-f(x^*)\bigr) \le\sum_{t=0}^{T-1}\bigl\langle\nabla f(x_t),\,x_t-x^*\bigr\rangle \le\frac{\phi(x_0)-\phi(x_T)}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\bigl\|\nabla f(x_t)\bigr\|_2^2.

Proof. Expanding the potential difference ϕ(xt+1)−ϕ(xt)\phi(x_{t+1})-\phi(x_t) for xt+1=xt−η∇f(xt)x_{t+1}=x_t-\eta\nabla f(x_t) gives ϕ(xt+1)−ϕ(xt)=12∥xt−η∇f(xt)−x∗∥22−12∥xt−x∗∥22=12⟨−2η∇f(xt), xt−x∗⟩+12η2∥∇f(xt)∥22=−η⟨∇f(xt), xt−x∗⟩+12η2∥∇f(xt)∥22. \begin{aligned} \phi(x_{t+1})-\phi(x_t) &=\frac12\bigl\|x_t-\eta\nabla f(x_t)-x^*\bigr\|_2^2-\frac12\|x_t-x^*\|_2^2\\ &=\frac12\bigl\langle -2\eta\nabla f(x_t),\,x_t-x^*\bigr\rangle+\frac12\eta^2\bigl\|\nabla f(x_t)\bigr\|_2^2\\ &=-\eta\bigl\langle\nabla f(x_t),\,x_t-x^*\bigr\rangle+\frac12\eta^2\bigl\|\nabla f(x_t)\bigr\|_2^2. \end{aligned} Dividing by η>0\eta>0 and rearranging gives the exact identity ⟨∇f(xt),xt−x∗⟩=ϕ(xt)−ϕ(xt+1)η+η2∥∇f(xt)∥22\langle\nabla f(x_t),x_t-x^*\rangle=\frac{\phi(x_t)-\phi(x_{t+1})}{\eta}+\frac{\eta}{2}\|\nabla f(x_t)\|_2^2. Combining this with the tangent-plane inequality f(xt)−f(x∗)≤⟨∇f(xt),xt−x∗⟩f(x_t)-f(x^*)\le\langle\nabla f(x_t),x_t-x^*\rangle and telescoping the sum over t=0,…,T−1t=0,\ldots,T-1 completes the proof.

Averaging, Lipschitzness, and Projected Gradient Descent

Three remarks turn Theorem 2 into a complete offline optimization guarantee:

  1. Averaging: Suppose the right-hand side of Theorem 2 is bounded by R(T)R(T), so that ∑t=0T−1(f(xt)−f(x∗))≤R(T)\sum_{t=0}^{T-1}\bigl(f(x_t)-f(x^*)\bigr)\le R(T). By Jensen’s inequality (Property 1), the averaged iterate xˉT:=1T∑t=0T−1xt∈V\bar x_T:=\frac1T\sum_{t=0}^{T-1}x_t\in V satisfies f(xˉT)−f(x∗)=f ⁣(1T∑t=0T−1xt)−f(x∗)≤1T∑t=0T−1(f(xt)−f(x∗))≤R(T)T. f(\bar x_T)-f(x^*)=f\!\left(\frac1T\sum_{t=0}^{T-1}x_t\right)-f(x^*) \le\frac1T\sum_{t=0}^{T-1}\bigl(f(x_t)-f(x^*)\bigr)\le\frac{R(T)}{T}. Whenever R(T)=o(T)R(T)=o(T), the optimality gap f(xˉT)−f(x∗)f(\bar x_T)-f(x^*) converges to 00 as T→∞T\to\infty.
  2. Lipschitzness of ff: To bound the second-order sum ∑t=0T−1∥∇f(xt)∥22\sum_{t=0}^{T-1}\|\nabla f(x_t)\|_2^2, assume ff is LL-Lipschitz in the sense that ∥∇f(xt)∥2≤L\|\nabla f(x_t)\|_2\le L for all tt. Since ϕ(xT)≥0\phi(x_T)\ge 0 and ϕ(x0)=12∥x0−x∗∥22\phi(x_0)=\frac12\|x_0-x^*\|_2^2, Theorem 2 yields ∑t=0T−1(f(xt)−f(x∗))≤∥x0−x∗∥222η+ηL2T2=∥x0−x∗∥2⋅LT \sum_{t=0}^{T-1}\bigl(f(x_t)-f(x^*)\bigr) \le\frac{\|x_0-x^*\|_2^2}{2\eta}+\frac{\eta L^2 T}{2} =\|x_0-x^*\|_2\cdot L\sqrt T upon choosing the optimal step size η=∥x0−x∗∥2LT\eta=\frac{\|x_0-x^*\|_2}{L\sqrt T}, which implies f(xˉT)−f(x∗)≤∥x0−x∗∥2⋅LTf(\bar x_T)-f(x^*)\le\frac{\|x_0-x^*\|_2\cdot L}{\sqrt T}.
  3. Constrained case: Projected Gradient Descent (PGD): When V⊊RnV\subsetneq\mathbb R^n is a constrained closed convex set, the raw gradient step yt+1:=xt−η∇f(xt)y_{t+1}:=x_t-\eta\nabla f(x_t) may step outside VV. To restore feasibility, we apply the Euclidean projection operator ΠV(y):=arg⁡min⁡x∈V∥x−y∥2\Pi_V(y):=\arg\min_{x\in V}\|x-y\|_2, obtaining Projected Gradient Descent: xt+1:=ΠV(xt−η∇f(xt)). x_{t+1}:=\Pi_V\bigl(x_t-\eta\nabla f(x_t)\bigr). Why does projection not hurt our potential analysis? Because projecting onto a convex set VV only moves the point closer to every point x∗∈Vx^*\in V:

Lemma 1 (Pythagorean contraction of Euclidean projection) Let V⊆RnV\subseteq\mathbb R^n be a closed convex set and let xt+1=ΠV(y)x_{t+1}=\Pi_V(y) be the Euclidean projection of y∈Rny\in\mathbb R^n onto VV. Then for every x∗∈Vx^*\in V, ⟨y−xt+1, x∗−xt+1⟩≤0and consequentlyϕ(ΠV(y))=12∥ΠV(y)−x∗∥22≤12∥y−x∗∥22=ϕ(y). \langle y-x_{t+1},\,x^*-x_{t+1}\rangle\le 0 \qquad\text{and consequently}\qquad \phi\bigl(\Pi_V(y)\bigr)=\frac12\bigl\|\Pi_V(y)-x^*\bigr\|_2^2\le\frac12\|y-x^*\|_2^2=\phi(y).

Proof. By definition of projection, xt+1=arg⁡min⁡x∈Vg(x)x_{t+1}=\arg\min_{x\in V}g(x) for the convex function g(x):=∥x−y∥22g(x):=\|x-y\|_2^2 with gradient ∇g(x)=2(x−y)\nabla g(x)=2(x-y). The first-order optimality condition on VV (Property 4) gives ⟨∇g(xt+1),x∗−xt+1⟩=2⟨xt+1−y,x∗−xt+1⟩≥0\langle\nabla g(x_{t+1}),x^*-x_{t+1}\rangle=2\langle x_{t+1}-y,x^*-x_{t+1}\rangle\ge 0, or ⟨y−xt+1,x∗−xt+1⟩≤0\langle y-x_{t+1},x^*-x_{t+1}\rangle\le 0. Expanding ∥y−x∗∥22\|y-x^*\|_2^2 around xt+1x_{t+1} then gives ∥y−x∗∥22=∥(y−xt+1)+(xt+1−x∗)∥22=∥y−xt+1∥22+∥xt+1−x∗∥22−2⟨y−xt+1, x∗−xt+1⟩≥∥xt+1−x∗∥22. \|y-x^*\|_2^2=\bigl\|(y-x_{t+1})+(x_{t+1}-x^*)\bigr\|_2^2=\|y-x_{t+1}\|_2^2+\|x_{t+1}-x^*\|_2^2-2\langle y-x_{t+1},\,x^*-x_{t+1}\rangle\ge\|x_{t+1}-x^*\|_2^2.

Applying Lemma 1 to y=xt−η∇f(xt)y=x_t-\eta\nabla f(x_t) gives ϕ(xt+1)−ϕ(xt)≤ϕ(xt−η∇f(xt))−ϕ(xt)\phi(x_{t+1})-\phi(x_t)\le\phi(x_t-\eta\nabla f(x_t))-\phi(x_t), so the convergence bound of Theorem 2 holds for constrained Projected Gradient Descent (with the single-step equality in Theorem 2 replaced by ≤\le)!

1.4 Online Gradient Descent and strong convexity

Now let us return to the online setting! When the decision set V⊆RnV\subseteq\mathbb R^n is a closed convex set and each round’s loss function ℓs:V→R\ell_s:V\to\mathbb R (s=0,1,…,T−1s=0,1,\ldots,T-1) is convex and differentiable, the online learning game (Definition 1) is called Online Convex Optimization (OCO).

Linearization reduction and the Online Projected Gradient Descent algorithm

Applying the first-order tangent-plane inequality ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩\ell_s(x_s)-\ell_s(x^*)\le\langle\nabla\ell_s(x_s),x_s-x^*\rangle at each round ss immediately reduces general convex losses to linear losses:

Lemma 2 (Linearization reduction) Let V⊆RnV\subseteq\mathbb R^n be a convex set and let ℓ0,…,ℓT−1:V→R\ell_0,\ldots,\ell_{T-1}:V\to\mathbb R be differentiable convex loss functions. Then for any sequence of iterates x0,…,xT−1∈Vx_0,\ldots,x_{T-1}\in V and every comparator x∗∈Vx^*\in V, R(T)=∑s=0T−1(ℓs(xs)−ℓs(x∗))≤∑s=0T−1⟨∇ℓs(xs), xs−x∗⟩. R(T)=\sum_{s=0}^{T-1}\bigl(\ell_s(x_s)-\ell_s(x^*)\bigr)\le\sum_{s=0}^{T-1}\bigl\langle\nabla\ell_s(x_s),\,x_s-x^*\bigr\rangle.

This motivates the online counterpart of Projected Gradient Descent, called Online Projected Gradient Descent (OGD): starting from an initial point x0∈Vx_0\in V, at each round s=0,1,…,T−1s=0,1,\ldots,T-1 the player plays xs∈Vx_s\in V, observes the gradient ∇ℓs(xs)\nabla\ell_s(x_s) of the revealed convex loss ℓs\ell_s, and updates with step size ηs>0\eta_s>0: xs+1:=ΠV(xs−ηs∇ℓs(xs)). x_{s+1}:=\Pi_V\bigl(x_s-\eta_s\nabla\ell_s(x_s)\bigr).

Look back at the single-step proof of Theorem 2 and Lemma 1 in §1.3: at step ss, the inequality ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩≤ϕ(xs)−ϕ(xs+1)ηs+ηs2∥∇ℓs(xs)∥22\ell_s(x_s)-\ell_s(x^*)\le\langle\nabla\ell_s(x_s),x_s-x^*\rangle\le\frac{\phi(x_s)-\phi(x_{s+1})}{\eta_s}+\frac{\eta_s}{2}\|\nabla\ell_s(x_s)\|_2^2 only used the convexity of the single function ℓs\ell_s at xsx_s—it never required the functions across different rounds to be identical! Consequently, Online Projected Gradient Descent satisfies the exact same regret bound against any adversarial sequence of convex losses ℓ0,…,ℓT−1\ell_0,\ldots,\ell_{T-1}.

Theorem 3 (Regret bound for Online Projected Gradient Descent) Let V⊆RnV\subseteq\mathbb R^n be a closed convex set and let ℓ0,…,ℓT−1:V→R\ell_0,\ldots,\ell_{T-1}:V\to\mathbb R be differentiable convex loss functions. For any initial point x0∈Vx_0\in V, any comparator x∗∈Vx^*\in V with ϕ(x)=12∥x−x∗∥22\phi(x)=\frac12\|x-x^*\|_2^2, and any step sizes ηs>0\eta_s>0, the Online Projected Gradient Descent iterates xs+1=ΠV(xs−ηs∇ℓs(xs))x_{s+1}=\Pi_V(x_s-\eta_s\nabla\ell_s(x_s)) satisfy, for every round s=0,…,T−1s=0,\ldots,T-1, ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs), xs−x∗⟩≤ϕ(xs)−ϕ(xs+1)ηs+ηs2∥∇ℓs(xs)∥22. \ell_s(x_s)-\ell_s(x^*)\le\bigl\langle\nabla\ell_s(x_s),\,x_s-x^*\bigr\rangle \le\frac{\phi(x_s)-\phi(x_{s+1})}{\eta_s}+\frac{\eta_s}{2}\bigl\|\nabla\ell_s(x_s)\bigr\|_2^2. In particular, if VV has Euclidean diameter diam⁡(V):=max⁡x,y∈V∥x−y∥2\operatorname{diam}(V):=\max_{x,y\in V}\|x-y\|_2, each ℓs\ell_s is LL-Lipschitz (∥∇ℓs(xs)∥2≤L\|\nabla\ell_s(x_s)\|_2\le L), and we take constant step size ηs=η=diam⁡(V)LT\eta_s=\eta=\frac{\operatorname{diam}(V)}{L\sqrt T}, then R(T)=∑s=0T−1(ℓs(xs)−ℓs(x∗))≤ϕ(x0)−ϕ(xT)η+η2∑s=0T−1∥∇ℓs(xs)∥22≤diam⁡(V)22η+ηL2T2=diam⁡(V)⋅LT. R(T)=\sum_{s=0}^{T-1}\bigl(\ell_s(x_s)-\ell_s(x^*)\bigr) \le\frac{\phi(x_0)-\phi(x_T)}{\eta}+\frac{\eta}{2}\sum_{s=0}^{T-1}\bigl\|\nabla\ell_s(x_s)\bigr\|_2^2 \le\frac{\operatorname{diam}(V)^2}{2\eta}+\frac{\eta L^2 T}{2} =\operatorname{diam}(V)\cdot L\sqrt T.

Continuous-time interpretation of Theorem 3 (see §2.1). Where does the second-order penalty ηs2∥∇ℓs(xs)∥22\frac{\eta_s}{2}\|\nabla\ell_s(x_s)\|_2^2 come from? In Reading Contents §2.1, we prove a general continuous-time discretization theorem for Online Mirror Descent (which includes Theorem 3 as the special case ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2): a continuous gradient flow dytdt=−gt\frac{\mathrm d y_t}{\mathrm d t}=-g_t pays only the potential drop ϕ(x0)−ϕ(xT)\phi(x_0)-\phi(x_T) with zero second-order loss, while ηs2∥∇ℓs(xs)∥22\frac{\eta_s}{2}\|\nabla\ell_s(x_s)\|_2^2 is the exact discretization lag incurred because the discrete player freezes at xsx_s during an interval of length ηs\eta_s.

Strongly convex losses: recovering O(log⁡T)O(\log T) regret on curved objectives

Notice an intriguing comparison: when we apply constant-step OGD (Theorem 3) to the online number guessing game ℓs(x)=(x−ys)2\ell_s(x)=(x-y_s)^2 on V=[0,1]V=[0,1], we get O(T)O(\sqrt T) regret—which is much worse than the 2HT=Θ(log⁡T)2H_T=\Theta(\log T) bound achieved by Follow the Leader (Theorem 1)! Why did Theorem 3 lose so much on (x−ys)2(x-y_s)^2?

Look at the first inequality in Theorem 3: we used the tangent-plane bound ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs),xs−x∗⟩\ell_s(x_s)-\ell_s(x^*)\le\langle\nabla\ell_s(x_s),x_s-x^*\rangle, which is tight only when ℓs\ell_s is linear. When ℓs\ell_s is strictly curved, the tangent hyperplane lies strictly below ℓs(x∗)\ell_s(x^*) by a quadratic margin!

Definition 2 (Euclidean strong convexity) A differentiable function f:V→Rf:V\to\mathbb R on a convex set V⊆RnV\subseteq\mathbb R^n is μ\mu-strongly convex (for a parameter μ>0\mu>0) if for all x,y∈Vx,y\in V, f(y)≥f(x)+⟨∇f(x), y−x⟩+μ2∥y−x∥22. f(y)\ge f(x)+\langle\nabla f(x),\,y-x\rangle+\frac{\mu}{2}\|y-x\|_2^2.

Retaining this −μ2∥xs−x∗∥22-\frac{\mu}{2}\|x_s-x^*\|_2^2 curvature bonus and decaying the step size as ηs=1μ(s+1)\eta_s=\frac{1}{\mu(s+1)} allows adjacent potential terms to telescope completely, recovering O(log⁡T)O(\log T) regret.

Theorem 4 (Logarithmic regret of OGD for strongly convex losses) Suppose each loss function ℓs:V→R\ell_s:V\to\mathbb R (s=0,1,…,T−1s=0,1,\ldots,T-1) is μ\mu-strongly convex and LL-Lipschitz (∥∇ℓs(xs)∥2≤L\|\nabla\ell_s(x_s)\|_2\le L). Then Online Projected Gradient Descent with decreasing step sizes ηs:=1μ(s+1)(s=0,1,…,T−1) \eta_s:=\frac{1}{\mu(s+1)}\qquad (s=0,1,\ldots,T-1) satisfies, for every x∗∈Vx^*\in V, R(T)=∑s=0T−1(ℓs(xs)−ℓs(x∗))≤L22μHT=O(log⁡T). R(T)=\sum_{s=0}^{T-1}\bigl(\ell_s(x_s)-\ell_s(x^*)\bigr)\le\frac{L^2}{2\mu}H_T=O(\log T).

Proof. By μ\mu-strong convexity of ℓs\ell_s and the single-round OGD inequality from Theorem 3, for each s=0,…,T−1s=0,\ldots,T-1, ℓs(xs)−ℓs(x∗)≤⟨∇ℓs(xs), xs−x∗⟩−μ2∥xs−x∗∥22≤ϕ(xs)−ϕ(xs+1)ηs+ηs2∥∇ℓs(xs)∥22−μ2∥xs−x∗∥22=(12ηs−μ2)∥xs−x∗∥22−12ηs∥xs+1−x∗∥22+ηs2∥∇ℓs(xs)∥22. \begin{aligned} \ell_s(x_s)-\ell_s(x^*) &\le\bigl\langle\nabla\ell_s(x_s),\,x_s-x^*\bigr\rangle-\frac{\mu}{2}\|x_s-x^*\|_2^2\\ &\le\frac{\phi(x_s)-\phi(x_{s+1})}{\eta_s}+\frac{\eta_s}{2}\bigl\|\nabla\ell_s(x_s)\bigr\|_2^2-\frac{\mu}{2}\|x_s-x^*\|_2^2\\ &=\left(\frac{1}{2\eta_s}-\frac{\mu}{2}\right)\|x_s-x^*\|_2^2-\frac{1}{2\eta_s}\|x_{s+1}-x^*\|_2^2+\frac{\eta_s}{2}\bigl\|\nabla\ell_s(x_s)\bigr\|_2^2. \end{aligned} Summing this inequality over s=0,1,…,T−1s=0,1,\ldots,T-1 and grouping the coefficients of ∥xs−x∗∥22\|x_s-x^*\|_2^2 gives ∑s=0T−1(ℓs(xs)−ℓs(x∗))≤(12η0−μ2)∥x0−x∗∥22−12ηT−1∥xT−x∗∥22+∑s=1T−1(12ηs−12ηs−1−μ2)∥xs−x∗∥22+∑s=0T−1ηs2∥∇ℓs(xs)∥22. \begin{aligned} \sum_{s=0}^{T-1}\bigl(\ell_s(x_s)-\ell_s(x^*)\bigr) &\le\left(\frac{1}{2\eta_0}-\frac{\mu}{2}\right)\|x_0-x^*\|_2^2-\frac{1}{2\eta_{T-1}}\|x_T-x^*\|_2^2\\ &\quad+\sum_{s=1}^{T-1}\left(\frac{1}{2\eta_s}-\frac{1}{2\eta_{s-1}}-\frac{\mu}{2}\right)\|x_s-x^*\|_2^2+\sum_{s=0}^{T-1}\frac{\eta_s}{2}\bigl\|\nabla\ell_s(x_s)\bigr\|_2^2. \end{aligned} Substituting 1ηs=μ(s+1)\frac{1}{\eta_s}=\mu(s+1) makes 12η0−μ2=0\frac{1}{2\eta_0}-\frac{\mu}{2}=0 and 12ηs−12ηs−1−μ2=0\frac{1}{2\eta_s}-\frac{1}{2\eta_{s-1}}-\frac{\mu}{2}=0 for every s≥1s\ge 1! Dropping the nonpositive terminal term −12ηT−1∥xT−x∗∥22-\frac{1}{2\eta_{T-1}}\|x_T-x^*\|_2^2 and using ∥∇ℓs(xs)∥2≤L\|\nabla\ell_s(x_s)\|_2\le L yields ∑s=0T−1(ℓs(xs)−ℓs(x∗))≤∑s=0T−1L22μ(s+1)=L22μHT. \sum_{s=0}^{T-1}\bigl(\ell_s(x_s)-\ell_s(x^*)\bigr)\le\sum_{s=0}^{T-1}\frac{L^2}{2\mu(s+1)}=\frac{L^2}{2\mu}H_T.

Matching FTL on the number guessing game. On ℓs(x)=(x−ys)2\ell_s(x)=(x-y_s)^2 over V=[0,1]V=[0,1] with ys∈[0,1]y_s\in[0,1], we have ℓs′′(x)=2\ell_s''(x)=2 (μ=2\mu=2), ∣ℓs′(x)∣=2∣x−ys∣≤2|\ell_s'(x)|=2|x-y_s|\le 2 (L=2L=2), and step size ηs=12(s+1)\eta_s=\frac{1}{2(s+1)}, so xs+1=xs−1s+1(xs−ys)=ss+1xs+1s+1ys=1s+1∑i=0syix_{s+1}=x_s-\frac{1}{s+1}(x_s-y_s)=\frac{s}{s+1}x_s+\frac{1}{s+1}y_s=\frac{1}{s+1}\sum_{i=0}^s y_i is literally the running average of y0,…,ysy_0,\ldots,y_s, and Theorem 4 gives R(T)≤222⋅2HT=HTR(T)\le\frac{2^2}{2\cdot 2}H_T=H_T, matching Follow the Leader!

1.5 Reducing discrete problems to OGD: experts and online shortest paths

At first glance, Online Projected Gradient Descent (Theorem 3) applies only when the action space V⊆RnV\subseteq\mathbb R^n is a continuous convex body. Remarkably, many discrete combinatorial online problems can be transformed into convex optimization problems by randomization—lifting a discrete set of choices into a convex space of probability distributions. Let us examine two canonical reductions.

1. Learning with expert advice lifted to the probability simplex

In the learning with expert advice problem, the discrete action set is V=[n]={1,…,n}V=[n]=\{1,\ldots,n\}: we treat each i∈[n]i\in[n] as an expert, and in each round t=1,…,Tt=1,\ldots,T, the player picks one expert it∈[n]i_t\in[n] to follow while the adversary reveals a loss vector ℓt∈[0,1]n\ell_t\in[0,1]^n. Clearly V=[n]V=[n] is finite and not convex, so gradient descent cannot be applied directly on [n][n]. Worse still, no deterministic strategy can achieve sublinear regret!

Contrast: the Halving algorithm when a perfect expert exists. If each expert announces a binary prediction y^t(i)∈{0,1}\hat y_t(i)\in\{0,1\} and at least one expert i∗∈[n]i^*\in[n] makes zero mistakes (ℓt(i∗)=0 ∀t\ell_t(i^*)=0\ \forall t), predicting the majority vote among surviving zero-mistake experts eliminates at least half of the survivors on every mistake, making at most ⌊log⁡2n⌋\lfloor\log_2 n\rfloor mistakes in total. Once every expert can make mistakes, however, eliminating erring experts empties the pool, and Example 2 forces randomization.

Example 2 (Failure of every deterministic strategy) Suppose n=2n=2 (or general n≥2n\ge 2) and the player chooses an expert it∈[n]i_t\in[n] deterministically in each round tt. Because the adversary knows the player’s deterministic rule, it can set ℓt(j)={1,if j=it,0,otherwise. \ell_t(j)=\begin{cases} 1, & \text{if }j=i_t,\\[2pt] 0, & \text{otherwise}. \end{cases} After TT rounds, the player suffers cumulative loss TT, whereas the sum of cumulative losses across all nn experts is ∑j=1n∑t=1Tℓt(j)=T\sum_{j=1}^n\sum_{t=1}^T\ell_t(j)=T, so the best expert in hindsight suffers at most T/nT/n (at most T/2T/2 when n=2n=2). Thus every deterministic strategy suffers linear regret Reg⁡T≥(1−1/n)T=Ω(T)\operatorname{Reg}_T\ge(1-1/n)T=\Omega(T).

Therefore, a randomized strategy is mandatory: in each round tt, instead of committing to a single expert, the player chooses a probability distribution xtx_t in the (n−1)(n-1)-dimensional probability simplex Δn:={x=(x1,…,xn)∈[0,1]n:∑i=1nxi=1}, \Delta_n:=\Bigl\{x=(x_1,\ldots,x_n)\in[0,1]^n:\sum_{i=1}^n x_i=1\Bigr\}, draws an action At∼xtA_t\sim x_t, and pays expected loss EAt∼xt[ℓt(At)]=∑i=1nxt(i)ℓt(i)=⟨ℓt,xt⟩\mathbb E_{A_t\sim x_t}[\ell_t(A_t)]=\sum_{i=1}^n x_t(i)\ell_t(i)=\langle\ell_t,x_t\rangle.

Definition 3 (Prediction with expert advice on Δn\Delta_n) Fix n≥2n\ge 2 experts and horizon T≥1T\ge 1. For each round t=1,…,Tt=1,\ldots,T:

  1. The player selects a probability distribution xt∈Δnx_t\in\Delta_n;
  2. The adversary chooses a loss vector ℓt∈[0,1]n\ell_t\in[0,1]^n;
  3. The player samples At∼xtA_t\sim x_t, incurs loss ℓt(At)\ell_t(A_t), and observes the full vector ℓt\ell_t.

The expected regret with respect to a fixed expert j∈[n]j\in[n] and the worst-case expected regret are E ⁣[∑t=1T(ℓt(At)−ℓt(j))]=E ⁣[∑t=1T⟨ℓt, xt−ej⟩],Reg⁡T:=max⁡j∈[n]E ⁣[∑t=1T⟨ℓt, xt−ej⟩], \mathbb E\!\left[\sum_{t=1}^T\bigl(\ell_t(A_t)-\ell_t(j)\bigr)\right]=\mathbb E\!\left[\sum_{t=1}^T\langle\ell_t,\,x_t-\mathbf e_j\rangle\right], \qquad \operatorname{Reg}_T:=\max_{j\in[n]}\mathbb E\!\left[\sum_{t=1}^T\langle\ell_t,\,x_t-\mathbf e_j\rangle\right], where ej∈Δn\mathbf e_j\in\Delta_n is the jj-th standard basis vector.

Because Δn⊆Rn\Delta_n\subseteq\mathbb R^n is compact and convex and the expected loss x↦⟨ℓt,x⟩x\mapsto\langle\ell_t,x\rangle is linear in xx with gradient ∇⟨ℓt,x⟩=ℓt\nabla\langle\ell_t,x\rangle=\ell_t, learning with expert advice is now an instance of Online Convex Optimization on Δn\Delta_n! Running Online Projected Gradient Descent from the uniform distribution x1=(1n,…,1n)x_1=(\frac1n,\ldots,\frac1n) with update xt+1=ΠΔn(xt−ηℓt) x_{t+1}=\Pi_{\Delta_n}(x_t-\eta\ell_t) immediately yields a sublinear regret bound via Theorem 3: since ∥x1−ej∥22=(1−1n)2+(n−1)1n2=1−1n≤1\|x_1-\mathbf e_j\|_2^2=(1-\frac1n)^2+(n-1)\frac{1}{n^2}=1-\frac1n\le 1 (and diam⁡(Δn)=2\operatorname{diam}(\Delta_n)=\sqrt 2) while ∥ℓt∥2=∑i=1nℓt(i)2≤n\|\ell_t\|_2=\sqrt{\sum_{i=1}^n\ell_t(i)^2}\le\sqrt n for ℓt∈[0,1]n\ell_t\in[0,1]^n, Theorem 3 gives E ⁣[∑t=1T(ℓt(At)−ℓt(j))]≤12η+ηnT2=nT(choosing η=1nT). \mathbb E\!\left[\sum_{t=1}^T\bigl(\ell_t(A_t)-\ell_t(j)\bigr)\right] \le\frac{1}{2\eta}+\frac{\eta nT}{2}=\sqrt{nT} \qquad\Bigl(\text{choosing }\eta=\frac{1}{\sqrt{nT}}\Bigr).

2. Online shortest paths via probability flows

What if the number of discrete experts NN is itself exponentially large in the natural problem size?

From edge flow pp to path distribution. A probability flow p:E→[0,1]p:E\to[0,1] on a DAG G=(V,E)G=(V,E) satisfies ∑y:(x,y)∈Ep(x,y)=∑y:(y,x)∈Ep(y,x)(∀x∈V∖{u,v}) \sum_{y:(x,y)\in E}p(x,y)=\sum_{y:(y,x)\in E}p(y,x)\quad (\forall x\in V\setminus\{u,v\}) and ∑y:(u,y)∈Ep(u,y)=∑y:(y,v)∈Ep(y,v)=1\sum_{y:(u,y)\in E}p(u,y)=\sum_{y:(y,v)\in E}p(y,v)=1. At each vertex xx with positive outflow, pp induces a transition distribution over out-neighbors: p^(x,y):=p(x,y)∑z:(x,z)∈Ep(x,z). \hat p(x,y):=\frac{p(x,y)}{\sum_{z:(x,z)\in E}p(x,z)}. Sampling a random walk from uu to vv according to p^\hat p generates a path P∈Pu,vP\in\mathcal P_{u,v} with Pr⁡[P]=∏e∈Pp^(e)\Pr[P]=\prod_{e\in P}\hat p(e) whose marginal probability of traversing any edge e∈Ee\in E is exactly Pr⁡[e∈P]=p(e)\Pr[e\in P]=p(e)!

Example 3 (Online shortest paths via the probability flow polytope) Let G=(V,E)G=(V,E) be a directed acyclic graph and fix two vertices u,v∈Vu,v\in V. In each round t=1,…,Tt=1,\ldots,T:

  1. The player picks a directed path Pt∈Pu,vP_t\in\mathcal P_{u,v}, where Pu,v\mathcal P_{u,v} is the set of all simple paths from uu to vv;
  2. The adversary reveals edge weights wt:E→[0,1]w_t:E\to[0,1];
  3. The player pays the path length ℓt(Pt):=∑e∈Ptwt(e)\ell_t(P_t):=\sum_{e\in P_t}w_t(e).

We want to minimize regret against the shortest fixed path in hindsight, P∗∈arg⁡min⁡P∈Pu,v∑t=1Tℓt(P)P^*\in\arg\min_{P\in\mathcal P_{u,v}}\sum_{t=1}^T\ell_t(P).

If we treat each path P∈Pu,vP\in\mathcal P_{u,v} as an independent expert in Δ∣Pu,v∣\Delta_{|\mathcal P_{u,v}|}, the vector dimension ∣Pu,v∣|\mathcal P_{u,v}| can be 2Ω(∣V∣)2^{\Omega(|V|)}, making it computationally infeasible to even write down the distribution vector! However, because the loss of a path ∑e∈Pwt(e)\sum_{e\in P}w_t(e) is additive over edges, the expected loss of any distribution over Pu,v\mathcal P_{u,v} depends only on the marginal probability p(e):=Pr⁡[e∈P]∈[0,1]p(e):=\Pr[e\in P]\in[0,1] that each edge e∈Ee\in E is traversed: EP[ℓt(P)]=∑e∈EPr⁡[e∈P] wt(e)=∑e∈Ep(e)wt(e)=⟨wt, p⟩. \mathbb E_{P}\bigl[\ell_t(P)\bigr]=\sum_{e\in E}\Pr[e\in P]\,w_t(e)=\sum_{e\in E}p(e)w_t(e)=\langle w_t,\,p\rangle. As shown in the margin note, the set of all valid edge-marginal vectors p∈[0,1]∣E∣p\in[0,1]^{|E|} is precisely the unit u–vu\text{–}v probability flow polytope P⊆R∣E∣\mathcal P\subseteq\mathbb R^{|E|}, defined by ∣V∣|V| linear flow-conservation equations and ∣E∣|E| box inequalities 0≤p(e)≤10\le p(e)\le 1. We can therefore run OGD directly on the ∣E∣|E|-dimensional convex polytope P\mathcal P and sample PtP_t in O(∣V∣+∣E∣)O(|V|+|E|) time per round!

1.6 Adversarial multi-armed bandits via Online Stochastic Gradient Descent

We open our second meeting by returning to the multi-armed bandit problem from Chapter 1! Compare the stochastic bandit model of Chapter 1 with the full-information expert model of §1.5:

When we drop Chapter 1’s i.i.d. assumption and allow the adversary to choose the loss vectors ℓ1,…,ℓT∈[0,1]n\ell_1,\ldots,\ell_T\in[0,1]^n arbitrarily while restricting observation to the single coordinate ℓt(At)\ell_t(A_t), we obtain the adversarial multi-armed bandit problem.

Definition 4 (Adversarial multi-armed bandit protocol) Fix n≥2n\ge 2 arms and a horizon T≥1T\ge 1. For each round t=1,…,Tt=1,\ldots,T:

  1. Based on the past history Ft−1=(A1,ℓ1(A1),…,At−1,ℓt−1(At−1))\mathcal F_{t-1}=\bigl(A_1,\ell_1(A_1),\ldots,A_{t-1},\ell_{t-1}(A_{t-1})\bigr), the player chooses a distribution xt∈Δnx_t\in\Delta_n and samples an arm At∼xtA_t\sim x_t;
  2. The adversary chooses ℓt∈[0,1]n\ell_t\in[0,1]^n (measurable with respect to Ft−1\mathcal F_{t-1}), and the player pays and observes only the scalar loss ℓt(At)∈[0,1]\ell_t(A_t)\in[0,1].

The player’s expected bandit regret (often called pseudo-regret when maximizing over fixed arms j∈[n]j\in[n] outside the expectation, which coincides with E[R(T)]\mathbb E[R(T)] against an oblivious adversary) is Reg⁡Tbandit:=max⁡j∈[n]E ⁣[∑t=1T(ℓt(At)−ℓt(j))]=max⁡j∈[n]∑t=1TE[⟨ℓt, xt−ej⟩]. \operatorname{Reg}_T^{\mathrm{bandit}}:=\max_{j\in[n]}\mathbb E\!\left[\sum_{t=1}^T\bigl(\ell_t(A_t)-\ell_t(j)\bigr)\right]=\max_{j\in[n]}\sum_{t=1}^T\mathbb E\bigl[\langle\ell_t,\,x_t-\mathbf e_j\rangle\bigr].

Guessing the unseen loss vector via importance weighting

How can we apply gradient-based online learning when n−1n-1 coordinates of the gradient ℓt\ell_t are hidden? The key idea is to construct a random vector ℓ^t∈R≥0n\hat\ell_t\in\mathbb R_{\ge0}^n from the single observation (At,ℓt(At))\bigl(A_t,\ell_t(A_t)\bigr) that is an unbiased estimator of ℓt\ell_t (E[ℓ^t∣Ft−1]=ℓt\mathbb E[\hat\ell_t\mid\mathcal F_{t-1}]=\ell_t), and feed ℓ^t\hat\ell_t into our online optimization algorithm in place of ℓt\ell_t!

ℓ̂ = 2ℓ(1) xₜ(1) = ½ ℓ̂ = 6ℓ(2) xₜ(2) = ⅙

Dividing by xt(i)x_t(i) keeps the expected rectangle area xt(i)⋅(ℓt(i)/xt(i))=ℓt(i)x_t(i)\cdot(\ell_t(i)/x_t(i))=\ell_t(i) unbiased, but a tiny sampling probability xt(i)x_t(i) creates a spike of height ℓt(i)/xt(i)\ell_t(i)/x_t(i) whose second moment scales as ℓt(i)2/xt(i)\ell_t(i)^2/x_t(i).

Specifically, for every arm i∈[n]i\in[n] with xt(i)>0x_t(i)>0, define the importance-weighted (Horvitz–Thompson) estimator: ℓ^t(i):=1{At=i}xt(i)ℓt(i)={ℓt(At)xt(At),if i=At,0,if i≠At. \hat\ell_t(i):=\frac{\mathbf 1_{\{A_t=i\}}}{x_t(i)}\ell_t(i) =\begin{cases} \dfrac{\ell_t(A_t)}{x_t(A_t)}, & \text{if }i=A_t,\\[6pt] 0, & \text{if }i\ne A_t. \end{cases} Notice that ℓ^t\hat\ell_t is completely computable from the bandit observation (At,ℓt(At))\bigl(A_t,\ell_t(A_t)\bigr) and the known distribution xtx_t. Let us record its conditional first and second moments given Ft−1\mathcal F_{t-1}.

Lemma 3 (Conditional mean and second moments of importance-weighted losses) Let Ft−1\mathcal F_{t-1} denote the σ\sigma-algebra of the first t−1t-1 rounds, so that xt∈int⁡(Δn)x_t\in\operatorname{int}(\Delta_n) and ℓt∈[0,1]n\ell_t\in[0,1]^n are Ft−1\mathcal F_{t-1}-measurable. Under the random draw At∼xtA_t\sim x_t:

  1. Conditional unbiasedness: For every i∈[n]i\in[n], E[ℓ^t(i)∣Ft−1]=ℓt(i)\mathbb E\bigl[\hat\ell_t(i)\mid\mathcal F_{t-1}\bigr]=\ell_t(i), and therefore for every comparator x∗∈Δnx^*\in\Delta_n, E[⟨ℓt, xt−x∗⟩]=E[⟨E[ℓ^t∣Ft−1], xt−x∗⟩]=E[E[⟨ℓ^t, xt−x∗⟩  ∣  Ft−1]]=E[⟨ℓ^t, xt−x∗⟩]. \mathbb E\bigl[\langle\ell_t,\,x_t-x^*\rangle\bigr] =\mathbb E\Bigl[\bigl\langle\mathbb E[\hat\ell_t\mid\mathcal F_{t-1}],\,x_t-x^*\bigr\rangle\Bigr] =\mathbb E\Bigl[\mathbb E\bigl[\langle\hat\ell_t,\,x_t-x^*\rangle\;\big|\;\mathcal F_{t-1}\bigr]\Bigr] =\mathbb E\bigl[\langle\hat\ell_t,\,x_t-x^*\rangle\bigr].
  2. Euclidean second moment versus xtx_t-weighted second moment: For every i∈[n]i\in[n], E[ℓ^t(i)2∣Ft−1]=ℓt(i)2xt(i)\mathbb E\bigl[\hat\ell_t(i)^2\mid\mathcal F_{t-1}\bigr]=\frac{\ell_t(i)^2}{x_t(i)}, so E[∥ℓ^t∥22  ∣  Ft−1]=∑i=1nxt(i)(ℓt(i)xt(i))2=∑i=1nℓt(i)2xt(i)≤∑i=1n1xt(i), \mathbb E\Bigl[\|\hat\ell_t\|_2^2\;\Big|\;\mathcal F_{t-1}\Bigr] =\sum_{i=1}^n x_t(i)\left(\frac{\ell_t(i)}{x_t(i)}\right)^2 =\sum_{i=1}^n\frac{\ell_t(i)^2}{x_t(i)}\le\sum_{i=1}^n\frac{1}{x_t(i)}, whereas the xtx_t-weighted sum of squares satisfies E[∑i=1nxt(i)ℓ^t(i)2  ∣  Ft−1]=∑i=1nxt(i)⋅ℓt(i)2xt(i)=∑i=1nℓt(i)2≤n. \mathbb E\Bigl[\sum_{i=1}^n x_t(i)\hat\ell_t(i)^2\;\Big|\;\mathcal F_{t-1}\Bigr] =\sum_{i=1}^n x_t(i)\cdot\frac{\ell_t(i)^2}{x_t(i)} =\sum_{i=1}^n\ell_t(i)^2\le n.
Proof: tower property and indicator moments

Conditional on Ft−1\mathcal F_{t-1}, the indicator 1{At=i}\mathbf 1_{\{A_t=i\}} takes value 11 with probability xt(i)x_t(i) and 00 with probability 1−xt(i)1-x_t(i). Hence E[ℓ^t(i)∣Ft−1]=xt(i)⋅ℓt(i)xt(i)+(1−xt(i))⋅0=ℓt(i) \mathbb E\bigl[\hat\ell_t(i)\mid\mathcal F_{t-1}\bigr]=x_t(i)\cdot\frac{\ell_t(i)}{x_t(i)}+\bigl(1-x_t(i)\bigr)\cdot 0=\ell_t(i) and E[ℓ^t(i)2∣Ft−1]=xt(i)⋅(ℓt(i)xt(i))2+(1−xt(i))⋅0=ℓt(i)2xt(i). \mathbb E\bigl[\hat\ell_t(i)^2\mid\mathcal F_{t-1}\bigr]=x_t(i)\cdot\left(\frac{\ell_t(i)}{x_t(i)}\right)^2+\bigl(1-x_t(i)\bigr)\cdot 0=\frac{\ell_t(i)^2}{x_t(i)}. Because xt−x∗x_t-x^* is Ft−1\mathcal F_{t-1}-measurable, linearity of conditional expectation and the law of total expectation give E[⟨ℓt,xt−x∗⟩]=E[⟨ℓ^t,xt−x∗⟩]\mathbb E[\langle\ell_t,x_t-x^*\rangle]=\mathbb E[\langle\hat\ell_t,x_t-x^*\rangle]. Summing E[ℓ^t(i)2∣Ft−1]\mathbb E[\hat\ell_t(i)^2\mid\mathcal F_{t-1}] with and without the factor xt(i)x_t(i) yields the two second-moment formulas.

Online Stochastic Gradient Descent (OSGD) and the O((nT)2/3)O((nT)^{2/3}) barrier

By Part 1 of Lemma 3, we can analyze the gradient descent update xt+1=ΠΔn(xt−ηℓ^t)x_{t+1}=\Pi_{\Delta_n}(x_t-\eta\hat\ell_t)—called Online Stochastic Gradient Descent (OSGD)—as if the random estimator ℓ^t\hat\ell_t were the true loss vector! Taking expectations in the Euclidean OGD bound (Theorem 3) with x1=(1n,…,1n)x_1=(\frac1n,\ldots,\frac1n) gives, for every arm j∈[n]j\in[n], E ⁣[∑t=1T⟨ℓt, xt−ej⟩]=E ⁣[∑t=1T⟨ℓ^t, xt−ej⟩]≤12η+η2∑t=1TE[∥ℓ^t∥22]≤12η+η2∑t=1TE ⁣[∑i=1n1xt(i)]. \mathbb E\!\left[\sum_{t=1}^T\langle\ell_t,\,x_t-\mathbf e_j\rangle\right] =\mathbb E\!\left[\sum_{t=1}^T\langle\hat\ell_t,\,x_t-\mathbf e_j\rangle\right] \le\frac{1}{2\eta}+\frac{\eta}{2}\sum_{t=1}^T\mathbb E\bigl[\|\hat\ell_t\|_2^2\bigr] \le\frac{1}{2\eta}+\frac{\eta}{2}\sum_{t=1}^T\mathbb E\!\left[\sum_{i=1}^n\frac{1}{x_t(i)}\right].

Caution (Variance blow-up of Euclidean OSGD at the simplex boundary).

Look at the second term ∑i=1n1xt(i)\sum_{i=1}^n\frac{1}{x_t(i)}: under Euclidean projection ΠΔn\Pi_{\Delta_n}, we have no lower bound on xt(i)x_t(i)—if an arm ii looks bad early on, ΠΔn\Pi_{\Delta_n} can push xt(i)x_t(i) arbitrarily close to 00 (or to 00 itself), causing the variance E[∥ℓ^t∥22∣Ft−1]=∑i=1nℓt(i)2xt(i)\mathbb E[\|\hat\ell_t\|_2^2\mid\mathcal F_{t-1}]=\sum_{i=1}^n\frac{\ell_t(i)^2}{x_t(i)} to explode!

To prevent 1/x~t(i)1/\tilde x_t(i) from blowing up under Euclidean OSGD, we are forced to mix xtx_t with the uniform distribution u:=(1n,…,1n)\mathbf u:=(\frac1n,\ldots,\frac1n) using an explicit exploration parameter α∈(0,1]\alpha\in(0,1]:

  1. Initialize x1=u=(1n,…,1n)x_1=\mathbf u=(\frac1n,\ldots,\frac1n).
  2. In each round t=1,…,Tt=1,\ldots,T:
    • Mix x~t:=(1−α)xt+αu\tilde x_t:=(1-\alpha)x_t+\alpha\mathbf u, and sample At∼x~tA_t\sim\tilde x_t;
    • Observe ℓt(At)\ell_t(A_t) and compute ℓ^t(i):=1{At=i}x~t(i)ℓt(i)\hat\ell_t(i):=\frac{\mathbf 1_{\{A_t=i\}}}{\tilde x_t(i)}\ell_t(i) for i∈[n]i\in[n];
    • Update xt+1:=ΠΔn(xt−ηℓ^t)x_{t+1}:=\Pi_{\Delta_n}(x_t-\eta\hat\ell_t).

Theorem 5 (Regret bound for Euclidean OSGD with uniform exploration) In the adversarial multi-armed bandit problem (Definition 4), running Euclidean OSGD with exploration parameter α=(n24T)1/3\alpha=\bigl(\frac{n^2}{4T}\bigr)^{1/3} and step size η=41/32(nT)−2/3\eta=\frac{4^{1/3}}{2}(nT)^{-2/3} (for T≥n2/4T\ge n^2/4) satisfies Reg⁡Tbandit≤12η+ηn2T2α+αT=3⋅4−1/3 (nT)2/3=O((nT)2/3). \operatorname{Reg}_T^{\mathrm{bandit}}\le\frac{1}{2\eta}+\frac{\eta n^2 T}{2\alpha}+\alpha T=3\cdot 4^{-1/3}\,(nT)^{2/3}=O\bigl((nT)^{2/3}\bigr).

Proof. Fix any comparator arm j∈[n]j\in[n]. Since the player pulls At∼x~t=(1−α)xt+αuA_t\sim\tilde x_t=(1-\alpha)x_t+\alpha\mathbf u and E[ℓ^t∣Ft−1]=ℓt\mathbb E[\hat\ell_t\mid\mathcal F_{t-1}]=\ell_t, the expected regret decomposes into the OGD regret of xtx_t plus the exploration bias of αu\alpha\mathbf u: E ⁣[∑t=1T⟨ℓt, x~t−ej⟩]=E ⁣[∑t=1T⟨ℓ^t, (1−α)xt+αu−ej⟩]=E ⁣[∑t=1T⟨ℓ^t, xt−ej⟩]+α E ⁣[∑t=1T⟨ℓ^t, u−xt⟩]≤12η+η2∑t=1TE ⁣[∑i=1nx~t(i)(ℓt(i)x~t(i))2]+α E ⁣[∑t=1T⟨ℓt,u⟩]. \begin{aligned} \mathbb E\!\left[\sum_{t=1}^T\langle\ell_t,\,\tilde x_t-\mathbf e_j\rangle\right] &=\mathbb E\!\left[\sum_{t=1}^T\bigl\langle\hat\ell_t,\,(1-\alpha)x_t+\alpha\mathbf u-\mathbf e_j\bigr\rangle\right]\\ &=\mathbb E\!\left[\sum_{t=1}^T\langle\hat\ell_t,\,x_t-\mathbf e_j\rangle\right]+\alpha\,\mathbb E\!\left[\sum_{t=1}^T\langle\hat\ell_t,\,\mathbf u-x_t\rangle\right]\\ &\le\frac{1}{2\eta}+\frac{\eta}{2}\sum_{t=1}^T\mathbb E\!\left[\sum_{i=1}^n\tilde x_t(i)\left(\frac{\ell_t(i)}{\tilde x_t(i)}\right)^2\right]+\alpha\,\mathbb E\!\left[\sum_{t=1}^T\langle\ell_t,\mathbf u\rangle\right]. \end{aligned} Because x~t(i)=(1−α)xt(i)+αn≥αn\tilde x_t(i)=(1-\alpha)x_t(i)+\frac{\alpha}{n}\ge\frac{\alpha}{n} and ℓt(i)∈[0,1]\ell_t(i)\in[0,1], each inner sum is bounded by ∑i=1n1x~t(i)≤n⋅nα=n2α\sum_{i=1}^n\frac{1}{\tilde x_t(i)}\le n\cdot\frac{n}{\alpha}=\frac{n^2}{\alpha}, while ⟨ℓt,u⟩≤1\langle\ell_t,\mathbf u\rangle\le 1. Therefore, Reg⁡Tbandit≤12η+ηn2T2α+αT. \operatorname{Reg}_T^{\mathrm{bandit}}\le\frac{1}{2\eta}+\frac{\eta n^2 T}{2\alpha}+\alpha T. Balancing the three terms by setting 12η=ηn2T2α=αT\frac{1}{2\eta}=\frac{\eta n^2 T}{2\alpha}=\alpha T gives η=41/32(nT)−2/3\eta=\frac{4^{1/3}}{2}(nT)^{-2/3}, α=(n24T)1/3\alpha=\bigl(\frac{n^2}{4T}\bigr)^{1/3}, and total regret 3⋅4−1/3(nT)2/33\cdot 4^{-1/3}(nT)^{2/3}.


1.7 From quadratic regularizer to Bregman divergence and the simplex mismatch

Stepping back from §§1.5–1.6, we now face two sharp puzzles left behind by Euclidean Online Gradient Descent on the probability simplex Δn\Delta_n:

  1. Full-information experts (§1.5): Why did Euclidean OGD pay O(nT)O(\sqrt{nT})—growing polynomially with the number of experts nn—when we will soon see that O(Tln⁡n)O(\sqrt{T\ln n}) is achievable?
  2. Adversarial bandits (§1.6): Why did Euclidean OSGD degrade from T1/2T^{1/2} to O((nT)2/3)O((nT)^{2/3}), forced to inject α\alpha-uniform exploration because E[∥ℓ^t∥22∣Ft−1]=∑i=1nℓt(i)2xt(i)\mathbb E[\|\hat\ell_t\|_2^2\mid\mathcal F_{t-1}]=\sum_{i=1}^n\frac{\ell_t(i)^2}{x_t(i)} blew up near the boundary?

To find the common culprit behind both inefficiencies, let us rewrite the projected gradient descent step xt+1=ΠV(xt−η∇f(xt))x_{t+1}=\Pi_V(x_t-\eta\nabla f(x_t)) as a local optimization problem.

Proposition 1 (Proximal quadratic regularizer view of Projected Gradient Descent) For any closed convex set V⊆RnV\subseteq\mathbb R^n, any point xt∈Vx_t\in V, and any step size η>0\eta>0, the projected gradient descent update xt+1=ΠV(xt−η∇f(xt))x_{t+1}=\Pi_V\bigl(x_t-\eta\nabla f(x_t)\bigr) is equivalent to xt+1=arg⁡min⁡x∈V{⟨∇f(xt), x−xt⟩+12η∥x−xt∥22}. x_{t+1}=\arg\min_{x\in V}\left\{\bigl\langle\nabla f(x_t),\,x-x_t\bigr\rangle+\frac{1}{2\eta}\|x-x_t\|_2^2\right\}.

Proof. Expanding the squared Euclidean distance in the definition of ΠV\Pi_V and dropping terms that do not depend on the optimization variable x∈Vx\in V, we obtain ΠV(xt−η∇f(xt))=arg⁡min⁡x∈V∥xt−η∇f(xt)−x∥22=arg⁡min⁡x∈V{⟨x,x⟩−2⟨x, xt−η∇f(xt)⟩}=arg⁡min⁡x∈V{2η⟨∇f(xt), x−xt⟩+∥x−xt∥22}, \begin{aligned} \Pi_V\bigl(x_t-\eta\nabla f(x_t)\bigr) &=\arg\min_{x\in V}\bigl\|x_t-\eta\nabla f(x_t)-x\bigr\|_2^2\\ &=\arg\min_{x\in V}\Bigl\{\langle x,x\rangle-2\bigl\langle x,\,x_t-\eta\nabla f(x_t)\bigr\rangle\Bigr\}\\ &=\arg\min_{x\in V}\Bigl\{2\eta\bigl\langle\nabla f(x_t),\,x-x_t\bigr\rangle+\|x-x_t\|_2^2\Bigr\}, \end{aligned} and dividing the objective by 2η>02\eta>0 gives the claimed formula.

Look at the intuition revealed by Proposition 1: at step tt, the only information we use about ff is its gradient ∇f(xt)\nabla f(x_t), which defines the local linear approximation L(x):=⟨∇f(xt),x−xt⟩L(x):=\langle\nabla f(x_t),x-x_t\rangle. Because minimizing L(x)L(x) alone is unbounded (min⁡L(x)=−∞\min L(x)=-\infty when V=RnV=\mathbb R^n, or jumps to a corner when V=ΔnV=\Delta_n) and L(x)≈f(x)L(x)\approx f(x) holds only for xx near xtx_t, we add a proximity regularizer 12η∥x−xt∥22\frac{1}{2\eta}\|x-x_t\|_2^2 to keep xt+1x_{t+1} close to xtx_t.

Primal V (‖p‖₁ ≤ 1) On Δₙ: D₁ = 2, D₂ = √2 Dual V* (‖ℓ‖∞ ≤ 1) ‖𝟏‖∞ = 1, ‖𝟏‖₂ = √n

In Rn\mathbb R^n, the unit ℓ∞\ell_\infty cube [−1,1]n[-1,1]^n has ℓ∞\ell_\infty radius 11, but its corners (1,…,1)(1,\ldots,1) lie at Euclidean distance n\sqrt n from the origin. Measuring ℓt∈[0,1]n\ell_t\in[0,1]^n in ∥⋅∥2\|\cdot\|_2 pays this n\sqrt n diagonal factor.

Example 4 (Why the quadratic regularizer 12∥x−xt∥22\frac12\|x-x_t\|_2^2 is mismatched to Δn\Delta_n) Choosing the quadratic function 12∥x−xt∥22\frac12\|x-x_t\|_2^2 in Proposition 1 was purely for algebraic convenience, and on the simplex Δn\Delta_n it fails in two ways:

  1. The (ℓ2,ℓ2)(\ell_2,\ell_2) versus (ℓ1,ℓ∞)(\ell_1,\ell_\infty) norm mismatch and the dual role of the distance function: Recall how the single-step OGD regret decomposition (Theorem 3) uses the same distance function 12∥y−x∥22\frac12\|y-x\|_2^2 in two places simultaneously: ⟨ℓt, xt−x∗⟩≤12∥xt−x∗∥22−12∥xt+1−x∗∥22η⏟telescopes over T rounds to ∥x1−x∗∥222η+(⟨ℓt, xt−xt+1⟩−12η∥xt−xt+1∥22)⏟non-telescoping per-round remainder ≤η2∥ℓt∥22. \langle\ell_t,\,x_t-x^*\rangle \le\underbrace{\frac{\frac12\|x_t-x^*\|_2^2-\frac12\|x_{t+1}-x^*\|_2^2}{\eta}}_{\text{telescopes over }T\text{ rounds to }\frac{\|x_1-x^*\|_2^2}{2\eta}} +\underbrace{\left(\langle\ell_t,\,x_t-x_{t+1}\rangle-\frac{1}{2\eta}\|x_t-x_{t+1}\|_2^2\right)}_{\text{non-telescoping per-round remainder }\le\frac{\eta}{2}\|\ell_t\|_2^2}.
    • In the non-telescoping remainder on the right, because ℓt∈[0,1]n\ell_t\in[0,1]^n is bounded coordinatewise in ∥ℓt∥∞≤1\|\ell_t\|_\infty\le 1, measuring ℓt\ell_t in ∥⋅∥2\|\cdot\|_2 via Cauchy–Schwarz pays ∥ℓt∥22≤n∥ℓt∥∞2≤n\|\ell_t\|_2^2\le n\|\ell_t\|_\infty^2\le n. To replace ∥ℓt∥22\|\ell_t\|_2^2 by ∥ℓt∥∞2≤1\|\ell_t\|_\infty^2\le 1 via Hölder’s inequality ⟨ℓt,xt−xt+1⟩≤∥ℓt∥∞∥xt−xt+1∥1\langle\ell_t,x_t-x_{t+1}\rangle\le\|\ell_t\|_\infty\|x_t-x_{t+1}\|_1, the regularizer divergence must lower-bound 12∥xt+1−xt∥12\frac12\|x_{t+1}-x_t\|_1^2.
    • In the telescoping potential drop on the left, requiring the divergence to lower-bound 12∥x∗−x1∥12\frac12\|x^*-x_1\|_1^2 would blow up by Θ(n)\Theta(n) if VV were the hypercube [−1,1]n[-1,1]^n—except that on the probability simplex Δn\Delta_n, every point satisfies ∥x∥1=∑i=1nxi=1\|x\|_1=\sum_{i=1}^n x_i=1, so the primal ℓ1\ell_1 diameter ∥x∗−x1∥1≤2\|x^*-x_1\|_1\le 2 is already O(1)O(1)! Because Δn\Delta_n is ℓ1\ell_1-bounded in the primal space while ℓt∈[0,1]n\ell_t\in[0,1]^n is ℓ∞\ell_\infty-bounded in the dual space, switching to an ℓ1\ell_1-compatible regularizer wins on both ends at once.
  2. Flat boundary curvature (ψ′′(xi)≡1\psi''(x_i)\equiv 1) and local spring stiffness: How does a regularizer ψ\psi control the local step size Δx:=xt+1−xt\Delta x:=x_{t+1}-x_t? In one dimension, the first-order Taylor remainder Bψ(x,xt):=ψ(x)−ψ(xt)−ψ′(xt)(x−xt)B_\psi(x,x_t):=\psi(x)-\psi(x_t)-\psi'(x_t)(x-x_t) has second-order Taylor approximation Bψ(x,xt)≈12ψ′′(xt)(Δx)2B_\psi(x,x_t)\approx\frac12\psi''(x_t)(\Delta x)^2 near xtx_t. Minimizing gtΔx+12ηψ′′(xt)(Δx)2g_t\Delta x+\frac{1}{2\eta}\psi''(x_t)(\Delta x)^2 over Δx\Delta x yields Δx=xt+1−xt≈−ηψ′′(xt) gt,gt(xt−xt+1)≈η gt2ψ′′(xt). \Delta x=x_{t+1}-x_t\approx -\frac{\eta}{\psi''(x_t)}\,g_t, \qquad g_t(x_t-x_{t+1})\approx\eta\,\frac{g_t^2}{\psi''(x_t)}. Geometrically, the second derivative ψ′′(xt)\psi''(x_t) acts as the local spring stiffness (damping coefficient) anchoring xtx_t. The Euclidean regularizer ψ(x)=12∑i=1nxi2\psi(x)=\frac12\sum_{i=1}^n x_i^2 has constant stiffness ψ′′(xi)≡1\psi''(x_i)\equiv 1 everywhere on Δn\Delta_n, penalizing a step near the cliff xi≈0x_i\approx 0 no more than at xi=1/nx_i=1/n; when bandit importance sampling produces a spike ℓ^t(i)=ℓt(i)/xt(i)\hat\ell_t(i)=\ell_t(i)/x_t(i), the flat Euclidean bowl cannot brake the step and suffers E[ℓ^t(i)2/ψ′′(xt(i))]=ℓt(i)2/xt(i)→∞\mathbb E[\hat\ell_t(i)^2/\psi''(x_t(i))]=\ell_t(i)^2/x_t(i)\to\infty. Designing a regularizer whose stiffness diverges as ψ′′(xi)=1xi→+∞\psi''(x_i)=\frac{1}{x_i}\to+\infty as xi→0+x_i\to 0^+ automatically scales steps as Δxi≈−ηxt(i)gt(i)\Delta x_i\approx -\eta x_t(i)g_t(i) (preventing boundary crashes) and cancels the 1/xt(i)1/x_t(i) bandit variance blow-up: E[ℓ^t(i)2/ψ′′(xt(i))]=ℓt(i)2xt(i)⋅xt(i)=ℓt(i)2≤1\mathbb E\bigl[\hat\ell_t(i)^2/\psi''(x_t(i))\bigr]=\frac{\ell_t(i)^2}{x_t(i)}\cdot x_t(i)=\ell_t(i)^2\le 1!

Bregman divergence and the two canonical geometries

How do we generalize 12∥x−xt∥22\frac12\|x-x_t\|_2^2 to an arbitrary strictly convex differentiable regularizer ψ:D→R\psi:\mathcal D\to\mathbb R? Notice that 12∥y−x∥22=12∥y∥22−[12∥x∥22+⟨x,y−x⟩]\frac12\|y-x\|_2^2=\frac12\|y\|_2^2-\bigl[\frac12\|x\|_2^2+\langle x,y-x\rangle\bigr] is simply the difference between ψ(y)\psi(y) and its first-order Taylor approximation at xx.

Definition 5 (Bregman divergence and strong convexity) Let ψ:D→R\psi:\mathcal D\to\mathbb R be continuously differentiable and strictly convex on a convex domain D⊆Rn\mathcal D\subseteq\mathbb R^n containing VV.

  1. For any y∈Dy\in\mathcal D and x∈int⁡(D)x\in\operatorname{int}(\mathcal D), the Bregman divergence of ψ\psi is defined by Bψ(y,x)≡Dψ(y,x):=ψ(y)−[ψ(x)+⟨∇ψ(x), y−x⟩]≥0. B_\psi(y,x)\equiv D_\psi(y,x):=\psi(y)-\bigl[\psi(x)+\langle\nabla\psi(x),\,y-x\rangle\bigr]\ge 0.
  2. Given a norm ∥⋅∥\|\cdot\| on Rn\mathbb R^n with dual norm ∥g∥∗=sup⁡∥v∥≤1⟨g,v⟩\|g\|_*=\sup_{\|v\|\le 1}\langle g,v\rangle, we say that ψ\psi is μ\mu-strongly convex with respect to ∥⋅∥\|\cdot\| on VV if Bψ(y,x)≥μ2∥y−x∥2B_\psi(y,x)\ge\frac{\mu}{2}\|y-x\|^2 for all x,y∈Vx,y\in V.

Let us compute Bψ(y,x)B_\psi(y,x) and the Hessian ∇2ψ(x)\nabla^2\psi(x) for our two flagship regularizers:

  1. Squared Euclidean norm ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2 on Rn\mathbb R^n: Here ∇ψ(x)=x\nabla\psi(x)=x, ∇2ψ(x)=I\nabla^2\psi(x)=I, and Bψ(y,x)=12∥y∥22−[12∥x∥22+⟨x, y−x⟩]=12∥y−x∥22. B_\psi(y,x)=\frac12\|y\|_2^2-\left[\frac12\|x\|_2^2+\langle x,\,y-x\rangle\right]=\frac12\|y-x\|_2^2.
  2. Negative Shannon entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i on R≥0n\mathbb R_{\ge0}^n: Recall that for a distribution p∈Δnp\in\Delta_n, its Shannon entropy is H(p):=−∑i=1npiln⁡piH(p):=-\sum_{i=1}^n p_i\ln p_i. For ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i, differentiating gives (∇ψ(x))i=1+ln⁡xi(\nabla\psi(x))_i=1+\ln x_i and the diagonal Hessian ∇2ψ(x)i,j={1xi,if i=j,0,otherwise. \nabla^2\psi(x)_{i,j}=\begin{cases} \dfrac{1}{x_i}, & \text{if }i=j,\\[6pt] 0, & \text{otherwise.} \end{cases} Its Bregman divergence on R>0n\mathbb R_{>0}^n is Bψ(y,x)=∑i=1nyiln⁡yi−∑i=1nxiln⁡xi−∑i=1n(yi−xi)(1+ln⁡xi)=∑i=1nyiln⁡yixi−∑i=1nyi+∑i=1nxi. B_\psi(y,x)=\sum_{i=1}^n y_i\ln y_i-\sum_{i=1}^n x_i\ln x_i-\sum_{i=1}^n(y_i-x_i)(1+\ln x_i) =\sum_{i=1}^n y_i\ln\frac{y_i}{x_i}-\sum_{i=1}^n y_i+\sum_{i=1}^n x_i. When restricted to probability distributions x,y∈Δnx,y\in\Delta_n (so ∑i=1nyi=∑i=1nxi=1\sum_{i=1}^n y_i=\sum_{i=1}^n x_i=1), the linear terms cancel and Bψ(y,x)B_\psi(y,x) becomes the Kullback–Leibler (KL) divergence (relative entropy): Bψ(y,x)=∑i=1nyiln⁡yixi=KL⁡(y∥x). B_\psi(y,x)=\sum_{i=1}^n y_i\ln\frac{y_i}{x_i}=\operatorname{KL}(y\|x).

Although Bψ(y,x)B_\psi(y,x) is in general asymmetric (Bψ(y,x)≠Bψ(x,y)B_\psi(y,x)\ne B_\psi(x,y)), it obeys an exact three-point identity that generalizes the Euclidean law of cosines.

Lemma 4 (Three-point identity for Bregman divergences) For any x∈Dx\in\mathcal D and y,z∈int⁡(D)y,z\in\operatorname{int}(\mathcal D), ⟨∇ψ(z)−∇ψ(y), x−y⟩=Bψ(x,y)−Bψ(x,z)+Bψ(y,z). \bigl\langle\nabla\psi(z)-\nabla\psi(y),\,x-y\bigr\rangle=B_\psi(x,y)-B_\psi(x,z)+B_\psi(y,z).

Proof: expanding the three Bregman divergences

Expanding the right-hand side directly from Definition 5, Bψ(x,y)−Bψ(x,z)+Bψ(y,z)=(ψ(x)−ψ(y)−⟨∇ψ(y),x−y⟩)−(ψ(x)−ψ(z)−⟨∇ψ(z),x−z⟩)+(ψ(y)−ψ(z)−⟨∇ψ(z),y−z⟩)=−⟨∇ψ(y),x−y⟩+⟨∇ψ(z),(x−z)−(y−z)⟩=⟨∇ψ(z)−∇ψ(y), x−y⟩. \begin{aligned} B_\psi(x,y)-B_\psi(x,z)+B_\psi(y,z) &=\bigl(\psi(x)-\psi(y)-\langle\nabla\psi(y),x-y\rangle\bigr) -\bigl(\psi(x)-\psi(z)-\langle\nabla\psi(z),x-z\rangle\bigr)\\ &\quad+\bigl(\psi(y)-\psi(z)-\langle\nabla\psi(z),y-z\rangle\bigr)\\ &=-\langle\nabla\psi(y),x-y\rangle+\langle\nabla\psi(z),(x-z)-(y-z)\rangle =\bigl\langle\nabla\psi(z)-\nabla\psi(y),\,x-y\bigr\rangle. \end{aligned}

1.8 Online Mirror Descent and its master regret bound

Now replace the quadratic proximity regularizer 12η∥x−xt∥22\frac{1}{2\eta}\|x-x_t\|_2^2 in Proposition 1 by the Bregman divergence 1ηBψ(x,xt)\frac1\eta B_\psi(x,x_t) of a strictly convex regularizer ψ\psi. The resulting updating rule is Online Mirror Descent (OMD): xt+1:=arg⁡min⁡x∈V{⟨∇f(xt), x−xt⟩+1ηBψ(x,xt)}. x_{t+1}:=\arg\min_{x\in V}\left\{\bigl\langle\nabla f(x_t),\,x-x_t\bigr\rangle+\frac{1}{\eta}B_\psi(x,x_t)\right\}.

Four complementary perspectives on Online Mirror Descent

Why is this algorithm called Mirror Descent, and how does it cure the simplex bottlenecks of §1.7? We can understand the update above from four complementary viewpoints:

  1. Primal proximal view: Directly replacing the flat Euclidean bowl 12∥x−xt∥22\frac12\|x-x_t\|_2^2 in Proposition 1 by the asymmetric Bregman potential well Bψ(x,xt)B_\psi(x,x_t) penalizes candidate moves x−xtx-x_t according to the geometry of ψ\psi.
  2. Local Riemannian metric view: By a second-order Taylor expansion at xtx_t, there exists ξ∈[xt,x]\xi\in[x_t,x] such that Bψ(x,xt)=12(x−xt)⊤∇2ψ(ξ)(x−xt)≈12∥x−xt∥∇2ψ(xt)2. B_\psi(x,x_t)=\frac12(x-x_t)^\top\nabla^2\psi(\xi)(x-x_t)\approx\frac12\|x-x_t\|_{\nabla^2\psi(x_t)}^2. Thus ψ\psi equips the primal space VV with a position-dependent Riemannian metric tensor G(xt)=∇2ψ(xt)G(x_t)=\nabla^2\psi(x_t). When ∂2ψ∂xi2→+∞\frac{\partial^2\psi}{\partial x_i^2}\to+\infty and ∂ψ∂xi→−∞\frac{\partial\psi}{\partial x_i}\to-\infty near the boundary xi→0+x_i\to 0^+, the diverging curvature strongly penalizes local coordinate steps toward 00 while the diverging mirror coordinate maps finite dual steps back to strictly positive coordinates, preventing iterates from crashing into 00 without artificial α\alpha-truncation.
  3. Dual mirror-space view (why Mirror Descent?): Mathematically, primal states xt∈Vx_t\in V and gradients ∇f(xt)∈V∗\nabla f(x_t)\in V^* (linear functionals on VV acting via ⟨∇f(xt),x⟩\langle\nabla f(x_t),x\rangle) live in dual vector spaces. Writing xt−η∇f(xt)x_t-\eta\nabla f(x_t) in Euclidean OGD only made sense because the identity gradient ∇(12∥x∥22)=x\nabla\bigl(\frac12\|x\|_2^2\bigr)=x tacitly identified VV with V∗V^*. In non-Euclidean geometries (such as ℓ1\ell_1 probabilities versus ℓ∞\ell_\infty loss prices), Mirror Descent first maps xt∈Vx_t\in V to its dual representation Zt=∇ψ(xt)∈V∗Z_t=\nabla\psi(x_t)\in V^* via the mirror map ∇ψ:V→V∗\nabla\psi:V\to V^*, takes the linear gradient step Zt+1=Zt−η∇f(xt)Z_{t+1}=Z_t-\eta\nabla f(x_t) inside V∗V^*, maps back via the inverse mirror map xt+1′=(∇ψ)−1(Zt+1)=∇ψ∗(Zt+1)x'_{t+1}=(\nabla\psi)^{-1}(Z_{t+1})=\nabla\psi^*(Z_{t+1}), and projects onto VV under BψB_\psi.
  4. FTRL historical-accumulation view: When V=RnV=\mathbb R^n is unconstrained (or on Δn\Delta_n under negative entropy, where projection shifts dual coordinates by a uniform constant), unfolding the dual recursion Zt+1=Z0−η∑s=0t∇ℓs(xs)Z_{t+1}=Z_0-\eta\sum_{s=0}^t\nabla\ell_s(x_s) from x0∈arg⁡min⁡x∈Vψ(x)x_0\in\arg\min_{x\in V}\psi(x) (∇ψ(x0)=0\nabla\psi(x_0)=0) shows that OMD coincides with Follow the Regularized Leader (FTRL): ∇ψ(xt+1)=−η∑s=0t∇ℓs(xs)⟺xt+1=arg⁡min⁡x∈V{∑s=0t⟨∇ℓs(xs), x⟩+1ηψ(x)}, \nabla\psi(x_{t+1})=-\eta\sum_{s=0}^t\nabla\ell_s(x_s) \quad\Longleftrightarrow\quad x_{t+1}=\arg\min_{x\in V}\left\{\sum_{s=0}^t\bigl\langle\nabla\ell_s(x_s),\,x\bigr\rangle+\frac{1}{\eta}\psi(x)\right\}, which directly cures the step-function oscillation of unregularized Follow the Leader (§1.2) by adding the strictly convex stabilizer 1ηψ(x)\frac1\eta\psi(x)!
Primal V (project to V) xₜ x′ₜ₊₁ xₜ₊₁ Π_V^ψ Dual V* (Zₜ = ∇ψ(xₜ)) Zₜ Zₜ₊₁ − η∇f(xₜ) ∇ψ (∇ψ)⁻¹

The mirror map ∇ψ:V→V∗\nabla\psi:V\to V^* sends xtx_t to Zt=∇ψ(xt)Z_t=\nabla\psi(x_t), takes a linear gradient step Zt+1=Zt−η∇f(xt)Z_{t+1}=Z_t-\eta\nabla f(x_t) in the dual space, maps back via xt+1′=(∇ψ)−1(Zt+1)x'_{t+1}=(\nabla\psi)^{-1}(Z_{t+1}), and projects onto VV with respect to BψB_\psi.

Origin of Mirror Descent and Legendre duality. Arkadi Nemirovski and David Yudin (Problem Complexity and Method Efficiency in Optimization, 1983) invented Mirror Descent after observing that xt∈Vx_t\in V and ∇f(xt)∈V∗\nabla f(x_t)\in V^* live in dual spaces. On the unconstrained interior (or on Δn\Delta_n under negative entropy), accumulating dual gradients Zt+1=Z0−η∑s=0t∇ℓs(xs)Z_{t+1}=Z_0-\eta\sum_{s=0}^t\nabla\ell_s(x_s) initialized at x0∈arg⁡min⁡x∈Vψ(x)x_0\in\arg\min_{x\in V}\psi(x) is also equivalent to Follow the Regularized Leader (FTRL): xt+1=arg⁡min⁡x∈V{η∑s=0t⟨∇ℓs(xs),x⟩+ψ(x)}x_{t+1}=\arg\min_{x\in V}\bigl\{\eta\sum_{s=0}^t\langle\nabla\ell_s(x_s),x\rangle+\psi(x)\bigr\}.

Proposition 2 (Mirror-map equivalence and Bregman projection) Let ψ:D→R\psi:\mathcal D\to\mathbb R be a Legendre convex function with dual image D∗=∇ψ(int⁡(D))\mathcal D^*=\nabla\psi(\operatorname{int}(\mathcal D)) containing Zt+1=Zt−η∇f(xt)Z_{t+1}=Z_t-\eta\nabla f(x_t).

  1. Unconstrained mirror step: When V=RnV=\mathbb R^n, the proximal update xt+1=arg⁡min⁡x∈Rn{⟨∇f(xt),x−xt⟩+1ηBψ(x,xt)}x_{t+1}=\arg\min_{x\in\mathbb R^n}\bigl\{\langle\nabla f(x_t),x-x_t\rangle+\frac1\eta B_\psi(x,x_t)\bigr\} is equivalent to ∇ψ(xt+1)=∇ψ(xt)−η∇f(xt). \nabla\psi(x_{t+1})=\nabla\psi(x_t)-\eta\nabla f(x_t).
  2. Constrained Bregman projection: When V⊆DV\subseteq\mathcal D is a closed convex subset, the constrained proximal update xt+1=arg⁡min⁡x∈V{⟨∇f(xt),x−xt⟩+1ηBψ(x,xt)}x_{t+1}=\arg\min_{x\in V}\bigl\{\langle\nabla f(x_t),x-x_t\rangle+\frac1\eta B_\psi(x,x_t)\bigr\} is equivalent to the four-step procedure:
    • Map to dual space: Zt=∇ψ(xt)Z_t=\nabla\psi(x_t);
    • Take a linear gradient step in dual space: Zt+1=Zt−η∇f(xt)Z_{t+1}=Z_t-\eta\nabla f(x_t);
    • Map back to primal space: xt+1′=(∇ψ)−1(Zt+1)x'_{t+1}=(\nabla\psi)^{-1}(Z_{t+1});
    • Project onto VV under the Bregman divergence: xt+1=ΠVψ(xt+1′):=arg⁡min⁡x∈VBψ(x,xt+1′), x_{t+1}=\Pi_V^\psi(x'_{t+1}):=\arg\min_{x\in V} B_\psi(x,x'_{t+1}), which satisfies the generalized Pythagorean contraction Bψ(x∗,xt+1)+Bψ(xt+1,xt+1′)≤Bψ(x∗,xt+1′)B_\psi(x^*,x_{t+1})+B_\psi(x_{t+1},x'_{t+1})\le B_\psi(x^*,x'_{t+1}) for all x∗∈Vx^*\in V.

Proof. Define G(x):=⟨∇f(xt),x−xt⟩+1ηBψ(x,xt)=⟨∇f(xt),x−xt⟩+1η[ψ(x)−ψ(xt)−⟨∇ψ(xt),x−xt⟩]G(x):=\langle\nabla f(x_t),x-x_t\rangle+\frac1\eta B_\psi(x,x_t)=\langle\nabla f(x_t),x-x_t\rangle+\frac1\eta\bigl[\psi(x)-\psi(x_t)-\langle\nabla\psi(x_t),x-x_t\rangle\bigr]. Differentiating with respect to xx gives ∇G(x)=∇f(xt)+1η(∇ψ(x)−∇ψ(xt)). \nabla G(x)=\nabla f(x_t)+\frac1\eta\bigl(\nabla\psi(x)-\nabla\psi(x_t)\bigr). In the unconstrained case, ∇G(x)=0  ⟺  ∇ψ(x)=∇ψ(xt)−η∇f(xt)\nabla G(x)=0\iff\nabla\psi(x)=\nabla\psi(x_t)-\eta\nabla f(x_t), proving (1). In the constrained case, let xt+1′x'_{t+1} satisfy ∇ψ(xt+1′)=∇ψ(xt)−η∇f(xt)\nabla\psi(x'_{t+1})=\nabla\psi(x_t)-\eta\nabla f(x_t); then ∇xBψ(x,xt+1′)=∇ψ(x)−∇ψ(xt+1′)=η∇G(x)\nabla_x B_\psi(x,x'_{t+1})=\nabla\psi(x)-\nabla\psi(x'_{t+1})=\eta\nabla G(x), so x↦Bψ(x,xt+1′)x\mapsto B_\psi(x,x'_{t+1}) and ηG(x)\eta G(x) differ by a constant independent of xx and share the exact same minimizer xt+1∈Vx_{t+1}\in V (see Proposition 5 in §2.2 for the generalized Pythagorean inequality).

The master regret bound and the local inverse-Hessian norm

How does replacing the Euclidean regularizer 12∥x∥22\frac12\|x\|_2^2 by a general regularizer ψ\psi change the regret bound of Theorem 3? For any positive definite matrix M≻0M\succ 0, write ∥v∥M:=v⊤Mv\|v\|_M:=\sqrt{v^\top M v} for the induced Mahalanobis norm. In Reading Contents §2.1, we prove the following master regret theorem for Online Mirror Descent via a unified continuous-time discretization analysis:

Theorem 6 (Master regret bound for Online Mirror Descent (local inverse-Hessian and global dual-norm forms)) Let V⊆D⊆RnV\subseteq\mathcal D\subseteq\mathbb R^n be a closed convex set, let ψ\psi be a twice continuously differentiable Legendre regularizer on D\mathcal D with ∇2ψ≻0\nabla^2\psi\succ 0 on int⁡(D)\operatorname{int}(\mathcal D) and dual steps Zt−ηtgt∈∇ψ(int⁡(D))Z_t-\eta_t g_t\in\nabla\psi(\operatorname{int}(\mathcal D)), and let ℓ0,…,ℓT−1:V→R\ell_0,\ldots,\ell_{T-1}:V\to\mathbb R be convex loss functions with gradients gt=∇ℓt(xt)g_t=\nabla\ell_t(x_t). Setting ϕ(x):=Bψ(x∗,x)\phi(x):=B_\psi(x^*,x), the Online Mirror Descent iterates xt+1=arg⁡min⁡x∈V{⟨gt,x−xt⟩+1ηtBψ(x,xt)}x_{t+1}=\arg\min_{x\in V}\bigl\{\langle g_t,x-x_t\rangle+\frac{1}{\eta_t}B_\psi(x,x_t)\bigr\} satisfy:

  1. Local inverse-Hessian norm bound: For every comparator x∗∈Vx^*\in V, R(T)=∑t=0T−1(ℓt(xt)−ℓt(x∗))≤∑t=0T−1[ϕ(xt)−ϕ(xt+1)ηt+ηt2sup⁡ξt∈[xt,xt+1′]∥∇ℓt(xt)∥(∇2ψ(ξt))−12]. R(T)=\sum_{t=0}^{T-1}\bigl(\ell_t(x_t)-\ell_t(x^*)\bigr) \le\sum_{t=0}^{T-1}\left[\frac{\phi(x_t)-\phi(x_{t+1})}{\eta_t}+\frac{\eta_t}{2}\sup_{\xi_t\in[x_t,x'_{t+1}]}\bigl\|\nabla\ell_t(x_t)\bigr\|_{(\nabla^2\psi(\xi_t))^{-1}}^2\right]. In particular, with constant step size ηt=η\eta_t=\eta, R(T)≤Bψ(x∗,x0)η+η2∑t=0T−1sup⁡ξt∈[xt,xt+1′]∥∇ℓt(xt)∥(∇2ψ(ξt))−12. R(T)\le\frac{B_\psi(x^*,x_0)}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\sup_{\xi_t\in[x_t,x'_{t+1}]}\bigl\|\nabla\ell_t(x_t)\bigr\|_{(\nabla^2\psi(\xi_t))^{-1}}^2.
  2. Global dual-norm bound: If ψ\psi is μ\mu-strongly convex with respect to a norm ∥⋅∥\|\cdot\| on VV (Bψ(y,x)≥μ2∥y−x∥2)\bigl(B_\psi(y,x)\ge\frac\mu2\|y-x\|^2\bigr), then for constant step size η>0\eta>0, R(T)≤Bψ(x∗,x0)η+η2μ∑t=0T−1∥∇ℓt(xt)∥∗2. R(T)\le\frac{B_\psi(x^*,x_0)}{\eta}+\frac{\eta}{2\mu}\sum_{t=0}^{T-1}\bigl\|\nabla\ell_t(x_t)\bigr\|_*^2.

Compare Theorem 6 side by side with the Euclidean OGD bound (Theorem 3) to see the exact structural Δ\Delta induced by replacing 12∥x∥22\frac12\|x\|_2^2 with a general Legendre regularizer ψ\psi:

Structural component Euclidean OGD (ψ(x)=12∥x∥22)\bigl(\psi(x)=\frac12\|x\|_2^2\bigr) Online Mirror Descent (general Legendre ψ\psi) First-principles origin of the Δ\Delta
Local proximity regularizer 12∥x−xt∥22\frac12\|x-x_t\|_2^2 Bψ(x,xt)≈12∥x−xt∥∇2ψ(xt)2B_\psi(x,x_t)\approx\frac12\|x-x_t\|_{\nabla^2\psi(x_t)}^2 Primal displacements x−xtx-x_t are measured in the local Hessian metric Ht=∇2ψ(xt)H_t=\nabla^2\psi(x_t)
Telescoping potential term 12∥x∗−x0∥22η\dfrac{\frac12\|x^*-x_0\|_2^2}{\eta} Bψ(x∗,x0)η\dfrac{B_\psi(x^*,x_0)}{\eta} Euclidean potential ϕ(x)=12∥x∗−x∥22\phi(x)=\frac12\|x^*-x\|_2^2 upgrades to the Bregman potential ϕ(x)=Bψ(x∗,x)\phi(x)=B_\psi(x^*,x)
Second-order discretization lag η2∑t=0T−1gt⊤I−1gt=η2∑t=0T−1∥gt∥22\dfrac{\eta}{2}\sum_{t=0}^{T-1}g_t^\top I^{-1}g_t=\dfrac{\eta}{2}\sum_{t=0}^{T-1}\|g_t\|_2^2 η2∑t=0T−1sup⁡ξt∈[xt,xt+1′]gt⊤(∇2ψ(ξt))−1gt⏟∥gt∥(∇2ψ(ξt))−12\dfrac{\eta}{2}\sum_{t=0}^{T-1}\sup_{\xi_t\in[x_t,x'_{t+1}]}\underbrace{g_t^\top\bigl(\nabla^2\psi(\xi_t)\bigr)^{-1}g_t}_{\|g_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2} Measuring primal steps in H=∇2ψH=\nabla^2\psi induces the dual quadratic norm with inverse matrix H−1=(∇2ψ)−1H^{-1}=(\nabla^2\psi)^{-1}: max⁡d{η⟨gt,d⟩−12d⊤Hd}=η22gt⊤H−1gt\max_d\{\eta\langle g_t,d\rangle-\frac12 d^\top H d\}=\frac{\eta^2}{2}g_t^\top H^{-1}g_t

In particular, when ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2, we have ∇2ψ(x)=I\nabla^2\psi(x)=I, so ∥gt∥(∇2ψ(ξt))−12=∥gt∥22\|g_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2=\|g_t\|_2^2 and Theorem 6 immediately recovers Theorem 3 as a special case! (See Reading Contents §2.1 for the complete continuous-time discretization proof and the three-line discrete verification via the Three-Point Identity.)

1.9 Understanding Multiplicative Weights (Hedge) and EXP3 via Online Mirror Descent

With the general Online Mirror Descent theorem (Theorem 6) in hand, we can now resolve both simplex puzzles (§§1.5–1.6) at a single stroke—and answer a foundational first-principles question: Why on earth would an online algorithm update probability distributions using an exponential function e−ηℓt(i)e^{-\eta\ell_t(i)}?

1. First-principles derivation of Multiplicative Weights from negative-entropy OMD

Return to the learning with expert advice problem (§1.5) on the probability simplex V=ΔnV=\Delta_n, where the round-tt loss is ⟨ℓt,xt⟩\langle\ell_t,x_t\rangle with gradient ∇ℓt(xt)=ℓt∈[0,1]n\nabla\ell_t(x_t)=\ell_t\in[0,1]^n. By Part 1 of Theorem 6, running Online Mirror Descent with any regularizer ψ\psi yields R(T)≤Bψ(x∗,x0)η+η2∑t=0T−1sup⁡ξt∈[xt,xt+1′]∥ℓt∥(∇2ψ(ξt))−12. R(T)\le\frac{B_\psi(x^*,x_0)}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\sup_{\xi_t\in[x_t,x'_{t+1}]}\|\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2. Look at the local inverse-Hessian norm ∥ℓt∥(∇2ψ(ξt))−12=ℓt⊤(∇2ψ(ξt))−1ℓt\|\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2=\ell_t^\top\bigl(\nabla^2\psi(\xi_t)\bigr)^{-1}\ell_t:

Now let us plug ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i—with mirror map ∇ψ(x)=(1+ln⁡x(1),…,1+ln⁡x(n))\nabla\psi(x)=\bigl(1+\ln x(1),\ldots,1+\ln x(n)\bigr)—into the four-step Online Mirror Descent procedure of Proposition 2:

  1. Map to dual space: Zt(i)=1+ln⁡xt(i)Z_t(i)=1+\ln x_t(i) for each i∈[n]i\in[n].
  2. Linear gradient step in dual space: Zt+1(i)=1+ln⁡xt(i)−ηℓt(i)Z_{t+1}(i)=1+\ln x_t(i)-\eta\ell_t(i).
  3. Invert the mirror map (1+ln⁡xt+1′(i)=Zt+1(i)1+\ln x'_{t+1}(i)=Z_{t+1}(i)): xt+1′(i)=xt(i) e−ηℓt(i)! x'_{t+1}(i)=x_t(i)\,e^{-\eta\ell_t(i)}!
  4. Bregman projection onto Δn\Delta_n: We solve xt+1=ΠΔnψ(xt+1′)=arg⁡min⁡x∈Δn{∑i=1nx(i)ln⁡x(i)xt+1′(i)−∑i=1nx(i)+∑i=1nxt+1′(i)}. x_{t+1}=\Pi_{\Delta_n}^\psi(x'_{t+1})=\arg\min_{x\in\Delta_n}\left\{\sum_{i=1}^n x(i)\ln\frac{x(i)}{x'_{t+1}(i)}-\sum_{i=1}^n x(i)+\sum_{i=1}^n x'_{t+1}(i)\right\}. Introducing a Lagrange multiplier λ∈R\lambda\in\mathbb R for the simplex constraint ∑i=1nx(i)=1\sum_{i=1}^n x(i)=1 and setting the derivative with respect to x(i)x(i) to zero gives ln⁡xt+1(i)xt+1′(i)+λ=0\ln\frac{x_{t+1}(i)}{x'_{t+1}(i)}+\lambda=0, so xt+1(i)=e−λxt+1′(i)x_{t+1}(i)=e^{-\lambda}x'_{t+1}(i). Enforcing ∑i=1nxt+1(i)=1\sum_{i=1}^n x_{t+1}(i)=1 simply normalizes the vector xt+1′x'_{t+1}: xt+1(i)=xt+1′(i)∑j=1nxt+1′(j)=xt(i)e−ηℓt(i)∑j=1nxt(j)e−ηℓt(j). x_{t+1}(i)=\frac{x'_{t+1}(i)}{\sum_{j=1}^n x'_{t+1}(j)}=\frac{x_t(i)e^{-\eta\ell_t(i)}}{\sum_{j=1}^n x_t(j)e^{-\eta\ell_t(j)}}.

Why exponential weights? We do not multiply probabilities by e−ηℓt(i)e^{-\eta\ell_t(i)} as an ad-hoc algebraic trick; rather, exp⁡(⋅)\exp(\cdot) is the inverse mirror map (∇ψ)−1(\nabla\psi)^{-1} of negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i, which is the unique separable regularizer on Δn\Delta_n whose curvature ∇2ψ(x)=diag⁡(1/xi)\nabla^2\psi(x)=\operatorname{diag}(1/x_i) matches the geometry of probability distributions!

Definition 6 (Multiplicative Weights Update (MWU / Hedge / Entropic OMD)) Fix a learning rate η>0\eta>0. Initialize weights w1(i):=1w_1(i):=1 (so p1(i):=x0(i)=1np_1(i):=x_0(i)=\frac1n) for all i∈[n]i\in[n]. At each round t=1,…,Tt=1,\ldots,T:

  1. Normalize the current weights into a probability distribution on Δn\Delta_n: Φt:=∑i=1nwt(i),pt(i):=wt(i)Φt(i∈[n]). \Phi_t:=\sum_{i=1}^n w_t(i), \qquad p_t(i):=\frac{w_t(i)}{\Phi_t}\quad (i\in[n]).
  2. Observe the loss vector ℓt\ell_t, incur expected loss ⟨pt,ℓt⟩\langle p_t,\ell_t\rangle, and update the weights multiplicatively: wt+1(i):=wt(i) e−ηℓt(i)=exp⁡ ⁣(−ηLt(i)),pt+1(i)=pt(i)e−ηℓt(i)∑j=1npt(j)e−ηℓt(j). w_{t+1}(i):=w_t(i)\,e^{-\eta\ell_t(i)}=\exp\!\bigl(-\eta L_t(i)\bigr), \qquad p_{t+1}(i)=\frac{p_t(i)e^{-\eta\ell_t(i)}}{\sum_{j=1}^n p_t(j)e^{-\eta\ell_t(j)}}.

Independent discoveries of Multiplicative Weights. The multiplicative-weights method was discovered independently across learning theory, optimization, and game theory: Nick Littlestone and Manfred Warmuth (The Weighted Majority Algorithm, Information and Computation 1994) introduced exponential updates for mistake bounds; Yoav Freund and Robert Schapire (A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting, JCSS 1997) formulated Hedge; and Sanjeev Arora, Elad Hazan, and Satyen Kale (The Multiplicative Weights Update Method: a Meta-Algorithm and Applications, Theory of Computing 2012) unified its algorithmic applications across TCS.

Perspective / Property General OMD formula Euclidean ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2 (OGD) Negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i (MWU / Hedge)
Mirror map ∇ψ:V→V∗\nabla\psi:V\to V^* Z=∇ψ(x)Z=\nabla\psi(x) Z=xZ=x Z(i)=1+ln⁡x(i)Z(i)=1+\ln x(i)
Inverse Hessian (∇2ψ(x))−1(\nabla^2\psi(x))^{-1} (∇2ψ(x))−1(\nabla^2\psi(x))^{-1} InI_n diag⁡(x1,…,xn)\operatorname{diag}(x_1,\ldots,x_n)
Update on Δn\Delta_n xt+1=ΠΔnψ((∇ψ)−1(∇ψ(xt)−ηℓt))x_{t+1}=\Pi_{\Delta_n}^\psi\bigl((\nabla\psi)^{-1}(\nabla\psi(x_t)-\eta\ell_t)\bigr) xt+1=ΠΔn(xt−ηℓt)x_{t+1}=\Pi_{\Delta_n}(x_t-\eta\ell_t) xt+1(i)=xt(i)e−ηℓt(i)∑jxt(j)e−ηℓt(j)x_{t+1}(i)=\frac{x_t(i)e^{-\eta\ell_t(i)}}{\sum_j x_t(j)e^{-\eta\ell_t(j)}}
Experts regret (ℓt∈[0,1]n\ell_t\in[0,1]^n) Bψ(x∗,x0)η+η2∑tsup⁡ξt∥ℓt∥(∇2ψ(ξt))−12\frac{B_\psi(x^*,x_0)}{\eta}+\frac{\eta}{2}\sum_t\sup_{\xi_t}\|\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2 12η+ηnT2=O(nT)\frac{1}{2\eta}+\frac{\eta nT}{2}=O(\sqrt{nT}) ln⁡nη+ηT2=O(Tln⁡n)\frac{\ln n}{\eta}+\frac{\eta T}{2}=O(\sqrt{T\ln n})

Example 5 (How MWU cures FTL’s alternating-loss oscillation) On the n=2n=2 alternating sequence of Example 1 (ℓ1=(1/2,0)\ell_1=(1/2,0), ℓt=(0,1)\ell_t=(0,1) for even tt, ℓt=(1,0)\ell_t=(1,0) for odd t≥3t\ge 3), before any round t≥2t\ge 2 the cumulative loss gap between the two experts is ∣Lt−1(1)−Lt−1(2)∣=1/2|L_{t-1}(1)-L_{t-1}(2)|=1/2. Instead of jumping to (0,1)(0,1) or (1,0)(1,0) like FTL, MWU plays the current leader with probability 11+e−η/2=12+η8+O(η3)≤12+η8, \frac{1}{1+e^{-\eta/2}}=\frac12+\frac{\eta}{8}+O(\eta^3)\le\frac12+\frac{\eta}{8}, so its excess loss above T/2T/2 over TT rounds is at most ηT8=O(T)\frac{\eta T}{8}=O(\sqrt T) when η=Θ(1/T)\eta=\Theta(1/\sqrt T).

2. Regret bounds for Multiplicative Weights: OMD proof and potential refinements

Let us now evaluate the two terms of our local inverse-Hessian OMD bound (Theorem 6) for ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i starting from the uniform distribution x0=u=(1n,…,1n)x_0=\mathbf u=(\frac1n,\ldots,\frac1n):

  1. Bregman diameter of Δn\Delta_n under ψ\psi: For any comparator distribution x∗∈Δnx^*\in\Delta_n, Bψ(x∗,x0)=KL⁡(x∗∥u)=∑i=1nx∗(i)ln⁡(nx∗(i))=ln⁡n−H(x∗)≤ln⁡n, B_\psi(x^*,x_0)=\operatorname{KL}(x^*\|\mathbf u)=\sum_{i=1}^n x^*(i)\ln\bigl(n x^*(i)\bigr)=\ln n-H(x^*)\le\ln n, so the first term Bψ(x∗,x0)η\frac{B_\psi(x^*,x_0)}{\eta} is at most ln⁡nη\frac{\ln n}{\eta}.
  2. Local inverse-Hessian norm term (and two subtle points on why ξt(i)≤xt(i)\xi_t(i)\le x_t(i)): Because ∇2ψ(ξt)=diag⁡(1/ξt(1),…,1/ξt(n))\nabla^2\psi(\xi_t)=\operatorname{diag}(1/\xi_t(1),\ldots,1/\xi_t(n)), we have ∥ℓt∥(∇2ψ(ξt))−12=∑i=1nℓt(i)2ξt(i)\|\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2=\sum_{i=1}^n\ell_t(i)^2\xi_t(i). To bound ξt(i)\xi_t(i) by xt(i)x_t(i), notice two crucial structural facts that hold for both full-information losses ℓt∈[0,1]n\ell_t\in[0,1]^n and importance-weighted bandit estimators ℓ^t(i)≥0\hat\ell_t(i)\ge 0:
    • Why does ξt\xi_t lie on the segment [xt+1′,xt][x'_{t+1},x_t] rather than [xt+1,xt][x_{t+1},x_t]? After normalization, xt+1∈Δnx_{t+1}\in\Delta_n still sums to 11, so xt+1(i)≤xt(i)x_{t+1}(i)\le x_t(i) cannot hold for all ii. However, in the four-step decomposition of Proposition 2, Step 4 (the Bregman projection xt+1′↦xt+1=ΠΔnψ(xt+1′)x'_{t+1}\mapsto x_{t+1}=\Pi_{\Delta_n}^\psi(x'_{t+1})) only contracts the potential Bψ(x∗,xt+1)≤Bψ(x∗,xt+1′)B_\psi(x^*,x_{t+1})\le B_\psi(x^*,x'_{t+1}) and introduces zero second-order error; the entire discretization lag arises during the unconstrained dual step xt→xt+1′x_t\to x'_{t+1}, so the Taylor mean-value point ξt\xi_t lies on the segment [xt+1′,xt][x'_{t+1},x_t] connecting xtx_t to the unnormalized point xt+1′x'_{t+1}!
    • Why does nonnegativity of the losses guarantee ξt(i)≤xt(i)\xi_t(i)\le x_t(i)? Whenever the input loss vector gtg_t has nonnegative coordinates gt(i)≥0g_t(i)\ge 0 (even if a bandit estimate ℓ^t(At)=ℓt(At)/xt(At)\hat\ell_t(A_t)=\ell_t(A_t)/x_t(A_t) is huge when xt(At)x_t(A_t) is small!), the factor e−ηgt(i)e^{-\eta g_t(i)} lies in (0,1](0,1], so every unnormalized coordinate shrinks: xt+1′(i)=xt(i)e−ηgt(i)≤xt(i)x'_{t+1}(i)=x_t(i)e^{-\eta g_t(i)}\le x_t(i). Consequently, every convex combination ξt∈[xt+1′,xt]\xi_t\in[x'_{t+1},x_t] satisfies ξt(i)≤xt(i)\xi_t(i)\le x_t(i) for all i∈[n]i\in[n]! (By contrast, updating with positive exponential rewards e+ηr^t(i)e^{+\eta\hat r_t(i)} would cause xt+1′(At)x'_{t+1}(A_t) to explode exponentially as xt(At)→0x_t(A_t)\to 0, breaking ξt(i)≤xt(i)\xi_t(i)\le x_t(i).)
    Using ξt(i)≤xt(i)\xi_t(i)\le x_t(i) for ℓt∈[0,1]n\ell_t\in[0,1]^n immediately yields η2∑t=0T−1sup⁡ξt∈[xt+1′,xt]∥ℓt∥(∇2ψ(ξt))−12=η2∑t=0T−1sup⁡ξt∑i=1nℓt(i)2ξt(i)≤η2∑t=0T−1∑i=1nℓt(i)2xt(i)≤η2∑t=0T−1∑i=1nxt(i)=ηT2. \frac{\eta}{2}\sum_{t=0}^{T-1}\sup_{\xi_t\in[x'_{t+1},x_t]}\|\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2 =\frac{\eta}{2}\sum_{t=0}^{T-1}\sup_{\xi_t}\sum_{i=1}^n\ell_t(i)^2\xi_t(i) \le\frac{\eta}{2}\sum_{t=0}^{T-1}\sum_{i=1}^n\ell_t(i)^2 x_t(i) \le\frac{\eta}{2}\sum_{t=0}^{T-1}\sum_{i=1}^n x_t(i)=\frac{\eta T}{2}.

Combining the two pieces gives R(T)≤ln⁡nη+η2∑t=0T−1∑i=1nxt(i)ℓt(i)2≤ln⁡nη+ηT2=2Tln⁡n(at η=2ln⁡nT). R(T)\le\frac{\ln n}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\sum_{i=1}^n x_t(i)\ell_t(i)^2\le\frac{\ln n}{\eta}+\frac{\eta T}{2}=\sqrt{2T\ln n}\qquad\Bigl(\text{at }\eta=\sqrt{\frac{2\ln n}{T}}\Bigr). Notice that the exact same O(Tln⁡n)O(\sqrt{T\ln n}) bound also follows from Part 2 of Theorem 6 via Pinsker’s inequality (KL⁡(y∥x)≥12∥y−x∥12\operatorname{KL}(y\|x)\ge\frac12\|y-x\|_1^2, proved in §2.3), which shows that ψ\psi is 11-strongly convex with respect to ∥⋅∥1\|\cdot\|_1, paired with the dual norm ∥ℓt∥∞≤1\|\ell_t\|_\infty\le 1!

Recall (Hoeffding’s Lemma, Chapter 1). For any random variable Z∈[a,b]Z\in[a,b] and λ∈R\lambda\in\mathbb R, ln⁡E[eλZ]≤λE[Z]+λ2(b−a)28\ln\mathbb E[e^{\lambda Z}]\le\lambda\mathbb E[Z]+\frac{\lambda^2(b-a)^2}{8}. Applying this directly to Φt+1Φt=∑i=1npt(i)e−ηℓt(i)=EI∼pt[e−ηℓt(I)]\frac{\Phi_{t+1}}{\Phi_t}=\sum_{i=1}^n p_t(i)e^{-\eta\ell_t(i)}=\mathbb E_{I\sim p_t}[e^{-\eta\ell_t(I)}] improves the variance constant from 1/21/2 to 1/81/8 when ℓt∈[0,1]n\ell_t\in[0,1]^n.

For our upcoming applications to zero-sum games (§1.10) and linear programming feasibility (§1.11), it is useful to record both the sharp Hoeffding constant for ℓt∈[0,1]n\ell_t\in[0,1]^n and the extension of the local second-moment bound to signed payoffs ℓt∈[−ρ,ρ]n\ell_t\in[-\rho,\rho]^n.

Theorem 7 (Regret bound for Multiplicative Weights on [0,1]n[0,1]^n) For any sequence of loss vectors ℓ1,…,ℓT∈[0,1]n\ell_1,\ldots,\ell_T\in[0,1]^n and any learning rate η>0\eta>0, the Multiplicative Weights Update algorithm (Definition 6) satisfies, for every expert i∗∈[n]i^*\in[n], Reg⁡T(i∗)=∑t=1T⟨pt,ℓt⟩−∑t=1Tℓt(i∗)≤ln⁡nη+ηT8. \operatorname{Reg}_T(i^*)=\sum_{t=1}^T\langle p_t,\ell_t\rangle-\sum_{t=1}^T\ell_t(i^*)\le\frac{\ln n}{\eta}+\frac{\eta T}{8}. In particular, setting η=22ln⁡nT\eta=2\sqrt{\frac{2\ln n}{T}} yields Reg⁡T≤Tln⁡n2\operatorname{Reg}_T\le\sqrt{\frac{T\ln n}{2}}.

Proof. Since pt(i)=wt(i)/Φtp_t(i)=w_t(i)/\Phi_t, at each round tt we have Φt+1Φt=∑i=1npt(i)e−ηℓt(i)=EIt∼pt[e−ηℓt(It)]\frac{\Phi_{t+1}}{\Phi_t}=\sum_{i=1}^n p_t(i)e^{-\eta\ell_t(i)}=\mathbb E_{I_t\sim p_t}[e^{-\eta\ell_t(I_t)}]. Because ℓt(It)∈[0,1]\ell_t(I_t)\in[0,1] has mean ⟨pt,ℓt⟩\langle p_t,\ell_t\rangle, Hoeffding’s Lemma gives ln⁡Φt+1Φt≤−η⟨pt,ℓt⟩+η28\ln\frac{\Phi_{t+1}}{\Phi_t}\le -\eta\langle p_t,\ell_t\rangle+\frac{\eta^2}{8}. Telescoping over t=1,…,Tt=1,\ldots,T (with Φ1=n\Phi_1=n) and comparing against the lower bound ln⁡ΦT+1≥ln⁡wT+1(i∗)=−η∑t=1Tℓt(i∗)\ln\Phi_{T+1}\ge\ln w_{T+1}(i^*)=-\eta\sum_{t=1}^T\ell_t(i^*) yields −η∑t=1Tℓt(i∗)−ln⁡n≤−η∑t=1T⟨pt,ℓt⟩+η2T8-\eta\sum_{t=1}^T\ell_t(i^*)-\ln n\le -\eta\sum_{t=1}^T\langle p_t,\ell_t\rangle+\frac{\eta^2 T}{8}. Dividing by η>0\eta>0 completes the proof.

Theorem 8 (Second-order Multiplicative Weights bound for signed and nonnegative losses) Suppose at each round t=1,…,Tt=1,\ldots,T the vector ℓt∈Rn\ell_t\in\mathbb R^n and learning rate η>0\eta>0 satisfy ηℓt(i)≥−1\eta\ell_t(i)\ge -1 for all i∈[n]i\in[n]. Then Multiplicative Weights (Definition 6) satisfies, for every i∗∈[n]i^*\in[n], ∑t=1T⟨pt,ℓt⟩−∑t=1Tℓt(i∗)≤ln⁡nη+η∑t=1T∑i=1npt(i)ℓt(i)2. \sum_{t=1}^T\langle p_t,\ell_t\rangle-\sum_{t=1}^T\ell_t(i^*)\le\frac{\ln n}{\eta}+\eta\sum_{t=1}^T\sum_{i=1}^n p_t(i)\ell_t(i)^2. If ℓt(i)≥0\ell_t(i)\ge 0 for all t,it,i (with no upper bound on ηℓt(i)\eta\ell_t(i)), the second term improves by a factor of two (matching the OMD local inverse-Hessian bound above): ∑t=1T⟨pt,ℓt⟩−∑t=1Tℓt(i∗)≤ln⁡nη+η2∑t=1T∑i=1npt(i)ℓt(i)2. \sum_{t=1}^T\langle p_t,\ell_t\rangle-\sum_{t=1}^T\ell_t(i^*)\le\frac{\ln n}{\eta}+\frac{\eta}{2}\sum_{t=1}^T\sum_{i=1}^n p_t(i)\ell_t(i)^2. In particular, if ℓt∈[−ρ,ρ]n\ell_t\in[-\rho,\rho]^n for all tt and 0<η≤1/ρ0<\eta\le 1/\rho, then Reg⁡T(i∗)≤ln⁡nη+ηρ2T\operatorname{Reg}_T(i^*)\le\frac{\ln n}{\eta}+\eta\rho^2 T.

Proof: quadratic Taylor bound on the partition function

Using e−u≤1−u+u2e^{-u}\le 1-u+u^2 (valid for all u≥−1u\ge -1, or e−u≤1−u+12u2e^{-u}\le 1-u+\frac12 u^2 for u≥0u\ge 0) and ln⁡(1+z)≤z\ln(1+z)\le z, we have ln⁡Φt+1Φt=ln⁡∑i=1npt(i)e−ηℓt(i)≤ln⁡(1−η⟨pt,ℓt⟩+η2∑i=1npt(i)ℓt(i)2)≤−η⟨pt,ℓt⟩+η2∑i=1npt(i)ℓt(i)2. \ln\frac{\Phi_{t+1}}{\Phi_t}=\ln\sum_{i=1}^n p_t(i)e^{-\eta\ell_t(i)}\le\ln\Bigl(1-\eta\langle p_t,\ell_t\rangle+\eta^2\sum_{i=1}^n p_t(i)\ell_t(i)^2\Bigr)\le -\eta\langle p_t,\ell_t\rangle+\eta^2\sum_{i=1}^n p_t(i)\ell_t(i)^2. Telescoping against ln⁡ΦT+1≥−η∑t=1Tℓt(i∗)\ln\Phi_{T+1}\ge -\eta\sum_{t=1}^T\ell_t(i^*) and dividing by η\eta gives both claims.

Polynomial weight update. One often writes the linearized variant wt+1(i)=wt(i)(1−ηℓt(i))w_{t+1}(i)=w_t(i)(1-\eta\ell_t(i)) for ℓt∈[−1,1]n\ell_t\in[-1,1]^n and η≤1/2\eta\le 1/2, which satisfies Reg⁡T(i∗)≤ln⁡nη+η∑t=1T∣ℓt(i∗)∣\operatorname{Reg}_T(i^*)\le\frac{\ln n}{\eta}+\eta\sum_{t=1}^T|\ell_t(i^*)| with the same O(Tln⁡n)O(\sqrt{T\ln n}) scaling.

3. Online Stochastic Mirror Descent (OSMD / EXP3): resolving the bandit boundary blow-up

Now return to the adversarial multi-armed bandit problem (§1.6). In §1.6, feeding the importance-weighted estimator ℓ^t(i)=1{At=i}xt(i)ℓt(i)\hat\ell_t(i)=\frac{\mathbf 1_{\{A_t=i\}}}{x_t(i)}\ell_t(i) into Euclidean OGD produced the variance E[∥ℓ^t∥22∣Ft−1]=∑i=1nℓt(i)2xt(i)\mathbb E[\|\hat\ell_t\|_2^2\mid\mathcal F_{t-1}]=\sum_{i=1}^n\frac{\ell_t(i)^2}{x_t(i)}, which blew up near xt(i)=0x_t(i)=0 and forced O((nT)2/3)O((nT)^{2/3}) regret.

What happens if we instead run Online Stochastic Mirror Descent (OSMD) with negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i on the exact same estimator ℓ^t\hat\ell_t? In primal coordinates, this update xt+1(i)=xt(i)exp⁡(−ηℓ^t(i))∑j=1nxt(j)exp⁡(−ηℓ^t(j)) x_{t+1}(i)=\frac{x_t(i)\exp\bigl(-\eta\hat\ell_t(i)\bigr)}{\sum_{j=1}^n x_t(j)\exp\bigl(-\eta\hat\ell_t(j)\bigr)} is the celebrated EXP3 algorithm (Exponential-weight algorithm for Exploration and Exploitation, Auer et al. 2002). Look at what happens when we plug ℓ^t(i)≥0\hat\ell_t(i)\ge 0 into our local inverse-Hessian OMD bound (Theorem 6)!

Caution (How the inverse Hessian of negative entropy cancels 1/xt(i)1/x_t(i)).

Because ℓ^t(i)≥0\hat\ell_t(i)\ge 0, every point ξt∈[xt+1′,xt]\xi_t\in[x'_{t+1},x_t] satisfies ξt(i)≤xt(i)\xi_t(i)\le x_t(i). In the local inverse-Hessian norm ∥ℓ^t∥(∇2ψ(ξt))−12=∑i=1nℓ^t(i)2ξt(i)≤∑i=1nℓ^t(i)2xt(i)\|\hat\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2=\sum_{i=1}^n\hat\ell_t(i)^2\xi_t(i)\le\sum_{i=1}^n\hat\ell_t(i)^2 x_t(i), the inverse Hessian (∇2ψ(xt))−1=diag⁡(xt(1),…,xt(n))(\nabla^2\psi(x_t))^{-1}=\operatorname{diag}(x_t(1),\ldots,x_t(n)) supplies an extra factor of xt(i)x_t(i) that cancels the remaining 1/xt(i)1/x_t(i) denominator in E[ℓ^t(i)2∣Ft−1]=ℓt(i)2xt(i)\mathbb E[\hat\ell_t(i)^2\mid\mathcal F_{t-1}]=\frac{\ell_t(i)^2}{x_t(i)}, without any artificial α\alpha-uniform exploration!

Theorem 9 (Regret bound for Online Stochastic Mirror Descent (EXP3) in adversarial bandits) In the adversarial multi-armed bandit problem (Definition 4) with losses ℓt∈[0,1]n\ell_t\in[0,1]^n, running Online Stochastic Mirror Descent with negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i from x0=u=(1n,…,1n)x_0=\mathbf u=(\frac1n,\ldots,\frac1n) on the importance-weighted estimator ℓ^t(i)=1{At=i}xt(i)ℓt(i)\hat\ell_t(i)=\frac{\mathbf 1_{\{A_t=i\}}}{x_t(i)}\ell_t(i) satisfies, for every fixed comparator x∗∈Δnx^*\in\Delta_n, E ⁣[∑t=0T−1⟨ℓt, xt−x∗⟩]≤ln⁡nη+ηnT2. \mathbb E\!\left[\sum_{t=0}^{T-1}\langle\ell_t,\,x_t-x^*\rangle\right] \le\frac{\ln n}{\eta}+\frac{\eta nT}{2}. In particular, maximizing over x∗=ejx^*=\mathbf e_j (j∈[n]j\in[n]) and choosing η=2ln⁡nnT\eta=\sqrt{\frac{2\ln n}{nT}} yields Reg⁡Tbandit≤2nTln⁡n=O(nTln⁡n). \operatorname{Reg}_T^{\mathrm{bandit}}\le\sqrt{2nT\ln n}=O\bigl(\sqrt{nT\ln n}\bigr).

Proof. Fix any deterministic comparator x∗∈Δnx^*\in\Delta_n. By conditional unbiasedness E[ℓ^t∣Ft−1]=ℓt\mathbb E[\hat\ell_t\mid\mathcal F_{t-1}]=\ell_t, Lemma 3, and Part 1 of Theorem 6 applied to the realized sequence ℓ^0,…,ℓ^T−1\hat\ell_0,\ldots,\hat\ell_{T-1} (using ξt(i)≤xt(i)\xi_t(i)\le x_t(i) since ℓ^t(i)≥0\hat\ell_t(i)\ge 0), taking expectations over A0,…,AT−1A_0,\ldots,A_{T-1} gives E ⁣[∑t=0T−1⟨ℓt, xt−x∗⟩]=E ⁣[∑t=0T−1⟨ℓ^t, xt−x∗⟩]≤Bψ(x∗,x0)η+η2 E ⁣[∑t=0T−1sup⁡ξt∈[xt+1′,xt]∥ℓ^t∥(∇2ψ(ξt))−12]≤ln⁡nη+η2∑t=0T−1E ⁣[∑i=1n(1{At=i}ℓt(i)xt(i))2xt(i)]≤ln⁡nη+η2∑t=0T−1E ⁣[∑i=1n1xt(i) xt(i)]=ln⁡nη+ηnT2=2nTln⁡n(at η=2ln⁡nnT). \begin{aligned} \mathbb E\!\left[\sum_{t=0}^{T-1}\langle\ell_t,\,x_t-x^*\rangle\right] =\mathbb E\!\left[\sum_{t=0}^{T-1}\langle\hat\ell_t,\,x_t-x^*\rangle\right] &\le\frac{B_\psi(x^*,x_0)}{\eta}+\frac{\eta}{2}\,\mathbb E\!\left[\sum_{t=0}^{T-1}\sup_{\xi_t\in[x'_{t+1},x_t]}\|\hat\ell_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2\right]\\ &\le\frac{\ln n}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\mathbb E\!\left[\sum_{i=1}^n\left(\frac{\mathbf 1_{\{A_t=i\}}\ell_t(i)}{x_t(i)}\right)^2 x_t(i)\right]\\ &\le\frac{\ln n}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\mathbb E\!\left[\sum_{i=1}^n\frac{1}{x_t(i)}\,x_t(i)\right]\\ &=\frac{\ln n}{\eta}+\frac{\eta nT}{2} =\sqrt{2nT\ln n}\qquad\Bigl(\text{at }\eta=\sqrt{\frac{2\ln n}{nT}}\Bigr). \end{aligned} Taking the maximum over x∗=ejx^*=\mathbf e_j (j∈[n]j\in[n]) gives Reg⁡Tbandit≤2nTln⁡n\operatorname{Reg}_T^{\mathrm{bandit}}\le\sqrt{2nT\ln n}.


1.10 Playing a zero-sum game repeatedly: von Neumann’s minimax theorem

Now that we have derived the Multiplicative Weights Update (MWU) algorithm from first principles via Online Mirror Descent, let us see why this online learning primitive is one of the most versatile tools in algorithm design. The central paradigm is self-play: if we pit a no-regret MWU learner against a dual opponent or constraint oracle that repeatedly exposes the worst-case violation, the time-averaged trajectory of the learner must converge to an equilibrium or feasible solution! Our first payoff is a constructive proof of von Neumann’s minimax theorem for two-player zero-sum games.

Zero-sum matrix games and weak duality

Let A∈[−1,1]m×nA\in[-1,1]^{m\times n} be a payoff matrix with mm rows and nn columns. Two players choose mixed strategies:

When they play (p,q)(p,q), the row player pays the expected amount p⊤Aq=∑i=1m∑j=1np(i)Ai,jq(j)p^\top A q=\sum_{i=1}^m\sum_{j=1}^n p(i)A_{i,j}q(j) to the column player. If the row player must commit to p∈Δmp\in\Delta_m first, the column player responds to maximize p⊤Aqp^\top A q, yielding the minimax (upper) value V‾:=min⁡p∈Δmmax⁡q∈Δnp⊤Aq=min⁡p∈Δmmax⁡j∈[n](p⊤A)j. \overline V:=\min_{p\in\Delta_m}\max_{q\in\Delta_n} p^\top A q=\min_{p\in\Delta_m}\max_{j\in[n]}(p^\top A)_j. If the column player commits to q∈Δnq\in\Delta_n first, the row player responds to minimize p⊤Aqp^\top A q, yielding the maximin (lower) value V‾:=max⁡q∈Δnmin⁡p∈Δmp⊤Aq=max⁡q∈Δnmin⁡i∈[m](Aq)i. \underline V:=\max_{q\in\Delta_n}\min_{p\in\Delta_m} p^\top A q=\max_{q\in\Delta_n}\min_{i\in[m]}(A q)_i.

Recall (Weierstrass extreme value theorem). Because p↦max⁡j∈[n](p⊤A)jp\mapsto\max_{j\in[n]}(p^\top A)_j and q↦min⁡i∈[m](Aq)iq\mapsto\min_{i\in[m]}(A q)_i are continuous on the compact simplices Δm\Delta_m and Δn\Delta_n, both extrema V‾\overline V and V‾\underline V are attained.

Lemma 5 (Weak minimax inequality) For any matrix A∈Rm×nA\in\mathbb R^{m\times n}, moving second can only help: V‾=max⁡q∈Δnmin⁡p∈Δmp⊤Aq≤min⁡p∈Δmmax⁡q∈Δnp⊤Aq=V‾. \underline V=\max_{q\in\Delta_n}\min_{p\in\Delta_m} p^\top A q\le\min_{p\in\Delta_m}\max_{q\in\Delta_n} p^\top A q=\overline V.

Proof. For every pair (p~,q~)∈Δm×Δn(\tilde p,\tilde q)\in\Delta_m\times\Delta_n, we have min⁡p∈Δmp⊤Aq~≤p~⊤Aq~≤max⁡q∈Δnp~⊤Aq\min_{p\in\Delta_m} p^\top A\tilde q\le\tilde p^\top A\tilde q\le\max_{q\in\Delta_n}\tilde p^\top A q. Taking max⁡q~∈Δn\max_{\tilde q\in\Delta_n} on the left and min⁡p~∈Δm\min_{\tilde p\in\Delta_m} on the right gives V‾≤V‾\underline V\le\overline V.

Closing the duality gap via no-regret self-play

To prove that V‾=V‾\underline V=\overline V over mixed strategies—and to compute an ε\varepsilon-approximate Nash equilibrium—we let the row player run Multiplicative Weights over TT rounds against a best-responding column player:

  1. At round t=1,…,Tt=1,\ldots,T, the row player outputs pt∈Δmp_t\in\Delta_m via MWU (Definition 6);
  2. The column player picks a best response qt∈arg⁡max⁡q∈Δnpt⊤Aqq_t\in\arg\max_{q\in\Delta_n} p_t^\top A q;
  3. The row player observes the loss vector ℓt:=Aqt∈[−1,1]m\ell_t:=A q_t\in[-1,1]^m and pays ⟨pt,ℓt⟩=pt⊤Aqt\langle p_t,\ell_t\rangle=p_t^\top A q_t.

From fixed-point topology to online self-play. John von Neumann (Zur Theorie der Gesellschaftsspiele, 1928) originally proved Theorem 10 via a Brouwer-type topological fixed-point argument, before George Dantzig and von Neumann linked zero-sum games to linear programming duality in 1947 (Chapter 6). Replacing Brown (1949) and Robinson’s (1951) fictitious play (FTL) with no-regret MWU accelerates convergence from exponential to O(ε−2ln⁡m)O(\varepsilon^{-2}\ln m) rounds.

Theorem 10 (von Neumann’s minimax theorem and approximate equilibrium via MWU) For every payoff matrix A∈[−1,1]m×nA\in[-1,1]^{m\times n}, the minimax and maximin values coincide: min⁡p∈Δmmax⁡q∈Δnp⊤Aq=max⁡q∈Δnmin⁡p∈Δmp⊤Aq=:V∗. \min_{p\in\Delta_m}\max_{q\in\Delta_n} p^\top A q=\max_{q\in\Delta_n}\min_{p\in\Delta_m} p^\top A q=:V^*. Moreover, if the row player runs Multiplicative Weights (Definition 6) with η=(2ln⁡m)/T\eta=\sqrt{(2\ln m)/T} on ℓt=Aqt∈[−1,1]m\ell_t=A q_t\in[-1,1]^m against best responses qt∈arg⁡max⁡q∈Δnpt⊤Aqq_t\in\arg\max_{q\in\Delta_n} p_t^\top A q for T=⌈2ln⁡mε2⌉T=\lceil\frac{2\ln m}{\varepsilon^2}\rceil rounds, then the time-averaged strategies pˉT:=1T∑t=1Tpt∈Δm\bar p_T:=\frac1T\sum_{t=1}^T p_t\in\Delta_m and qˉT:=1T∑t=1Tqt∈Δn\bar q_T:=\frac1T\sum_{t=1}^T q_t\in\Delta_n form an ε\varepsilon-approximate Nash equilibrium: max⁡q∈ΔnpˉT⊤Aq−ε≤V∗≤min⁡p∈Δmp⊤AqˉT+ε. \max_{q\in\Delta_n}\bar p_T^\top A q-\varepsilon\le V^*\le\min_{p\in\Delta_m} p^\top A\bar q_T+\varepsilon.

Proof. Because ℓt=Aqt∈[−1,1]m\ell_t=A q_t\in[-1,1]^m takes values in an interval of length 22, shifting and scaling ℓt\ell_t to (ℓt+1)/2∈[0,1]m(\ell_t+\mathbf 1)/2\in[0,1]^m with learning rate 2η2\eta leaves ptp_t invariant and scales Theorem 7 by 22, giving 1T∑t=1Tpt⊤Aqt−min⁡i∗∈[m]1T∑t=1Tei∗⊤Aqt=Reg⁡TT≤ln⁡mηT+η2=2ln⁡mT≤ε. \frac1T\sum_{t=1}^T p_t^\top A q_t-\min_{i^*\in[m]}\frac1T\sum_{t=1}^T\mathbf e_{i^*}^\top A q_t =\frac{\operatorname{Reg}_T}{T} \le\frac{\ln m}{\eta T}+\frac{\eta}{2} =\sqrt{\frac{2\ln m}{T}}\le\varepsilon. Now sandwich the time-averaged game value 1T∑t=1Tpt⊤Aqt\frac1T\sum_{t=1}^T p_t^\top A q_t between V‾\overline V and V‾\underline V: V‾≤V‾=min⁡p∈Δmmax⁡q∈Δnp⊤Aq≤max⁡q∈ΔnpˉT⊤Aq=max⁡q∈Δn(1T∑t=1Tpt⊤Aq)≤1T∑t=1Tmax⁡q∈Δnpt⊤Aq=1T∑t=1Tpt⊤Aqt≤min⁡i∗∈[m]ei∗⊤A(1T∑t=1Tqt)+2ln⁡mT=min⁡p∈Δmp⊤AqˉT+2ln⁡mT≤V‾+2ln⁡mT. \begin{aligned} \underline V\le\overline V &=\min_{p\in\Delta_m}\max_{q\in\Delta_n} p^\top A q \le\max_{q\in\Delta_n}\bar p_T^\top A q =\max_{q\in\Delta_n}\Bigl(\frac1T\sum_{t=1}^T p_t^\top A q\Bigr) \le\frac1T\sum_{t=1}^T\max_{q\in\Delta_n} p_t^\top A q\\ &=\frac1T\sum_{t=1}^T p_t^\top A q_t \le\min_{i^*\in[m]}\mathbf e_{i^*}^\top A\Bigl(\frac1T\sum_{t=1}^T q_t\Bigr)+\sqrt{\frac{2\ln m}{T}} =\min_{p\in\Delta_m} p^\top A\bar q_T+\sqrt{\frac{2\ln m}{T}} \le\underline V+\sqrt{\frac{2\ln m}{T}}. \end{aligned} Letting T→∞T\to\infty forces V‾=V‾=V∗\underline V=\overline V=V^*, and setting T=⌈2ε−2ln⁡m⌉T=\lceil 2\varepsilon^{-2}\ln m\rceil gives the ε\varepsilon-equilibrium guarantee.

Example 6 (Asymmetric penalty-kick game and why time-averaging is essential) Consider the 2×22\times 2 game A=(1002)A=\bigl(\begin{smallmatrix}1 & 0\\ 0 & 2\end{smallmatrix}\bigr), whose unique minimax equilibrium is p∗=q∗=(2/3,1/3)p^*=q^*=(2/3,1/3) with value V∗=2/3V^*=2/3. Notice why Theorem 10 outputs the time-averaged strategies (pˉT,qˉT)(\bar p_T,\bar q_T) rather than the last iterate (pT,qT)(p_T,q_T): each best response qt∈arg⁡max⁡qpt⊤Aqq_t\in\arg\max_q p_t^\top A q is a pure column (1,0)(1,0) or (0,1)(0,1), which by itself only secures min⁡pp⊤Aqt=0<2/3\min_p p^\top A q_t=0<2/3. However, to keep ptp_t near (2/3,1/3)(2/3,1/3), the column player must pick column 11 on 2/32/3 of the rounds and column 22 on 1/31/3 of the rounds, so the empirical frequency qˉT=1T∑t=1Tqt\bar q_T=\frac1T\sum_{t=1}^T q_t converges to q∗=(2/3,1/3)q^*=(2/3,1/3)!

1.11 Solving linear feasibility via an oracle: the Plotkin–Shmoys–Tardos framework

Our second algorithmic application turns Multiplicative Weights into a general-purpose solver for linear programming feasibility (and fractional packing/covering optimization). Suppose we want to decide whether there exists a point x∈Rdx\in\mathbb R^d satisfying x∈PandAix≥bifor all i=1,…,m, x\in\mathcal P\qquad\text{and}\qquad A_i x\ge b_i\quad\text{for all }i=1,\ldots,m, where P⊆Rd\mathcal P\subseteq\mathbb R^d is a “simple” convex set (such as a box or decoupled single-commodity flows) and Ax≥bA x\ge b is a system of mm “hard” coupling linear inequalities.

Collapsing mm constraints into one weighted average

Suppose that although checking all mm constraints Aix≥biA_i x\ge b_i simultaneously is hard, for any probability distribution p=(p(1),…,p(m))∈Δmp=\bigl(p(1),\ldots,p(m)\bigr)\in\Delta_m over the mm constraints we can efficiently check the single collapsed linear inequality ∃? x∈P:p⊤Ax≥p⊤b⟺∑i=1mp(i)(Aix−bi)≥0. \exists?\,x\in\mathcal P:\quad p^\top A x\ge p^\top b\quad\Longleftrightarrow\quad\sum_{i=1}^m p(i)\bigl(A_i x-b_i\bigr)\ge 0. Note first that if for some p∈Δmp\in\Delta_m no such x∈Px\in\mathcal P exists, then {x∈P:Ax≥b}\{x\in\mathcal P:Ax\ge b\} is certainly infeasible, and pp is a compact infeasibility certificate.

Definition 7 (ρ\rho-bounded average-constraint oracle) Given a convex set P⊆Rd\mathcal P\subseteq\mathbb R^d, linear constraints Aix≥biA_i x\ge b_i (i∈[m]i\in[m]), and a width parameter ρ>0\rho>0, a ρ\rho-bounded average-constraint oracle takes any p∈Δmp\in\Delta_m and either:

  1. returns a point x∈Px\in\mathcal P satisfying the average constraint and width bound ∑i=1mp(i)(Aix−bi)≥0and∣Aix−bi∣≤ρfor all i∈[m]; \sum_{i=1}^m p(i)\bigl(A_i x-b_i\bigr)\ge 0 \qquad\text{and}\qquad \bigl|A_i x-b_i\bigr|\le\rho\quad\text{for all }i\in[m];
  2. or correctly declares that no x∈Px\in\mathcal P satisfies p⊤Ax≥p⊤bp^\top A x\ge p^\top b (certifying infeasibility).

Why is positive slack treated as a positive loss? In the PST game, each constraint i∈[m]i\in[m] is an “expert.” If the oracle’s point xtx_t satisfies constraint ii with a large margin Aixt−bi≫0A_i x_t-b_i\gg 0, constraint ii is “easy,” so assigning it a positive loss ℓt(i)=Aixt−bi>0\ell_t(i)=A_i x_t-b_i>0 shrinks its weight wt+1(i)=wt(i)e−ηℓt(i)w_{t+1}(i)=w_t(i)e^{-\eta\ell_t(i)}. Conversely, violated constraints have ℓt(i)<0\ell_t(i)<0, so their weights grow exponentially, forcing subsequent oracle calls to satisfy them!

To find a single point xˉT∈P\bar x_T\in\mathcal P satisfying all mm constraints within ε\varepsilon, we treat the mm constraints as experts in MWU and define the round-tt loss of constraint ii as its slack ℓt(i):=Aixt−bi∈[−ρ,ρ]\ell_t(i):=A_i x_t-b_i\in[-\rho,\rho].

Theorem 11 (Plotkin–Shmoys–Tardos approximate feasibility theorem) Let P⊆Rd\mathcal P\subseteq\mathbb R^d be a convex set and let Aix≥biA_i x\ge b_i (i∈[m]i\in[m]) be equipped with a ρ\rho-bounded average-constraint oracle (Definition 7). Given 0<ε≤2ρ0<\varepsilon\le 2\rho, run Multiplicative Weights (Definition 6) over the mm constraints with learning rate η=ε2ρ2≤1ρ\eta=\frac{\varepsilon}{2\rho^2}\le\frac{1}{\rho} and losses ℓt(i)=Aixt−bi\ell_t(i)=A_i x_t-b_i for T:=⌈4ρ2ln⁡mε2⌉ T:=\left\lceil\frac{4\rho^2\ln m}{\varepsilon^2}\right\rceil rounds. Then either some oracle call certifies that {x∈P:Ax≥b}\{x\in\mathcal P:Ax\ge b\} is infeasible, or the average point xˉT:=1T∑t=1Txt∈P\bar x_T:=\frac1T\sum_{t=1}^T x_t\in\mathcal P satisfies every constraint within ε\varepsilon: AixˉT≥bi−εfor all i∈[m]. A_i\bar x_T\ge b_i-\varepsilon\qquad\text{for all }i\in[m].

Proof. Suppose all TT oracle calls return x1,…,xT∈Px_1,\ldots,x_T\in\mathcal P, so xˉT=1T∑t=1Txt∈P\bar x_T=\frac1T\sum_{t=1}^T x_t\in\mathcal P by convexity. At every round tt, the oracle guarantees ⟨pt,ℓt⟩=∑i=1mpt(i)(Aixt−bi)≥0\langle p_t,\ell_t\rangle=\sum_{i=1}^m p_t(i)(A_i x_t-b_i)\ge 0 and ∣ℓt(i)∣≤ρ|\ell_t(i)|\le\rho. Since η=ε2ρ2≤1ρ\eta=\frac{\varepsilon}{2\rho^2}\le\frac{1}{\rho}, the signed second-order MWU bound (Theorem 8) gives, for every constraint i∗∈[m]i^*\in[m], 0≤1T∑t=1T⟨pt,ℓt⟩≤1T∑t=1T(Ai∗xt−bi∗)+ln⁡mηT+ηρ2≤(Ai∗xˉT−bi∗)+ε2+ε2, 0\le\frac1T\sum_{t=1}^T\langle p_t,\ell_t\rangle \le\frac1T\sum_{t=1}^T(A_{i^*}x_t-b_{i^*})+\frac{\ln m}{\eta T}+\eta\rho^2 \le(A_{i^*}\bar x_T-b_{i^*})+\frac{\varepsilon}{2}+\frac{\varepsilon}{2}, which rearranges to Ai∗xˉT≥bi∗−εA_{i^*}\bar x_T\ge b_{i^*}-\varepsilon for all i∗∈[m]i^*\in[m].

Example 7 (Balancing traffic across two parallel links via PST) To route 11 unit of flow across m=2m=2 parallel links with capacity 1/21/2 each, set P={x=(x1,x2)∈R≥02:x1+x2=1}\mathcal P=\{x=(x_1,x_2)\in\mathbb R_{\ge0}^2:x_1+x_2=1\} and constraints 12−xi≥0\frac12-x_i\ge 0 (i=1,2i=1,2), so ℓt(i)=12−xt,i∈[−1/2,1/2]\ell_t(i)=\frac12-x_{t,i}\in[-1/2,1/2] has width ρ=1/2\rho=1/2. Given weights pt=(pt(1),pt(2))∈Δ2p_t=(p_t(1),p_t(2))\in\Delta_2, the oracle maximizes 12−⟨pt,x⟩\frac12-\langle p_t,x\rangle over x∈Δ2x\in\Delta_2 by sending the entire unit of flow along whichever link currently has the smaller weight pt(i)p_t(i). Although each individual response xt∈{(1,0),(0,1)}x_t\in\{(1,0),(0,1)\} overloads one link, MWU exponentially boosts the overloaded link’s weight, forcing the oracle to alternate between the two links so that xˉT=1T∑t=1Txt\bar x_T=\frac1T\sum_{t=1}^T x_t converges to (1/2,1/2)(1/2,1/2) in O(ε−2)O(\varepsilon^{-2}) rounds. (See §2.4 for general multicommodity network routing.)

Where we stand, and the question for Chapter 5

Look back at the arc we have traversed: starting from Follow the Leader on the number guessing game, we developed offline and online Projected Gradient Descent, exposed the n\sqrt n and (nT)2/3(nT)^{2/3} bottlenecks of the Euclidean quadratic regularizer on Δn\Delta_n, generalized to Online Mirror Descent via the local inverse-Hessian norm ∥∇ℓt(xt)∥(∇2ψ(ξt))−12\|\nabla\ell_t(x_t)\|_{(\nabla^2\psi(\xi_t))^{-1}}^2, and saw how negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i derives both MWU / Hedge (O(Tln⁡n)O(\sqrt{T\ln n}) under full information) and EXP3 / OSMD (O(nTln⁡n)O(\sqrt{nT\ln n}) under bandit feedback) from first principles.

This leaves a fundamental question for Chapter 5 (Information Budgets and Lower Bounds): Is the extra n\sqrt n factor in EXP3 (O(nTln⁡n)O(\sqrt{nT\ln n}) versus O(Tln⁡n)O(\sqrt{T\ln n})) merely an artifact of importance weighting, or is Ω(nT)\Omega(\sqrt{nT}) an unbreakable information-theoretic barrier for every bandit algorithm? In Chapter 5 we will see that the exact same relative entropy KL⁡(p∥q)\operatorname{KL}(p\|q) that appeared in §1.7 as the Bregman divergence of negative entropy on the simplex also governs how much statistical evidence a bandit transcript can accumulate—proving that no bandit algorithm can beat Ω(nT)\Omega(\sqrt{nT}).

II. Reading Contents

The sections below provide self-contained derivations and algorithmic extensions for both classroom meetings: the unified continuous-time discretization proof of Online Mirror Descent and Online Gradient Descent (§2.1), the convex-analytic foundation of Mirror Descent via Fenchel conjugation and the generalized Pythagorean inequality (§2.2), a complete proof of Pinsker’s inequality and its connection to normed strong convexity (§2.3), a concrete instantiation of the Plotkin–Shmoys–Tardos framework for multicommodity network routing (§2.4), and the online-to-batch conversion linking online regret to stochastic convex optimization (§2.5).

2.1 Continuous-time discretization proof of Online Mirror Descent and OGD

In §1.4 and §1.8 we stated the regret bounds of Online Projected Gradient Descent (Theorem 3) and Online Mirror Descent (Theorem 6). While one can verify both bounds algebraically in discrete time, a continuous-time approach gives much deeper insight into why the regret bound has the exact form “potential drop plus a second-order local norm penalty”—and proves Theorem 6 (and therefore Theorem 3 as the special case ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2) in a single unified calculation.

1. Why continuous time? Gradient flow and the origin of the second-order penalty

To see the intuition before introducing general regularizers, return first to the unconstrained offline Gradient Descent recurrence xt+1−xt=−η∇f(xt)x_{t+1}-x_t=-\eta\nabla f(x_t) from §1.3. Dividing both sides by η\eta and letting the step size η→0\eta\to 0 turns the discrete difference equation into a continuous-time ordinary differential equation—the gradient flow: dx(t)dt=−∇f(x(t))(t≥0). \frac{\mathrm d x(t)}{\mathrm d t}=-\nabla f\bigl(x(t)\bigr)\qquad (t\ge 0). Let us track the Euclidean potential ϕ(t):=12∥x(t)−x∗∥22\phi(t):=\frac12\|x(t)-x^*\|_2^2 along this continuous trajectory. By the chain rule and convexity of ff, dϕ(t)dt=⟨x(t)−x∗, dx(t)dt⟩=−⟨∇f(x(t)), x(t)−x∗⟩≤f(x∗)−f(x(t)). \frac{\mathrm d\phi(t)}{\mathrm d t} =\Bigl\langle x(t)-x^*,\,\frac{\mathrm d x(t)}{\mathrm d t}\Bigr\rangle =-\bigl\langle\nabla f\bigl(x(t)\bigr),\,x(t)-x^*\bigr\rangle \le f(x^*)-f\bigl(x(t)\bigr). Integrating from t=0t=0 to t=Tt=T yields the continuous-time convergence law ∫0T(f(x(t))−f(x∗)) dt≤∫0T⟨∇f(x(t)), x(t)−x∗⟩ dt=ϕ(0)−ϕ(T)! \int_0^T\bigl(f(x(t))-f(x^*)\bigr)\,\mathrm d t \le\int_0^T\bigl\langle\nabla f(x(t)),\,x(t)-x^*\bigr\rangle\,\mathrm d t =\phi(0)-\phi(T)! Compare this continuous formula with the discrete bound ∑t=0T−1(f(xt)−f(x∗))≤ϕ(x0)−ϕ(xT)η+η2∑t=0T−1∥∇f(xt)∥22\sum_{t=0}^{T-1}(f(x_t)-f(x^*))\le\frac{\phi(x_0)-\phi(x_T)}{\eta}+\frac{\eta}{2}\sum_{t=0}^{T-1}\|\nabla f(x_t)\|_2^2 from Theorem 2: a continuous learner moving instantaneously along −∇f(x(t))-\nabla f(x(t)) pays only the potential difference ϕ(0)−ϕ(T)\phi(0)-\phi(T), with no second-order gradient term at all!

Why, then, does a discrete online algorithm pay a second-order penalty? Because during a discrete round ss of step size ηs\eta_s, the discrete player remains frozen at xsx_s while the continuous trajectory yty_t has already started moving in direction −gs=−∇ℓs(xs)-g_s=-\nabla\ell_s(x_s). Let us now formalize this discretization lag directly for Online Mirror Descent with an arbitrary Legendre regularizer ψ\psi, proving Theorem 6 and Theorem 3 simultaneously.

2. Continuous-time Bregman flow and proof of the Master OMD Theorem (Theorem 6)

Recall from §1.8 that at each round s=0,1,…,T−1s=0,1,\ldots,T-1, Online Mirror Descent observes gs:=∇ℓs(xs)g_s:=\nabla\ell_s(x_s) and updates via Proposition 2: ∇ψ(xs+1′)=∇ψ(xs)−ηsgs,xs+1=ΠVψ(xs+1′)=arg⁡min⁡x∈VBψ(x,xs+1′). \nabla\psi(x'_{s+1})=\nabla\psi(x_s)-\eta_s g_s, \qquad x_{s+1}=\Pi_V^\psi(x'_{s+1})=\arg\min_{x\in V} B_\psi(x,x'_{s+1}). We embed the discrete rounds s=0,1,…,T−1s=0,1,\ldots,T-1 into continuous time t∈[0,TT]t\in[0,\mathcal T_T] by defining the cumulative time grid T0:=0,Ts:=∑i=0s−1ηi(s=1,…,T), \mathcal T_0:=0, \qquad \mathcal T_s:=\sum_{i=0}^{s-1}\eta_i\quad (s=1,\ldots,T), and the piecewise-constant gradient signal gt:=gs=∇ℓs(xs)g_t:=g_s=\nabla\ell_s(x_s) for t∈[Ts,Ts+1)t\in[\mathcal T_s,\mathcal T_{s+1}). On each interval [Ts,Ts+1][\mathcal T_s,\mathcal T_{s+1}] of length Ts+1−Ts=ηs\mathcal T_{s+1}-\mathcal T_s=\eta_s, define the continuous-time mirror trajectory yt∈int⁡(D)y_t\in\operatorname{int}(\mathcal D) starting from yTs:=xsy_{\mathcal T_s}:=x_s by the dual differential equation ddt∇ψ(yt)=−gs(t∈[Ts,Ts+1]). \frac{\mathrm d}{\mathrm d t}\nabla\psi(y_t)=-g_s\qquad\bigl(t\in[\mathcal T_s,\mathcal T_{s+1}]\bigr). Integrating from Ts\mathcal T_s to t∈[Ts,Ts+1]t\in[\mathcal T_s,\mathcal T_{s+1}] gives ∇ψ(yt)=∇ψ(xs)−(t−Ts)gs\nabla\psi(y_t)=\nabla\psi(x_s)-(t-\mathcal T_s)g_s, which immediately implies:

Proposition 3 (Grid interpolation of the continuous mirror trajectory) On each round s=0,1,…,T−1s=0,1,\ldots,T-1, the continuous trajectory yty_t starts at yTs=xsy_{\mathcal T_s}=x_s and reaches the unprojected mirror step at t=Ts+1t=\mathcal T_{s+1}: yTs+1=xs+1′=(∇ψ)−1(∇ψ(xs)−ηsgs). y_{\mathcal T_{s+1}}=x'_{s+1}=(\nabla\psi)^{-1}\bigl(\nabla\psi(x_s)-\eta_s g_s\bigr). In particular, when V=RnV=\mathbb R^n is unconstrained, yTs=xsy_{\mathcal T_s}=x_s at every grid point s=0,1,…,Ts=0,1,\ldots,T.

Now measure the distance from any comparator x∗∈Vx^*\in V to yty_t by the Bregman potential ϕ(y):=Bψ(x∗,y)=ψ(x∗)−ψ(y)−⟨∇ψ(y), x∗−y⟩. \phi(y):=B_\psi(x^*,y)=\psi(x^*)-\psi(y)-\bigl\langle\nabla\psi(y),\,x^*-y\bigr\rangle. Differentiating ϕ(yt)\phi(y_t) with respect to t∈(Ts,Ts+1)t\in(\mathcal T_s,\mathcal T_{s+1}) via the product and chain rules produces an exact cancellation: ddtϕ(yt)=−⟨∇ψ(yt), dytdt⟩−⟨ddt∇ψ(yt), x∗−yt⟩+⟨∇ψ(yt), dytdt⟩=⟨ddt∇ψ(yt), yt−x∗⟩=−⟨gs, yt−x∗⟩! \begin{aligned} \frac{\mathrm d}{\mathrm d t}\phi(y_t) &=-\Bigl\langle\nabla\psi(y_t),\,\frac{\mathrm d y_t}{\mathrm d t}\Bigr\rangle -\Bigl\langle\frac{\mathrm d}{\mathrm d t}\nabla\psi(y_t),\,x^*-y_t\Bigr\rangle +\Bigl\langle\nabla\psi(y_t),\,\frac{\mathrm d y_t}{\mathrm d t}\Bigr\rangle\\ &=\Bigl\langle\frac{\mathrm d}{\mathrm d t}\nabla\psi(y_t),\,y_t-x^*\Bigr\rangle =-\langle g_s,\,y_t-x^*\rangle! \end{aligned} Integrating this exact derivative from t=Tst=\mathcal T_s to t=Ts+1t=\mathcal T_{s+1} and applying the generalized Pythagorean contraction ϕ(xs+1)=Bψ(x∗,xs+1)≤Bψ(x∗,xs+1′)=ϕ(yTs+1)\phi(x_{s+1})=B_\psi(x^*,x_{s+1})\le B_\psi(x^*,x'_{s+1})=\phi(y_{\mathcal T_{s+1}}) (Proposition 5 in §2.2) yields the continuous-time conservation inequality on round ss: ∫TsTs+1⟨gs, yt−x∗⟩ dt=ϕ(yTs)−ϕ(yTs+1)=ϕ(xs)−ϕ(xs+1′)≤ϕ(xs)−ϕ(xs+1). \int_{\mathcal T_s}^{\mathcal T_{s+1}}\langle g_s,\,y_t-x^*\rangle\,\mathrm d t =\phi(y_{\mathcal T_s})-\phi(y_{\mathcal T_{s+1}}) =\phi(x_s)-\phi(x'_{s+1}) \le\phi(x_s)-\phi(x_{s+1}).

Now compare the discrete learner’s linearized loss ηs⟨gs,xs−x∗⟩\eta_s\langle g_s,x_s-x^*\rangle on round ss with the continuous integral ∫TsTs+1⟨gs,yt−x∗⟩ dt\int_{\mathcal T_s}^{\mathcal T_{s+1}}\langle g_s,y_t-x^*\rangle\,\mathrm d t. Because ∫TsTs+1⟨gs,x∗⟩ dt=ηs⟨gs,x∗⟩\int_{\mathcal T_s}^{\mathcal T_{s+1}}\langle g_s,x^*\rangle\,\mathrm d t=\eta_s\langle g_s,x^*\rangle, their difference is the discretization lag between the frozen point xsx_s and the moving point yty_t: ηs⟨gs, xs−x∗⟩−∫TsTs+1⟨gs, yt−x∗⟩ dt=∫TsTs+1⟨gs, xs−yt⟩ dt. \eta_s\langle g_s,\,x_s-x^*\rangle-\int_{\mathcal T_s}^{\mathcal T_{s+1}}\langle g_s,\,y_t-x^*\rangle\,\mathrm d t =\int_{\mathcal T_s}^{\mathcal T_{s+1}}\langle g_s,\,x_s-y_t\rangle\,\mathrm d t. Let us evaluate ⟨gs,xs−yt⟩\langle g_s,x_s-y_t\rangle for t∈[Ts,Ts+1]t\in[\mathcal T_s,\mathcal T_{s+1}]:

Alternative discrete verification via the Three-Point Identity and Taylor’s theorem

Both parts of Theorem 6 also follow in three lines from the discrete optimality condition of xt+1x_{t+1}. By first-order optimality of xt+1=arg⁡min⁡x∈V{ηt⟨gt,x⟩+Bψ(x,xt)}x_{t+1}=\arg\min_{x\in V}\{\eta_t\langle g_t,x\rangle+B_\psi(x,x_t)\} and the Three-Point Identity (Lemma 4), ηt⟨gt, xt−x∗⟩=ηt⟨gt, xt+1−x∗⟩+ηt⟨gt, xt−xt+1⟩≤⟨∇ψ(xt+1)−∇ψ(xt), x∗−xt+1⟩+ηt⟨gt, xt−xt+1⟩=Bψ(x∗,xt)−Bψ(x∗,xt+1)−Bψ(xt+1,xt)+ηt⟨gt, xt−xt+1⟩. \begin{aligned} \eta_t\langle g_t,\,x_t-x^*\rangle &=\eta_t\langle g_t,\,x_{t+1}-x^*\rangle+\eta_t\langle g_t,\,x_t-x_{t+1}\rangle\\ &\le\bigl\langle\nabla\psi(x_{t+1})-\nabla\psi(x_t),\,x^*-x_{t+1}\bigr\rangle+\eta_t\langle g_t,\,x_t-x_{t+1}\rangle\\ &=B_\psi(x^*,x_t)-B_\psi(x^*,x_{t+1})-B_\psi(x_{t+1},x_t)+\eta_t\langle g_t,\,x_t-x_{t+1}\rangle. \end{aligned} For Part 2, Bψ(xt+1,xt)≥μ2∥xt−xt+1∥2B_\psi(x_{t+1},x_t)\ge\frac\mu2\|x_t-x_{t+1}\|^2 and Fenchel–Young gives ηt⟨gt,xt−xt+1⟩≤ηt22μ∥gt∥∗2+μ2∥xt−xt+1∥2\eta_t\langle g_t,x_t-x_{t+1}\rangle\le\frac{\eta_t^2}{2\mu}\|g_t\|_*^2+\frac\mu2\|x_t-x_{t+1}\|^2. For Part 1, by the generalized Pythagorean inequality Bψ(x∗,xt+1)≤Bψ(x∗,xt+1′)B_\psi(x^*,x_{t+1})\le B_\psi(x^*,x'_{t+1}) and the Three-Point Identity with ∇ψ(xt)−∇ψ(xt+1′)=ηtgt\nabla\psi(x_t)-\nabla\psi(x'_{t+1})=\eta_t g_t and dt:=xt−xt+1′d_t:=x_t-x'_{t+1}, there is ξt∈[xt,xt+1′]\xi_t\in[x_t,x'_{t+1}] with Bψ(xt+1′,xt)=12dt⊤∇2ψ(ξt)dtB_\psi(x'_{t+1},x_t)=\frac12 d_t^\top\nabla^2\psi(\xi_t)d_t, so ηt⟨gt,xt−x∗⟩−Bψ(x∗,xt)+Bψ(x∗,xt+1′)=Bψ(xt,xt+1′)=ηt⟨gt,dt⟩−Bψ(xt+1′,xt)=ηt⟨gt,dt⟩−12dt⊤∇2ψ(ξt)dt≤ηt22∥gt∥(∇2ψ(ξt))−12 \begin{aligned} \eta_t\langle g_t,x_t-x^*\rangle-B_\psi(x^*,x_t)+B_\psi(x^*,x'_{t+1}) &=B_\psi(x_t,x'_{t+1}) =\eta_t\langle g_t,d_t\rangle-B_\psi(x'_{t+1},x_t)\\ &=\eta_t\langle g_t,d_t\rangle-\frac12 d_t^\top\nabla^2\psi(\xi_t)d_t \le\frac{\eta_t^2}{2}\|g_t\|_{(\nabla^2\psi(\xi_t))^{-1}}^2 \end{aligned} for some ξt∈[xt,xt+1′]\xi_t\in[x_t,x'_{t+1}]. Dividing by ηt\eta_t and summing over t=0,…,T−1t=0,\ldots,T-1 completes both proofs.

2.2 Fenchel conjugation, the Young–Fenchel gap, and constrained Bregman projections

In §§1.7–1.8 we introduced the mirror map ∇ψ:V→V∗\nabla\psi:V\to V^*, its inverse (∇ψ)−1=∇ψ∗(\nabla\psi)^{-1}=\nabla\psi^*, and the Bregman divergence Bψ(x,y)B_\psi(x,y). Here we develop the convex duality behind these objects from first principles and prove that the one-step proximal update of Online Mirror Descent is equivalent to an unconstrained dual gradient step followed by a Bregman projection.

Paired spaces and the Fenchel conjugate

Let V=RnV=\mathbb R^n be the primal vector space (where decision points x∈Kx\in K live) and let V∗=RnV^*=\mathbb R^n be the dual space of linear functionals (where gradients y=gty=g_t live), paired by the bilinear evaluation map ⟨y,x⟩=∑i=1ny(i)x(i)\langle y,x\rangle=\sum_{i=1}^n y(i)x(i). Although VV and V∗V^* are both Rn\mathbb R^n as sets of coordinates, keeping their roles distinct prevents us from confusing primal displacements x−x′∈Vx-x'\in V with dual covectors y∈V∗y\in V^*.

Given a convex function ψ:D→R\psi:\mathcal D\to\mathbb R on a convex domain D⊆V\mathcal D\subseteq V, every affine function x↦⟨y,x⟩−cx\mapsto\langle y,x\rangle-c with slope y∈V∗y\in V^* that lies below ψ\psi on D\mathcal D must satisfy ⟨y,x⟩−c≤ψ(x)\langle y,x\rangle-c\le\psi(x) for all x∈Dx\in\mathcal D, or equivalently c≥sup⁡x∈D{⟨y,x⟩−ψ(x)}c\ge\sup_{x\in\mathcal D}\{\langle y,x\rangle-\psi(x)\}. The tightest such intercept cc as a function of the slope y∈V∗y\in V^* defines the Fenchel conjugate.

Definition 8 (Fenchel conjugate) For a function ψ:D→R\psi:\mathcal D\to\mathbb R on D⊆V\mathcal D\subseteq V, its Fenchel conjugate ψ∗:V∗→R∪{+∞}\psi^*:V^*\to\mathbb R\cup\{+\infty\} is defined by ψ∗(y):=sup⁡x∈D{⟨y,x⟩−ψ(x)}(y∈V∗). \psi^*(y):=\sup_{x\in\mathcal D}\Bigl\{\langle y,x\rangle-\psi(x)\Bigr\}\qquad (y\in V^*).

Because ψ∗(y)\psi^*(y) is the pointwise supremum of a family of affine functions y↦⟨y,x⟩−ψ(x)y\mapsto\langle y,x\rangle-\psi(x), ψ∗\psi^* is always a convex function on V∗V^*. Moreover, directly from the definition of supremum, for every x∈Dx\in\mathcal D and every y∈V∗y\in V^* we have the Young–Fenchel inequality: ψ(x)+ψ∗(y)≥⟨y,x⟩. \psi(x)+\psi^*(y)\ge\langle y,x\rangle.

When does equality hold in the Young–Fenchel inequality, and how does ψ∗\psi^* relate to the Bregman divergence Bψ(x,z)B_\psi(x,z)?

Proposition 4 (Inverse mirror map and Bregman divergence as the Young–Fenchel gap) Suppose ψ:D→R\psi:\mathcal D\to\mathbb R is strictly convex and differentiable on int⁡(D)\operatorname{int}(\mathcal D) with ∥∇ψ(x)∥→∞\|\nabla\psi(x)\|\to\infty as xx approaches the boundary of D\mathcal D (a Legendre function). Let D∗:=∇ψ(int⁡(D))⊆V∗\mathcal D^*:=\nabla\psi(\operatorname{int}(\mathcal D))\subseteq V^* denote the open dual image of ∇ψ\nabla\psi. Then:

  1. For every z∈int⁡(D)z\in\operatorname{int}(\mathcal D) and y∈D∗y\in\mathcal D^*, equality ψ(z)+ψ∗(y)=⟨y,z⟩\psi(z)+\psi^*(y)=\langle y,z\rangle holds if and only if y=∇ψ(z)y=\nabla\psi(z), which in turn holds if and only if z=∇ψ∗(y)z=\nabla\psi^*(y). In particular, the gradient of the Fenchel conjugate is the inverse mirror map on D∗\mathcal D^*: ∇ψ∗=(∇ψ)−1. \nabla\psi^*=(\nabla\psi)^{-1}.
  2. For any x∈Dx\in\mathcal D and z∈int⁡(D)z\in\operatorname{int}(\mathcal D) with dual image y=∇ψ(z)y=\nabla\psi(z), the Bregman divergence Bψ(x,z)B_\psi(x,z) equals the Young–Fenchel gap between x∈Vx\in V and y=∇ψ(z)∈V∗y=\nabla\psi(z)\in V^*: Bψ(x,z)=ψ(x)+ψ∗(∇ψ(z))−⟨∇ψ(z),x⟩≥0. B_\psi(x,z)=\psi(x)+\psi^*\bigl(\nabla\psi(z)\bigr)-\bigl\langle\nabla\psi(z),x\bigr\rangle\ge 0.
Proof: Legendre stationarity and Young–Fenchel substitution

For (1), fix y∈D∗y\in\mathcal D^*. Since ψ\psi is strictly convex and y∈∇ψ(int⁡(D))y\in\nabla\psi(\operatorname{int}(\mathcal D)), the concave function x↦⟨y,x⟩−ψ(x)x\mapsto\langle y,x\rangle-\psi(x) attains its unique maximum on int⁡(D)\operatorname{int}(\mathcal D) at the point zz where its gradient vanishes: y−∇ψ(z)=0y-\nabla\psi(z)=0. At this maximizer zz, the supremum in Definition 8 is achieved: ψ∗(∇ψ(z))=⟨∇ψ(z),z⟩−ψ(z). \psi^*\bigl(\nabla\psi(z)\bigr)=\bigl\langle\nabla\psi(z),z\bigr\rangle-\psi(z). By the envelope theorem for the unique maximizer z(y)z(y) of x↦⟨y,x⟩−ψ(x)x\mapsto\langle y,x\rangle-\psi(x) (or by the chain rule when ∇2ψ(z)≻0\nabla^2\psi(z)\succ 0, which gives ∇ψ∗(y)=z(y)+(Dz(y))⊤(y−∇ψ(z(y)))=z(y)\nabla\psi^*(y)=z(y)+(D z(y))^\top(y-\nabla\psi(z(y)))=z(y)), we have ∇ψ∗(y)=z(y)\nabla\psi^*(y)=z(y), so ∇ψ∗=(∇ψ)−1\nabla\psi^*=(\nabla\psi)^{-1}.

For (2), substitute the identity −ψ(z)=ψ∗(∇ψ(z))−⟨∇ψ(z),z⟩-\psi(z)=\psi^*(\nabla\psi(z))-\langle\nabla\psi(z),z\rangle from (1) into the definition of Bψ(x,z)B_\psi(x,z): Bψ(x,z)=ψ(x)−ψ(z)−⟨∇ψ(z),x−z⟩=ψ(x)+ψ∗(∇ψ(z))−⟨∇ψ(z),x⟩, B_\psi(x,z) =\psi(x)-\psi(z)-\bigl\langle\nabla\psi(z),x-z\bigr\rangle =\psi(x)+\psi^*\bigl(\nabla\psi(z)\bigr)-\bigl\langle\nabla\psi(z),x\bigr\rangle, which is nonnegative by the Young–Fenchel inequality.

Example 8 (Fenchel conjugates of the Euclidean norm and negative entropy) Let us compute ψ∗(y)\psi^*(y) and ∇ψ∗(y)\nabla\psi^*(y) explicitly for our two canonical regularizers:

  1. Squared Euclidean norm ψ(x)=12∥x∥22\psi(x)=\frac12\|x\|_2^2 on V=RnV=\mathbb R^n: ψ∗(y)=sup⁡x∈Rn{⟨y,x⟩−12∥x∥22}. \psi^*(y)=\sup_{x\in\mathbb R^n}\Bigl\{\langle y,x\rangle-\frac12\|x\|_2^2\Bigr\}. Setting the gradient y−x=0y-x=0 gives the maximizer x=yx=y, so ψ∗(y)=12∥y∥22\psi^*(y)=\frac12\|y\|_2^2 and ∇ψ∗(y)=y\nabla\psi^*(y)=y. The Young–Fenchel gap is 12∥x∥22+12∥y∥22−⟨y,x⟩=12∥x−y∥22\frac12\|x\|_2^2+\frac12\|y\|_2^2-\langle y,x\rangle=\frac12\|x-y\|_2^2.
  2. Negative entropy ψ(p)=∑i=1np(i)ln⁡p(i)\psi(p)=\sum_{i=1}^n p(i)\ln p(i) restricted to the simplex D=Δn\mathcal D=\Delta_n: ψ∗(y)=max⁡p∈Δn{∑i=1ny(i)p(i)−∑i=1np(i)ln⁡p(i)}. \psi^*(y)=\max_{p\in\Delta_n}\left\{\sum_{i=1}^n y(i)p(i)-\sum_{i=1}^n p(i)\ln p(i)\right\}. Solving via a Lagrange multiplier for ∑i=1np(i)=1\sum_{i=1}^n p(i)=1, the maximizer is the Gibbs (softmax) distribution p∗(i)=ey(i)∑j=1ney(j)=(∇ψ∗(y))i. p^*(i)=\frac{e^{y(i)}}{\sum_{j=1}^n e^{y(j)}}=\bigl(\nabla\psi^*(y)\bigr)_i. Substituting p∗p^* back into the objective shows that the Fenchel conjugate of negative entropy on Δn\Delta_n is the log-sum-exp (log-partition) function: ψ∗(y)=∑i=1np∗(i)(y(i)−y(i)+ln⁡∑j=1ney(j))=ln⁡∑j=1ney(j). \psi^*(y)=\sum_{i=1}^n p^*(i)\left(y(i)-y(i)+\ln\sum_{j=1}^n e^{y(j)}\right)=\ln\sum_{j=1}^n e^{y(j)}.

Equivalence of proximal OMD and two-step Bregman projection

In §1.8 (Proposition 2) we proved that the proximal update xs+1=arg⁡min⁡x∈K{ηs⟨∇ℓs(xs),x⟩+Bψ(x,xs)}x_{s+1}=\arg\min_{x\in K}\{\eta_s\langle\nabla\ell_s(x_s),x\rangle+B_\psi(x,x_s)\} coincides with the two-step Bregman projection ΠKψ(xs+1′)=arg⁡min⁡x∈KBψ(x,xs+1′)\Pi_K^\psi(x'_{s+1})=\arg\min_{x\in K}B_\psi(x,x'_{s+1}). Let us record the generalized Pythagorean inequality that holds for any Bregman projection onto a closed convex set K⊆DK\subseteq\mathcal D.

Proposition 5 (Generalized Pythagorean inequality for Bregman projections) Let ψ:D→R\psi:\mathcal D\to\mathbb R be a Legendre function and let K⊆DK\subseteq\mathcal D be closed and convex. For any xs+1′∈int⁡(D)x'_{s+1}\in\operatorname{int}(\mathcal D), its Bregman projection xs+1=ΠKψ(xs+1′)=arg⁡min⁡x∈KBψ(x,xs+1′)x_{s+1}=\Pi_K^\psi(x'_{s+1})=\arg\min_{x\in K}B_\psi(x,x'_{s+1}) satisfies, for every comparator x∗∈Kx^*\in K, ⟨∇ψ(xs+1′)−∇ψ(xs+1), x∗−xs+1⟩≤0 \bigl\langle\nabla\psi(x'_{s+1})-\nabla\psi(x_{s+1}),\,x^*-x_{s+1}\bigr\rangle\le 0 and consequently Bψ(x∗,xs+1)+Bψ(xs+1,xs+1′)≤Bψ(x∗,xs+1′). B_\psi(x^*,x_{s+1})+B_\psi(x_{s+1},x'_{s+1})\le B_\psi(x^*,x'_{s+1}).

Proof. First-order optimality of xs+1=arg⁡min⁡x∈KBψ(x,xs+1′)x_{s+1}=\arg\min_{x\in K} B_\psi(x,x'_{s+1}) gives ⟨∇xBψ(xs+1,xs+1′),x∗−xs+1⟩=⟨∇ψ(xs+1)−∇ψ(xs+1′),x∗−xs+1⟩≥0\langle\nabla_x B_\psi(x_{s+1},x'_{s+1}),x^*-x_{s+1}\rangle=\langle\nabla\psi(x_{s+1})-\nabla\psi(x'_{s+1}),x^*-x_{s+1}\rangle\ge 0. Applying the Three-Point Identity (Lemma 4) with (x,y,z)=(x∗,xs+1,xs+1′)(x,y,z)=(x^*,x_{s+1},x'_{s+1}) yields Bψ(x∗,xs+1′)−Bψ(x∗,xs+1)−Bψ(xs+1,xs+1′)=⟨∇ψ(xs+1)−∇ψ(xs+1′), x∗−xs+1⟩≥0, B_\psi(x^*,x'_{s+1})-B_\psi(x^*,x_{s+1})-B_\psi(x_{s+1},x'_{s+1}) =\bigl\langle\nabla\psi(x_{s+1})-\nabla\psi(x'_{s+1}),\,x^*-x_{s+1}\bigr\rangle\ge 0, which rearranges to Bψ(x∗,xs+1)+Bψ(xs+1,xs+1′)≤Bψ(x∗,xs+1′)B_\psi(x^*,x_{s+1})+B_\psi(x_{s+1},x'_{s+1})\le B_\psi(x^*,x'_{s+1}).

2.3 Proof of Pinsker’s inequality: strong convexity of negative entropy in ℓ1\ell_1

In §§1.8–1.9 we analyzed Online Mirror Descent using the local inverse-Hessian norm ∥g∥(∇2ψ(ξ))−12=g⊤(∇2ψ(ξ))−1g\|g\|_{(\nabla^2\psi(\xi))^{-1}}^2=g^\top(\nabla^2\psi(\xi))^{-1}g. A complementary global viewpoint bounds the Hessian from below with respect to a fixed norm ∥⋅∥\|\cdot\| on VV: if ψ\psi is 11-strongly convex with respect to ∥⋅∥\|\cdot\| in the sense that h⊤∇2ψ(ξ)h≥∥h∥2h^\top\nabla^2\psi(\xi)h\ge\|h\|^2 for all hh, then its inverse Hessian is bounded by the squared dual norm, ∥g∥(∇2ψ(ξ))−12≤∥g∥∗2:=sup⁡∥h∥≤1⟨g,h⟩2. \|g\|_{(\nabla^2\psi(\xi))^{-1}}^2\le\|g\|_*^2:=\sup_{\|h\|\le 1}\langle g,h\rangle^2. For negative Shannon entropy ψ(p)=∑i=1np(i)ln⁡p(i)\psi(p)=\sum_{i=1}^n p(i)\ln p(i) on Δn\Delta_n, the natural primal norm is the ℓ1\ell_1 norm ∥p−q∥1=∑i=1n∣p(i)−q(i)∣\|p-q\|_1=\sum_{i=1}^n|p(i)-q(i)| (whose dual norm is ∥g∥∞=max⁡i∣g(i)∣\|g\|_\infty=\max_i|g(i)|). The fact that ψ\psi is 11-strongly convex on Δn\Delta_n with respect to ∥⋅∥1\|\cdot\|_1 is Pinsker’s inequality, which we will use again in Chapter 5 to prove information-theoretic lower bounds.

History of Pinsker’s inequality and its sharp constant. Mark Pinsker (Information and Information Stability of Random Variables and Processes, 1960) originally proved that total variation distance is bounded by a multiple of KL⁡(p∥q)\sqrt{\operatorname{KL}(p\|q)} with a suboptimal constant. The sharp inequality KL⁡(p∥q)≥12∥p−q∥12=2 dTV(p,q)2\operatorname{KL}(p\|q)\ge\frac12\|p-q\|_1^2=2\,d_{\mathrm{TV}}(p,q)^2 was established independently by Imre Csiszár (1967), Solomon Kullback (1967), and Johannes Kemperman (1969).

Theorem 12 (Pinsker’s inequality) For any two probability distributions p,q∈Δnp,q\in\Delta_n, Bψ(p,q)=KL⁡(p∥q)=∑i=1np(i)ln⁡p(i)q(i)≥12∥p−q∥12=2 dTV(p,q)2. B_\psi(p,q)=\operatorname{KL}(p\|q)=\sum_{i=1}^n p(i)\ln\frac{p(i)}{q(i)}\ge\frac12\|p-q\|_1^2=2\,d_{\mathrm{TV}}(p,q)^2. In particular, ψ(p)=∑i=1np(i)ln⁡p(i)\psi(p)=\sum_{i=1}^n p(i)\ln p(i) is 11-strongly convex on Δn\Delta_n with respect to ∥⋅∥1\|\cdot\|_1.

Proof. We proceed in two steps: first we reduce the nn-state inequality to a two-state Bernoulli distribution, and then we prove the scalar Bernoulli case by calculus.

Step 1 (Coarse-graining to two states via the log-sum inequality): Define the subset of coordinates where pp exceeds qq: A:={i∈[n]:p(i)≥q(i)},Ac=[n]∖A={i∈[n]:p(i)<q(i)}. A:=\bigl\{i\in[n]:p(i)\ge q(i)\bigr\}, \qquad A^c=[n]\setminus A=\bigl\{i\in[n]:p(i)<q(i)\bigr\}. Let u:=p(A)=∑i∈Ap(i)∈[0,1]u:=p(A)=\sum_{i\in A}p(i)\in[0,1] and v:=q(A)=∑i∈Aq(i)∈[0,1]v:=q(A)=\sum_{i\in A}q(i)\in[0,1]. As we saw in Chapter 2, the ℓ1\ell_1 distance satisfies ∥p−q∥1=∑i∈A(p(i)−q(i))+∑i∈Ac(q(i)−p(i))=(u−v)+((1−v)−(1−u))=2(u−v). \|p-q\|_1=\sum_{i\in A}\bigl(p(i)-q(i)\bigr)+\sum_{i\in A^c}\bigl(q(i)-p(i)\bigr)=(u-v)+\bigl((1-v)-(1-u)\bigr)=2(u-v). Meanwhile, since t↦tln⁡tt\mapsto t\ln t is convex on (0,∞)(0,\infty), Jensen’s inequality applied to the weights q(i)/q(A)q(i)/q(A) on AA gives ∑i∈Ap(i)ln⁡p(i)q(i)=q(A)∑i∈Aq(i)q(A)(p(i)q(i)ln⁡p(i)q(i))≥q(A)(p(A)q(A)ln⁡p(A)q(A))=uln⁡uv. \sum_{i\in A}p(i)\ln\frac{p(i)}{q(i)} =q(A)\sum_{i\in A}\frac{q(i)}{q(A)}\left(\frac{p(i)}{q(i)}\ln\frac{p(i)}{q(i)}\right) \ge q(A)\left(\frac{p(A)}{q(A)}\ln\frac{p(A)}{q(A)}\right) =u\ln\frac{u}{v}. Applying the same inequality on AcA^c and adding the two parts shows that coarse-graining [n][n] into {A,Ac}\{A,A^c\} can only decrease KL divergence: KL⁡(p∥q)≥uln⁡uv+(1−u)ln⁡1−u1−v=:dKL(u∥v). \operatorname{KL}(p\|q)\ge u\ln\frac{u}{v}+(1-u)\ln\frac{1-u}{1-v}=:d_{\mathrm{KL}}(u\|v).

Step 2 (Scalar calculus for binary KL divergence): Since ∥p−q∥1=2(u−v)\|p-q\|_1=2(u-v), it remains to show that for all 0≤v≤u≤10\le v\le u\le 1, g(u,v):=uln⁡uv+(1−u)ln⁡1−u1−v−2(u−v)2≥0. g(u,v):=u\ln\frac{u}{v}+(1-u)\ln\frac{1-u}{1-v}-2(u-v)^2\ge 0. Fix u∈(0,1)u\in(0,1) and view h(v):=g(u,v)h(v):=g(u,v) as a function of v∈(0,u]v\in(0,u]. At v=uv=u, we have h(u)=0h(u)=0. Differentiating with respect to vv gives h′(v)=−uv+1−u1−v+4(u−v)=(u−v)(4−1v(1−v)). h'(v)=-\frac{u}{v}+\frac{1-u}{1-v}+4(u-v) =(u-v)\left(4-\frac{1}{v(1-v)}\right). Since v(1−v)≤14v(1-v)\le\frac14 for all v∈(0,1)v\in(0,1) by AM–GM, we have 1v(1−v)≥4\frac{1}{v(1-v)}\ge 4, and since v≤uv\le u, it follows that h′(v)≤0h'(v)\le 0 for all v∈(0,u]v\in(0,u]. Therefore h(v)h(v) is nonincreasing on (0,u](0,u], which implies g(u,v)=h(v)≥h(u)=0g(u,v)=h(v)\ge h(u)=0.

2.4 Concrete instantiation of Plotkin–Shmoys–Tardos: minimum-congestion routing

To see how the Plotkin–Shmoys–Tardos oracle framework (§1.11) turns a coupled linear program into a sequence of fast combinatorial subproblems, let us instantiate it on the fractional minimum-congestion multicommodity flow problem.

Problem setup and decomposing the constraints

Let G=(V,E)G=(V,E) be a directed graph with m=∣E∣m=|E| edges, where each edge e∈Ee\in E has capacity ce>0c_e>0. We are given kk commodities (traffic demands) (s1,t1,d1),…,(sk,tk,dk)(s_1,t_1,d_1),\ldots,(s_k,t_k,d_k), where commodity r∈[k]r\in[k] must route dr>0d_r>0 units of flow from source sr∈Vs_r\in V to sink tr∈Vt_r\in V. Given a target congestion factor λ>0\lambda>0, we want to decide whether there exist flows f1,…,fkf_1,\ldots,f_k such that:

  1. Individual flow conservation (P\mathcal P): For each commodity r∈[k]r\in[k], frf_r is a valid sr–trs_r\text{–}t_r flow of value drd_r in which no single commodity sends more than λce\lambda c_e along any edge ee. Setting P:=P1×⋯×Pk\mathcal P:=\mathcal P_1\times\cdots\times\mathcal P_k, where Pr:={fr∈R≥0E: fr routes dr units from sr to tr and fr(e)≤λce ∀e∈E}, \mathcal P_r:=\bigl\{f_r\in\mathbb R_{\ge0}^E:\,f_r\text{ routes }d_r\text{ units from }s_r\text{ to }t_r\text{ and }f_r(e)\le\lambda c_e\ \forall e\in E\bigr\}, linear optimization over P\mathcal P decomposes into kk independent single-commodity flow problems.
  2. Coupling edge-capacity constraints (Ax≥bA x\ge b): Across all kk commodities combined, the total flow on each edge e∈Ee\in E does not exceed λce\lambda c_e. Dividing by λce\lambda c_e to normalize all right-hand sides to 11, we write these mm coupling constraints in the standard form Aef≥beA_e f\ge b_e: 1−∑r=1kfr(e)λce≥0for every e∈E. 1-\sum_{r=1}^k\frac{f_r(e)}{\lambda c_e}\ge 0\qquad\text{for every }e\in E.

What does the average-constraint oracle do?

For any probability distribution p=(p(e))e∈E∈ΔEp=\bigl(p(e)\bigr)_{e\in E}\in\Delta_E over the mm edges, the collapsed average constraint asks whether there exists f=(f1,…,fk)∈Pf=(f_1,\ldots,f_k)\in\mathcal P such that ∑e∈Ep(e)(1−∑r=1kfr(e)λce)≥0⟺∑r=1k∑e∈E(p(e)λce)fr(e)≤1. \sum_{e\in E}p(e)\left(1-\sum_{r=1}^k\frac{f_r(e)}{\lambda c_e}\right)\ge 0 \quad\Longleftrightarrow\quad \sum_{r=1}^k\sum_{e\in E}\left(\frac{p(e)}{\lambda c_e}\right)f_r(e)\le 1. Look at the left-hand side of the second inequality: the kk commodities are now completely decoupled! Assigning each edge e∈Ee\in E the nonnegative length ℓp(e):=p(e)λce\ell_p(e):=\frac{p(e)}{\lambda c_e}, minimizing ∑r=1k∑e∈Eℓp(e)fr(e)\sum_{r=1}^k\sum_{e\in E}\ell_p(e)f_r(e) over f=(f1,…,fk)∈P1×⋯×Pkf=(f_1,\ldots,f_k)\in\mathcal P_1\times\cdots\times\mathcal P_k simply requires solving kk independent single-commodity minimum-cost flow problems—one for each commodity r∈[k]r\in[k]!

What is the width ρ\rho of this oracle? For any f∈Pf\in\mathcal P, each commodity satisfies 0≤fr(e)≤λce0\le f_r(e)\le\lambda c_e, so 0≤∑r=1kfr(e)λce≤k0\le\sum_{r=1}^k\frac{f_r(e)}{\lambda c_e}\le k. Thus the normalized slack ℓt(e)=1−∑r=1kft,r(e)λce\ell_t(e)=1-\sum_{r=1}^k\frac{f_{t,r}(e)}{\lambda c_e} lies in [−(k−1),1]⊆[−k,k][-(k-1),1]\subseteq[-k,k], giving width ρ≤k\rho\le k. By Theorem 11, T=⌈4k2ε−2ln⁡m⌉T=\lceil 4k^2\varepsilon^{-2}\ln m\rceil iterations of Multiplicative Weights—each solving kk independent single-commodity flow problems under edge lengths ℓpt(e)=pt(e)/(λce)\ell_{p_t}(e)=p_t(e)/(\lambda c_e)—either certify that congestion λ\lambda is impossible or produce a multicommodity flow fˉT=1T∑t=1Tft∈P\bar f_T=\frac1T\sum_{t=1}^T f_t\in\mathcal P whose total load on every edge e∈Ee\in E is at most (1+ε)λce(1+\varepsilon)\lambda c_e.

2.5 Online-to-batch conversion for stochastic convex optimization

In §§1.4–1.9 we designed Online Convex Optimization algorithms that guarantee ∑t=1T(ℓt(xt)−ℓt(x∗))≤R(T)\sum_{t=1}^T(\ell_t(x_t)-\ell_t(x^*))\le R(T) against arbitrary sequences ℓ1,…,ℓT\ell_1,\ldots,\ell_T. As an immediate dividend, any such online algorithm can be used directly as a stochastic convex optimization solver when the functions ℓt\ell_t are drawn i.i.d. from an unknown population distribution.

Theorem 13 (Online-to-batch conversion) Let K⊆RnK\subseteq\mathbb R^n be a compact convex set, and suppose we wish to minimize a population risk F(x):=Eζ∼D[f(x;ζ)](x∈K), F(x):=\mathbb E_{\zeta\sim\mathcal D}\bigl[f(x;\zeta)\bigr]\qquad (x\in K), where x↦f(x;ζ)x\mapsto f(x;\zeta) is convex for every ζ\zeta. Given TT i.i.d. samples ζ1,…,ζT∼D\zeta_1,\ldots,\zeta_T\sim\mathcal D, feed the random loss functions ℓt(x):=f(x;ζt)\ell_t(x):=f(x;\zeta_t) sequentially into any Online Convex Optimization algorithm producing iterates x1,…,xT∈Kx_1,\ldots,x_T\in K (where xtx_t depends only on ζ1,…,ζt−1\zeta_1,\ldots,\zeta_{t-1}). Then the averaged output xˉT:=1T∑t=1Txt∈K\bar x_T:=\frac1T\sum_{t=1}^T x_t\in K satisfies, for every x∗∈Kx^*\in K, E[F(xˉT)]−F(x∗)≤E[R(T)]T. \mathbb E\bigl[F(\bar x_T)\bigr]-F(x^*)\le\frac{\mathbb E[R(T)]}{T}. In particular, when ψ\psi is 11-strongly convex w.r.t. ∥⋅∥\|\cdot\| and E[∥∇ℓt(xt)∥∗2]≤G2\mathbb E[\|\nabla\ell_t(x_t)\|_*^2]\le G^2, running Online Mirror Descent (Theorem 6) with step size η=2Bψ(x∗,x1)/(TG2)\eta=\sqrt{2B_\psi(x^*,x_1)/(T G^2)} achieves E[F(xˉT)]−F(x∗)≤G2Bψ(x∗,x1)T\mathbb E[F(\bar x_T)]-F(x^*)\le G\sqrt{\frac{2B_\psi(x^*,x_1)}{T}} for x∗∈arg⁡min⁡x∈KF(x)x^*\in\arg\min_{x\in K}F(x).

Proof. Because FF is convex on KK, Jensen’s inequality gives F(xˉT)=F ⁣(1T∑t=1Txt)≤1T∑t=1TF(xt). F(\bar x_T)=F\!\left(\frac1T\sum_{t=1}^T x_t\right)\le\frac1T\sum_{t=1}^T F(x_t). Now fix any round tt. Conditional on the past samples Ft−1=(ζ1,…,ζt−1)\mathcal F_{t-1}=(\zeta_1,\ldots,\zeta_{t-1}), the iterate xtx_t is deterministic while ζt∼D\zeta_t\sim\mathcal D is drawn fresh and independently of Ft−1\mathcal F_{t-1}. Therefore, E[ℓt(xt)∣Ft−1]=Eζt∼D[f(xt;ζt)∣xt]=F(xt), \mathbb E\bigl[\ell_t(x_t)\mid\mathcal F_{t-1}\bigr]=\mathbb E_{\zeta_t\sim\mathcal D}\bigl[f(x_t;\zeta_t)\mid x_t\bigr]=F(x_t), and similarly E[ℓt(x∗)]=F(x∗)\mathbb E[\ell_t(x^*)]=F(x^*) for any fixed x∗∈Kx^*\in K. Taking full expectations and applying Reg⁡T(x∗)=∑t=1T(ℓt(xt)−ℓt(x∗))≤R(T)\operatorname{Reg}_T(x^*)=\sum_{t=1}^T(\ell_t(x_t)-\ell_t(x^*))\le R(T) yields E[F(xˉT)]−F(x∗)≤1T∑t=1TE[F(xt)−F(x∗)]=1TE ⁣[∑t=1T(ℓt(xt)−ℓt(x∗))]=E[Reg⁡T(x∗)]T≤E[R(T)]T. \mathbb E\bigl[F(\bar x_T)\bigr]-F(x^*) \le\frac1T\sum_{t=1}^T\mathbb E\bigl[F(x_t)-F(x^*)\bigr] =\frac1T\mathbb E\!\left[\sum_{t=1}^T\bigl(\ell_t(x_t)-\ell_t(x^*)\bigr)\right] =\frac{\mathbb E\bigl[\operatorname{Reg}_T(x^*)\bigr]}{T} \le\frac{\mathbb E[R(T)]}{T}.

III. Further Topics

The following topics connect the online learning and regularization toolkit to broader themes in optimization, machine learning, and semidefinite relaxations. Each entry supplies background, main results, and references for independent exploration.

3.1 Lazy versus greedy projection: when FTRL and Mirror Descent diverge

Background. In §1.8 we saw that Online Mirror Descent applies a greedy Bregman projection xs+1=ΠKψ(xs+1′)x_{s+1}=\Pi_K^\psi(x'_{s+1}) at every round. On an unconstrained space V=RnV=\mathbb R^n (or on the simplex Δn\Delta_n with negative entropy, whose steep gradient at the boundary keeps iterates strictly in the interior), Online Mirror Descent initialized at x0∈arg⁡min⁡x∈Kψ(x)x_0\in\arg\min_{x\in K}\psi(x) with constant step size η\eta produces the exact same sequence of iterates as Follow the Regularized Leader (FTRL): xs+1FTRL:=arg⁡min⁡x∈K{η∑τ=0s⟨∇ℓτ(xτ),x⟩+ψ(x)}. x_{s+1}^{\mathrm{FTRL}}:=\arg\min_{x\in K}\left\{\eta\sum_{\tau=0}^s\langle\nabla\ell_\tau(x_\tau),x\rangle+\psi(x)\right\}. What happens on a general constrained convex body KK (such as the box [−1,1]n[-1,1]^n or a clipped simplex) when the unconstrained mirror step ∇ψ(xs+1′)=∇ψ(xs)−η∇ℓs(xs)\nabla\psi(x'_{s+1})=\nabla\psi(x_s)-\eta\nabla\ell_s(x_s) leaves KK and hits the boundary?

Main distinction. On a general constrained set KK, the two algorithms handle boundary projections differently:

With a fixed learning rate η\eta, both algorithms satisfy the same O(T)O(\sqrt T) regret bound. Under time-varying step sizes ηs=1/s\eta_s=1/\sqrt s, however, greedy OMD can suffer linear regret on specific adversarial sequences whenever the Bregman diameter sup⁡x,y∈KBψ(x,y)\sup_{x,y\in K}B_\psi(x,y) is unbounded (even on a compact domain like Δn\Delta_n), whereas FTRL / Nesterov Dual Averaging maintains O(T)O(\sqrt T) regret under bounded gradients and bounded regularizer range because its dual state ys+1y_{s+1} never forgets past gradients clipped by boundary projections.

References. Yurii Nesterov, Primal-Dual Subgradient Methods for Convex Problems, Mathematical Programming 120 (2009), 221–259; Francesco Orabona, A Modern Introduction to Online Learning (2019), Chapters 6–7; Elad Hazan, Introduction to Online Convex Optimization, Chapter 5.

3.2 Boosting weak learners via Multiplicative Weights: AdaBoost

Background. Given mm labeled training examples (z1,y1),…,(zm,ym)∈Z×{−1,+1}(z_1,y_1),\ldots,(z_m,y_m)\in\mathcal Z\times\{-1,+1\}, suppose we have access to a weak learning oracle that, for any probability distribution p∈Δmp\in\Delta_m over the mm training examples, can find a hypothesis h:Z→{−1,+1}h:\mathcal Z\to\{-1,+1\} with classification accuracy at least 12+γ\frac12+\gamma (where γ>0\gamma>0 is a small edge over random guessing). Can we combine weak hypotheses into a single composite classifier H(z)=sgn⁡(∑t=1Tαtht(z))H(z)=\operatorname{sgn}\bigl(\sum_{t=1}^T\alpha_t h_t(z)\bigr) that achieves zero training error on all mm examples?

Main result. The fixed-rate Multiplicative Weights boosting prototype underlying Freund and Schapire’s AdaBoost algorithm (which refines the vote weights to αt=12ln⁡1−εtεt\alpha_t=\frac12\ln\frac{1-\varepsilon_t}{\varepsilon_t}) is a direct instance of the MWU game (§§1.9–1.11):

By the Multiplicative Weights guarantee (Theorem 7), if an unweighted majority vote (αt=1\alpha_t=1) of h1,…,hTh_1,\ldots,h_T still errs on any training example i∗∈[m]i^*\in[m], that example has cumulative loss LT(i∗)≤T/2L_T(i^*)\le T/2, whereas the weak learner guarantees ⟨pt,ℓt⟩≥12+γ\langle p_t,\ell_t\rangle\ge\frac12+\gamma at every round. The regret bound γT≤(Tln⁡m)/2\gamma T\le\sqrt{(T\ln m)/2} immediately implies that the training error of the combined classifier drops to 00 as soon as T>ln⁡m2γ2T>\frac{\ln m}{2\gamma^2}.

References. Yoav Freund and Robert E. Schapire, A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting, Journal of Computer and System Sciences 55 (1997), 119–139; Arora, Hazan, and Kale, The Multiplicative Weights Update Method, §3.2.

3.3 Matrix Multiplicative Weights and semidefinite optimization

Background. In §§1.9–1.11 our decision variable was a probability vector xt∈Δnx_t\in\Delta_n, which can be viewed as a nonnegative diagonal matrix diag⁡(xt)\operatorname{diag}(x_t) of trace 11. In Chapter 6 we will study semidefinite programs (SDPs), where the optimization variable is a symmetric positive semidefinite matrix X⪰0X\succeq 0. Can we generalize Multiplicative Weights and negative-entropy Mirror Descent from the probability simplex Δn\Delta_n to the spectraplex Sn:={X∈Rn×n:X=X⊤, X⪰0, Tr⁡(X)=1} \mathcal S_n:=\bigl\{X\in\mathbb R^{n\times n}:X=X^\top,\ X\succeq 0,\ \operatorname{Tr}(X)=1\bigr\} of density matrices?

Main result. Replacing the coordinatewise negative entropy ∑i=1nxiln⁡xi\sum_{i=1}^n x_i\ln x_i by the negative von Neumann quantum entropy ψ(X):=Tr⁡(Xln⁡X)\psi(X):=\operatorname{Tr}(X\ln X) (where Xln⁡X=∑i=1nλi(X)ln⁡λi(X) vivi⊤X\ln X=\sum_{i=1}^n\lambda_i(X)\ln\lambda_i(X)\,v_i v_i^\top in the eigenbasis of XX) yields the Matrix Multiplicative Weights (MMW) algorithm: given symmetric loss matrices M1,…,MT∈Rn×nM_1,\ldots,M_T\in\mathbb R^{n\times n} with −I⪯Mt⪯I-I\preceq M_t\preceq I and learning rate 0<η≤10<\eta\le 1, update Wt+1:=exp⁡ ⁣(−η∑s=1tMs),Xt+1:=Wt+1Tr⁡(Wt+1)∈Sn. W_{t+1}:=\exp\!\left(-\eta\sum_{s=1}^t M_s\right), \qquad X_{t+1}:=\frac{W_{t+1}}{\operatorname{Tr}(W_{t+1})}\in\mathcal S_n. Because matrices do not commute (eA+B≠eAeBe^{A+B}\ne e^A e^B), the one-step potential analysis uses the Golden–Thompson inequality Tr⁡(eA+B)≤Tr⁡(eAeB)\operatorname{Tr}(e^{A+B})\le\operatorname{Tr}(e^A e^B) to establish the matrix regret bound ∑t=1TTr⁡(XtMt)−λmin⁡ ⁣(∑t=1TMt)≤ln⁡nη+η∑t=1TTr⁡(XtMt2). \sum_{t=1}^T\operatorname{Tr}(X_t M_t)-\lambda_{\min}\!\left(\sum_{t=1}^T M_t\right)\le\frac{\ln n}{\eta}+\eta\sum_{t=1}^T\operatorname{Tr}(X_t M_t^2). Just as scalar MWU solves packing/covering LPs in §1.11, Matrix Multiplicative Weights solves semidefinite programs (such as Max-Cut, balanced separator, and spectral sparsification) in O(ε−2ln⁡n)O(\varepsilon^{-2}\ln n) oracle iterations.

References. Koji Tsuda, Gunnar Rätsch, and Manfred K. Warmuth, Matrix Exponentiated Gradient Updates for On-line Learning and Bregman Projection, JMLR 6 (2005), 995–1018; Sanjeev Arora and Satyen Kale, A Combinatorial, Primal-Dual Approach to Semidefinite Programs, JACM 63 (2016), Article 12.

3.4 Tsallis-INF and best-of-both-worlds bandits

Background. In §1.9 we proved that running Online Stochastic Mirror Descent with negative entropy ψ(x)=∑i=1nxiln⁡xi\psi(x)=\sum_{i=1}^n x_i\ln x_i on importance-weighted bandit losses (EXP3) achieves R(T)=O(nTln⁡n)R(T)=O(\sqrt{nT\ln n}). Notice two open questions raised by this bound:

  1. Can we remove the extra ln⁡n\sqrt{\ln n} factor and achieve Θ(nT)\Theta(\sqrt{nT}) minimax regret in adversarial bandits?
  2. In Chapter 1 we saw that when losses are drawn i.i.d. with suboptimality gaps Δi=μi−μi∗>0\Delta_i=\mu_i-\mu_{i^*}>0, UCB achieves logarithmic regret O(∑i:Δi>0ln⁡TΔi)O\bigl(\sum_{i:\Delta_i>0}\frac{\ln T}{\Delta_i}\bigr). Can a single algorithm achieve O(nT)O(\sqrt{nT}) against an adversary while automatically adapting to achieve O(∑iln⁡TΔi)O\bigl(\sum_i\frac{\ln T}{\Delta_i}\bigr) whenever the environment happens to be stochastic—without knowing in advance which world it is in?

Main result. Both questions are answered simultaneously by Tsallis-INF, which runs FTRL on importance-weighted losses with a scheduled learning rate ηt\eta_t and replaces negative Shannon entropy by the negative 1/21/2-Tsallis entropy ψ1/2(x):=−2∑i=1nxi(x∈Δn). \psi_{1/2}(x):=-2\sum_{i=1}^n\sqrt{x_i}\qquad (x\in\Delta_n). Because ∇2ψ1/2(x)=diag⁡(12x1−3/2,…,12xn−3/2)\nabla^2\psi_{1/2}(x)=\operatorname{diag}\bigl(\frac12 x_1^{-3/2},\ldots,\frac12 x_n^{-3/2}\bigr) has curvature xi−3/2x_i^{-3/2} instead of negative entropy’s xi−1x_i^{-1}, the local inverse-Hessian norm 2∑i=1nxt(i)3/2ℓ^t(i)22\sum_{i=1}^n x_t(i)^{3/2}\hat\ell_t(i)^2 has conditional expectation 2∑i=1nxt(i) ℓt(i)2≤2n2\sum_{i=1}^n\sqrt{x_t(i)}\,\ell_t(i)^2\le 2\sqrt n (by Cauchy–Schwarz ∑ixt(i)≤n\sum_i\sqrt{x_t(i)}\le\sqrt n), while the Bregman radius max⁡x∗∈ΔnBψ1/2(x∗,1/n)≤2n\max_{x^*\in\Delta_n} B_{\psi_{1/2}}(x^*,\mathbf 1/n)\le 2\sqrt n also scales as n\sqrt n. Balancing these two n\sqrt n factors in Tsallis-INF (Audibert–Bubeck 2009; Zimmert–Seldin 2021) yields:

References. Jean-Yves Audibert and Sébastien Bubeck, Minimax Policies for Adversarial and Stochastic Bandits, COLT 2009, 217–226; Julian Zimmert and Yevgeny Seldin, Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Bandits, JMLR 22 (2021), 1–49; Tor Lattimore and Csaba Szepesvári, Bandit Algorithms, Chapters 11 and 28.

References and source acknowledgments. The progression from the Online Number Guessing Game and Follow the Leader (§§1.1–1.2) to offline and online Projected Gradient Descent (§§1.3–1.4), reductions of Experts, Online Shortest Paths, and Euclidean OSGD Bandits (§§1.5–1.6), Online Mirror Descent (§§1.7–1.8) and its continuous-time Bregman discretization proof (§2.1), and the first-principles derivation of Multiplicative Weights (Hedge) and EXP3 (§1.9) is adapted directly from Chihao Zhang’s CS3958 Advanced Algorithms lecture notes [1], which in turn draw on Francesco Orabona [2] and Yin Tat Lee and Santosh S. Vempala’s notes on Convex Optimization [3]. The game-theoretic and algorithmic applications of Multiplicative Weights to zero-sum minimax duality (§1.10) and the Plotkin–Shmoys–Tardos LP feasibility framework (§1.11, §2.4) follow Sanjeev Arora, Elad Hazan, and Satyen Kale [4] and Plotkin, Shmoys, and Tardos [9]. Further developments of Mirror Descent, Fenchel conjugation, and bandit optimization draw on Nemirovski and Yudin [5], Hazan [6], Bubeck [7], Auer et al. [8], and Lattimore and Szepesvári [10].

  1. Chihao Zhang, CS3958 Advanced Algorithms: Lectures 6–9 (Online Learning, Online Gradient Descent, Reductions, and Online Mirror Descent), Shanghai Jiao Tong University, Fall 2021 & Fall 2022.
  2. Francesco Orabona, A Modern Introduction to Online Learning, arXiv:1912.13213 (2019).
  3. Yin Tat Lee and Santosh S. Vempala, Theory and Algorithms for Continuous Optimization (Lecture notes on Mirror Descent and continuous-time potentials).
  4. Sanjeev Arora, Elad Hazan, and Satyen Kale, The Multiplicative Weights Update Method: a Meta-Algorithm and Applications, Theory of Computing 8 (2012), 121–164.
  5. Arkadi S. Nemirovski and David B. Yudin, Problem Complexity and Method Efficiency in Optimization, Wiley, 1983.
  6. Elad Hazan, Introduction to Online Convex Optimization, 2nd edition, MIT Press, 2022.
  7. Sébastien Bubeck, Convex Optimization: Algorithms and Complexity, Foundations and Trends in Machine Learning 8 (2015), 231–357.
  8. Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire, The Nonstochastic Multiarmed Bandit Problem, SIAM Journal on Computing 32 (2002), 48–77.
  9. Serge A. Plotkin, David B. Shmoys, and Éva Tardos, Fast Approximation Algorithms for Fractional Packing and Covering Problems, Mathematics of Operations Research 20 (1995), 257–301.
  10. Tor Lattimore and Csaba Szepesvári, Bandit Algorithms, Cambridge University Press, 2020.