Randomness can be super powerful in designing algorithms. It can help an algorithm run faster and avoid worst-case inputs. Concentration inequalities are a key tool to both analyze and design randomized algorithms. We develop basic concentration tools and study two concrete examples in this section. Morris’s algorithm counts a long stream using a small random counter. Explore-Then-Commit uses samples to decide which action to take in a multi-armed bandit problem. The same concentration tools control the accuracy of the counter and the reliability of the decision.

I. Course Content

1.1 Markov and Chebyshev inequalities

We first recall Markov’s inequality and Chebyshev’s inequality we met in the probability theory course.

In order to motivate the study of concentration inequalities, let’s first look at the streaming model.

1.2 The streaming model

Suppose we have a router with limited memory, but need to solve some computational tasks with large input data such as monitoring the IDs of devices visiting it. The following questions are usually asked.

• How many numbers are in a given data stream?
• How many distinct numbers?
• What is the most frequent number?

In order to study these problems systematically, we need to formally define the streaming model. In the streaming model, the input is a sequence $\sigma=\langle a_1,a_2,\ldots,a_m\rangle$ where each $a_i\in[n]$. We should notice that the data arrive one by one as suggested by the word “streaming” in the name. We now focus on the basic problem: How many numbers are in the stream (what is $m$)?

Clearly we can maintain a counter $k$, and whenever a number $a_i$ arrives, increase $k$ by one. To represent every possible count from $0$ to $m$, we need $\lceil\log_2(m+1)\rceil$ bits of memory.

Can we design a more clever algorithm with only $o(\log m)$ memory? It turns out that computing the exact answer is impossible with fewer than $\lceil\log_2(m+1)\rceil$ bits. The reason is as follows: suppose an algorithm uses fewer bits, so it has fewer than $m+1$ memory states. Denote by $\mathcal M(i)$ the memory state of the algorithm after an input $\sigma$ of length $i$. Then there exist distinct $i,j\in\{0,1,\ldots,m\}$ such that $\mathcal M(i)=\mathcal M(j)$, and the algorithm cannot output the correct length for both inputs.

Even though we cannot get a better algorithm for the exact answer, it is possible to save a lot of memory if approximation is allowed. That is, for every $\varepsilon>0$, the algorithm computes a number $\widehat m$ such that $$1-\varepsilon\le\frac{\widehat m}{m}\le1+\varepsilon$$ with high probability.

Morris’s algorithm

Morris’s algorithm is presented as follows.

This is a randomized algorithm with approximately $O(\log\log m)$ bits of memory. We first look at the expectation of its output.

It is now clear that Morris’s algorithm is an unbiased estimator for $m$. However, for a practical randomized algorithm, we further require its output to concentrate around the expectation. That is, we want to establish a concentration inequality of the form $$\mathbb P(|\widehat m-m|\ge\varepsilon m) \le\delta$$ for $\varepsilon,delta>0$. It is natural to see that for fixed $\varepsilon$, the smaller $\delta$ is, the better the algorithm is.

To use Chebyshev’s inequality to analyze the concentration of Morris’s algorithm, we have to compute the variance of $\widehat m$.

With the above lemma, we can compute the variance as follows: $$\operatorname{Var}(\widehat m) =\mathbb E[(2^{X_m}-1)^2]-m^2 =\frac{m(m-1)}2\le\frac{m^2}{2}.$$ Applying Chebyshev’s inequality, we obtain that for every $\varepsilon>0$, $$\mathbb P(|\widehat m-m|\ge\varepsilon m) \le\frac{1}{2\varepsilon^2}.$$ However, we observe that as $\varepsilon$ becomes smaller, the above bound is not useful. Thus, it is necessary to improve the concentration of the algorithm. We now introduce two common tricks to achieve this.

The averaging trick

Chebyshev’s inequality tells us that we can improve the concentration by reducing the variance. Let’s first review some properties of variances. Let $X$ be a random variable. We have $$\operatorname{Var}(aX)=a^2\operatorname{Var}(X)$$ for any constant $a$. For any two independent random variables $X$ and $Y$, we have $$\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y).$$

We can design a new algorithm by independently running Morris’s algorithm $t$ times in parallel. Denote the corresponding outputs by $\widehat m_1,\ldots,\widehat m_t$. The final output is $$\widehat m^*:=\frac{\sum_{i=1}^t\widehat m_i}{t}.$$ By the above two properties, we have $\operatorname{Var}(\widehat m^*)=\operatorname{Var}(\widehat m_1)/t$. We can apply Chebyshev’s inequality to $\widehat m^*$ and obtain $$\mathbb P(|\widehat m^*-m|\ge\varepsilon m) \le\frac{1}{2t\varepsilon^2}.$$ For $t\ge1/(2\varepsilon^2\delta)$, we have $$\mathbb P(|\widehat m^*-m|\ge\varepsilon m)\le\delta.$$

The new algorithm uses approximately $$O\!\left(\frac{\log\log m}{\varepsilon^2\delta}\right)$$ bits of memory. It shows a trade-off between the accuracy of the randomized algorithm and the consumption of memory space. We can further improve the dependence on $\delta$ using the Chernoff bound below.

1.3 Chernoff bounds and success amplification

Like Chebyshev’s inequality, if we choose $f(x)=e^{\alpha x}$ for $\alpha>0$ and apply Markov’s inequality to $f(X)$, the bound amounts to bounding $\mathbb E[e^{\alpha X}]$, which is the moment generating function of $X$. In case $\mathbb E[e^{\alpha X}]$ can be well bounded, we obtain sharp concentration bounds.

The following form of the Chernoff bound is more convenient to use (but weaker).

The median trick

We can further boost the performance of Morris’s algorithm using the median trick. We choose $t=\lceil3/(2\varepsilon^2)\rceil$ in the algorithm introduced in the averaging trick and independently run it $s$ times in parallel. Denote the outputs by $\widehat m_1^*,\widehat m_2^*,\ldots,\widehat m_s^*$. It holds that for every $i=1,\ldots,s$, $$\mathbb P(|\widehat m_i^*-m|\ge\varepsilon m)\le\frac13.$$ At last, we output the median $\widehat m^{**}$ of $\widehat m_1^*,\ldots,\widehat m_s^*$.

Then we can apply the Chernoff bound to analyze the result obtained by the median trick. For every $i=1,\ldots,s$, let $Y_i$ be the indicator of the good event $$|\widehat m_i^*-m|<\varepsilon m.$$ Then $Y:=\sum_{i=1}^sY_i$ satisfies $\mathbb E Y\ge2s/3$. If the median $\widehat m^{**}$ is bad, then at least half of the $\widehat m_i^*$ are bad. Equivalently, $Y\le s/2$. By the Chernoff bound, $$\mathbb P\!\left(|Y-\mathbb E Y|\ge\frac{s}{6}\right) \le2\exp\!\left(-\frac{s}{72}\right).$$ Therefore, for $t=O(1/\varepsilon^2)$ and $s=O(\log(1/\delta))$, we have $$\mathbb P(|\widehat m^{**}-m|\ge\varepsilon m)\le\delta.$$ This new algorithm uses $$O\!\left(\frac1{\varepsilon^2}\log\frac1\delta\cdot\log\log m\right)$$ bits of memory.

1.4 Hoeffding’s inequality

One annoying restriction of the Chernoff bound is that each $X_i$ needs to be a Bernoulli random variable. Hoeffding’s inequality generalizes the Chernoff bound by allowing $X_i$ to follow any distribution, provided its value is almost surely bounded.

We learnt from the proof of the Chernoff bound that the key to establishing concentration inequalities of this form is to obtain a nice upper bound on the moment generating function. Therefore, the following Hoeffding lemma will be the main technical ingredient to prove the inequality.

Armed with Hoeffding’s lemma, it is routine to prove Hoeffding’s inequality.

1.5 Multi-armed bandits: Explore-Then-Commit

Suppose there is a $k$-arm bandit, and the reward of each arm follows some distribution $f_i$ supported on $[0,1]$ with mean $\mu_i$; rewards are independent across arms and rounds. We assume without loss of generality that $\mu_1\ge\mu_2\ge\cdots\ge\mu_k$. Now suppose you can pull the bandit for $T$ rounds and the goal is to obtain maximum reward in expectation. If we know $\mu_1,\ldots,\mu_k$, the optimal strategy is to pull arm $1$ for $T$ times, and the expected reward is $T\mu_1$. However, in case we do not know the distributions, we have to design some strategy to explore the bandit first.

Denote by $a_t$ the arm pulled in round $t$, and thus the reward in the $t$th round satisfies $X_t\sim f_{a_t}$. The regret of a strategy is defined as the gap between $T\mu_1$ and the expected rewards of the strategy in $T$ rounds, namely the regret of not always choosing the first arm: $$R(T):=T\mu_1-\mathbb E\!\left[\sum_{t=1}^TX_t\right]\ge0.$$

For every $i\in[k]$, denote $\Delta_i:=\mu_1-\mu_i\ge0$ as the gap between the reward of the $i$th arm and the optimal arm. The naive strategy that pulls each arm equally often is bad. When $k$ divides $T$, its regret is $$R(T)=\frac{\sum_{i=1}^k\Delta_i}{k}\,T,$$ which is linear in $T$. We consider a strategy/algorithm good if $\lim_{T\to\infty}R(T)/T=0$, or equivalently $R(T)=o(T)$.

We also write $R_i(T):=\Delta_i\mathbb E[n_i(T)]$ for every $i\in[k]$, and then $R(T)=\sum_{i=1}^kR_i(T)$.

The algorithm and the trade-off

To get small regret, our strategy should identify the best arm as soon as possible. The most straightforward way to find the best arm is to try each arm a few times and pick the one with the best empirical reward. The Explore-Then-Commit algorithm implements this idea: pull every arm $i$ for $L$ times (so $kL$ times in total for exploration), and calculate $\widehat\mu_i$ (the average reward gained in those $L$ times). After this, always pull an arm $J$ with the greatest $\widehat\mu_i$. We assume $kL\le T$ and use any fixed rule to break ties. We can write its regret as

$$R(T) =L\sum_{i=1}^k\Delta_i +(T-kL)\mathbb E[\Delta_J].$$ Moreover, $$\mathbb E[\Delta_J] =\sum_{i=2}^k\Delta_i\mathbb P(J=i).$$ When $i\ne1$, $$\mathbb P(J=i)\le\mathbb P(\widehat\mu_i\ge\widehat\mu_1).$$

We bound the above probability by concentration inequalities. To this end, let $X_j$ be the $j$th reward from $f_i$, and let $Y_j$ be the $j$th reward from $f_1$. Let $Z_j=X_j-Y_j\in[-1,1]$; then $\mathbb E Z_j=-\Delta_i\le0$. Let $Z=\sum_{j=1}^LZ_j$; then $\mathbb E Z=-L\Delta_i$. By Hoeffding’s inequality, $$\begin{aligned} \mathbb P(\widehat\mu_i\ge\widehat\mu_1) &=\mathbb P(Z\ge0)\\ &=\mathbb P(Z-\mathbb E Z\ge L\Delta_i)\\ &\le\exp\!\left(-\frac{2(L\Delta_i)^2}{\sum_{j=1}^L2^2}\right)\\ &=\exp\!\left(-\frac{L\Delta_i^2}{2}\right). \end{aligned}$$ Therefore, $$\begin{aligned} R(T) &\le L\sum_{i=1}^k\Delta_i +T\sum_{i=2}^k\Delta_i e^{-L\Delta_i^2/2}\\ &\le\sum_{i=1}^k \left(L+T\Delta_i e^{-L\Delta_i^2/2}\right). \end{aligned}$$

A bound without knowing the gaps

To further upper bound $R(T)$, define $$g(L,\Delta_i) :=L+T\Delta_i\exp\!\left(-\frac{L\Delta_i^2}{2}\right).$$ We would like to determine $L$ minimizing the upper bound of $R(T)$ among all possible $\Delta_i$, i.e., $\min_L\max_{\Delta_i}R(T)$. First we calculate $\max_{\Delta_i}g(L,\Delta_i)$: $$\frac{\partial g(L,\Delta_i)}{\partial\Delta_i} =T(1-L\Delta_i^2)\exp\!\left(-\frac{L\Delta_i^2}{2}\right).$$ We have $\partial g/\partial\Delta_i>0$ when $0\le\Delta_i<1/\sqrt L$, and $\partial g/\partial\Delta_i<0$ when $1\ge\Delta_i>1/\sqrt L$. Thus, for all $L>1$, $$g(L,\Delta_i) \le g\!\left(L,\frac1{\sqrt L}\right) =L+\frac{Te^{-1/2}}{\sqrt L}.$$

Finally, assuming $kL\le T$ and setting $L=\Theta(T^{2/3})$, we have $$R(T) \le\sum_{i=1}^k\left(L+\frac{Te^{-1/2}}{\sqrt L}\right) =O(kT^{2/3}).$$

The Explore-Then-Commit algorithm enjoys sublinear regret for fixed $k$, which is good, but still suboptimal. The main disadvantage is that it treats all arms equally in the exploration step and pulls each of them for a fixed $L$ times regardless of the rewards already obtained.

II. Reading Contents

The classroom arguments leave two natural questions. Can we obtain concentration when observations are dependent, or when variables are not bounded? And can we use concentration to decide how many samples to collect? The following sections develop these questions without changing the basic exponential-moment strategy.

2.1 Concentration with martingales

Our concentration proofs for sums have used mutual independence to factor an MGF. When new observations depend on the past, factorization is no longer available. Conditional expectation gives a replacement: we bound the next factor after fixing everything observed so far, and then work backward.

A fair game and its information

Imagine a fair game in which the size of the next bet may depend on previous wins and losses. The increments need not be independent. What matters is that their conditional expected value is zero.

For example, if $X_1,X_2,\ldots$ are independent and integrable, then $Z_t=\sum_{i=1}^t(X_i-\mathbb EX_i)$ is a martingale with $Z_0=0$. We have also already met a dependent example: for Morris’s counter, $2^{X_t}-1-t$ is a martingale, by the conditional expectation calculation in Part I. Its increments are not uniformly small, so it does not directly give a useful bounded-increment estimate.

Revealing a function one coordinate at a time

Let $W=f(X_1,\ldots,X_n)$ be integrable. Reveal the inputs successively and keep track of the current prediction of $W$: $$Z_k=\mathbb E[W\mid X_1,\ldots,X_k], \qquad 0\le k\le n.$$ Here $Z_0=\mathbb EW$ and $Z_n=W$. The tower property gives $$\mathbb E[Z_k\mid\mathcal F_{k-1}] =\mathbb E[W\mid\mathcal F_{k-1}]=Z_{k-1},$$ where $\mathcal F_k=\sigma(X_1,\ldots,X_k)$. This is the Doob martingale of $W$. The construction itself does not require independent inputs.

This converts a deviation of $f$ into a sum of changes in conditional predictions. To prove concentration, we need to control how much one new observation can change that prediction.

Notice the distinction between an absolute increment bound and a conditional range length. A variable in $[-1,1]$ has range length $2$, not $1$. Tracking the conditional range is what gives the sharp constant in the next application.

Bounded differences: McDiarmid’s inequality

Suppose $X_1,\ldots,X_n$ are independent. A function $f$ has coordinate bounded differences $c_1,\ldots,c_n$ if changing just coordinate $i$ changes its value by at most $c_i$. Writing $x^{(i\leftarrow y)}$ for $x$ with coordinate $i$ replaced by $y$, this is $$|f(x)-f(x^{(i\leftarrow y)})|\le c_i.$$ The condition is imposed on the product of the input spaces. It describes sensitivity to one input, rather than a Euclidean Lipschitz constant.

2.2 Sub-Gaussian random variables

Recall the calculation underlying Hoeffding. For a centered variable, a quadratic upper bound on its log-MGF led to a Gaussian-shaped tail. Boundedness was one way to obtain that upper bound; it need not be the only way.

For an integrable $X$, write $$\psi_X(\lambda)=\log\mathbb E e^{\lambda(X-\mathbb EX)},$$ allowing the value $+\infty$ if the MGF diverges.

Exponential Markov, optimized at $\lambda=t/\sigma^2$, immediately gives, when $\sigma^2>0$, $$\mathbb P(|X-\mathbb EX|\ge t) \le2e^{-t^2/(2\sigma^2)}.$$ For proxy zero, $X$ is constant almost surely.