Comments on Hoeffding inequality

Comments on Hoeffding inequality

by JUAN BAEZA RUIZ-HENESTROSA -
Number of replies: 0

Here, I just share some notes I found on internet about Hoeffding inequality. There, you can find a very detailed prove of the inequality and its general formulation. I found them really useful mainly because in its general formulation you can see one fundametal hypothesis for the inequality to hold: the Z_i random variables of the theorem must be INDEPENDENT among them. This is the hypothesis that holds when we are testing a given hypothesis from the hypothesis space (H) but that is violated when we are choosing/selecting a specific hypothesis from H (i.e. when we are learning!!). 

Also, if you think about it, it also explain why the more the bad events overlap the closer we get to the Hoeffding Inequality bound, because the more the more they overlap, the closer the Z_i random variables from the theorem formulation are from being indepent.  (it is more difficult to differenciate among the hypothesis of H given the in-sample errors!).

I hope this might help anyone that is still struggling to fully understand why we cannot directly apply the Hoeffding inequality when learning and we need to use the Union Bound instead.