Dear Students,
I uploaded a text, which you can find in the "course materials" section.
The task I would like to assign you is the following:
you should estimate the Shannon entropy H(X) of the text, assuming that each character is an independent draw from an ideal "character distribution". Nothing more than that. Please include punctuation characters ",", ";", "." , space " " and newline "\n" in the list of characters, but consider lowercase and uppercase ("a" and ""A, "b" and "B") as one and the same character.
Of course, the characters are not independent draws (the above assumption does not hold). In fact, adjacent characters (more in general, neighboring characters) are correlated. To verify this, try to estimate the mutual information I(X_n;X_{n+1})=H(X_n)+H(X_{n+1})-H(X_n,X_{n+1}) where X_n is one character and X_{n+1} the next character. [assume that each pair of successive characters is an independent draw from a bivariate distribution; again, this assumption does not rigorously hold, but is works for a starter].
For any question related to this exercise, please use the course channel, do
All the best,
Michele