Information and Uncertainty are quantities that describe any decision making process - like selecting one or more objects from a set of objects. Both these ideas and much more, related to communication theory, were introduced by Calude E. Shannon in his seminal paper “A mathematical theory of communication,” Bell. Syst. Tech. J. (1948) [pdf]. This post introduces the concepts of Information and Uncertainty (or the Information Entropy) without equations. (The next post will do it with equations!)



For example, consider the case of one of the two boxes shown below, containing a ball in it. You are asked to find which box the ball is by asking me questions. I will answer only with an ‘yes’ or a ‘no’ much like the Catch-20 game we play in our college cultural.


IE1


Figure 1


Before you ask any question to me, you have no clue as to which box contains the ball. In other words, you are UNCERTAIN and have no INFORMATION about the system so that you can DECIDE on the correct answer of which box contains the ball. Once you ask questions and receive answers - in this case, one is enough - you gain INFORMATION and are no more UNCERTAIN, for making your decision. This implies that the decision that has to be made by you between the two boxes will depend on some reason and not purely a random event.