INFORMATION THEORY:– Entropy, Information rate, classification of codes, Kraft

McMillan inequality, Source coding theorem, Shannon-Fano coding, Huffman coding, Extended
Huffman coding - Joint and conditional entropies, Mutual information - Discrete memory less
channels – BSC, BEC – Channel capacity, Shannon limit

***************************************************************************************************

Information is the source of a communication system, whether it is analog or digital.

Information theory is a mathematical approach to the study of coding of information along
with the quantification, storage, and communication of information.

Conditions of Occurrence of Events

If we consider an event, there are three conditions of occurrence.

 If the event has not occurred, there is a condition of uncertainty.
 If the event has just occurred, there is a condition of surprise.
 If the event has occurred, a time back, there is a condition of having some information.
These three events occur at different times. The differences in these conditions help us gain
knowledge on the probabilities of the occurrence of events.
Definition of Information:
ENTROPY:
Entropy:When we observe the possibilities of the occurrence of an event, how surprising or

uncertain it would be, it means that we are trying to have an idea on the average content of the
information from the source of the event.

Entropy can be defined as a measure of the average information content per source symbol.

Where pi is the probability of the occurrence of character number i from a given stream of
characters and b is the base of the algorithm used. Hence, this is also called as Shannon’s
Entropy.
Conditional Entropy: The amount of uncertainty remaining about the channel input after
observing the channel output, is called as Conditional Entropy.

It is denoted by
Mutual Information :

Let us consider a channel whose output is Y and input is X

Let the entropy for prior uncertainty be X = Hx

Thisisassumedbeforetheinputisapplied
To know about the uncertainty of the output, after the input is applied, let us consider
Conditional Entropy, given that Y = yk

This is a random variable for

With probabilities p (yo)….p(yk-1)

respectively
Now, considering both the uncertainty
conditions beforeandafterapplyingtheinputsbeforeandafterapplyingtheinputs,

we come to know that the difference, i.e. H(x)−H(x∣y)must represent the uncertainty about the channel
input that is resolved by observing the channel output.

This is called as the Mutual Information of the channel.

Denoting the Mutual Information as I(x;y),we can write the whole thing in an equation, as follows

Hence, this is the equational representation of Mutual Information.

Discrete Memoryless Channel (DMC):
Fig1: Noiseless Channel
Fig2: Deterministic Channel
Fig2: Noiseless Channel
Next Page 72 ?????

SHANNON- FANO CODING:

 MCQ Test

1. Self information should be

a) Negative
b) Positive
c) Positive & Negative
d) None of the mentioned Ans: b

2. The unit of average mutual information is

a) Bytes per symbol
b) Bytes
c) Bits per symbol
d) Bits Ans: d

3. In discrete memoryless source, the current letter produced by a source is

statistically independent of _____
a. Past output
b. Future output
c. Both a and b
d. None of the aboveAns: c
4. When the base of the logarithm is 2, then the unit of measure of information is
a) Bits
b) Bytes
c) Nats
d) None of the mentioned Ans: a
5. The self information of random variable is
a) 0
b) 1
c) Infinite
d) Cannot be determined Ans: c
6. Entropy of a random variable is
a) 0
b) 1
c) Infinite
d) Cannot be determined Ans: c

7. Which is more efficient method?

a) Encoding each symbol of a block
b) Encoding block of symbols
 c) Encoding each symbol of a block & Encoding block of symbols
d) None of the mentioned Ans: b

8. The mutual information between a pair of events is

a) Positive
b) Negative
c) Zero
d) All of the mentionedAns: d
9. When the base of the logarithm is e, the unit of measure of information is
a) Bits
b) Bytes
c) Nats
d) None of the mentionedAns: c

10. When probability of error during transmission is 0.5, it indicates that

a) Channel is very noisy
b) No information is received
c) Channel is very noisy & No information is received
d) None of the mentioned Ans: c

11.Types of compression

a. Lossless
b. Lossy.
c. both a and b
d. None of the aboveAns: C
12. What is significance of D- frames in video coding
a. They generate low resolution picture.
b. highly compressed technique.
c. They generate high resolution picture.
d. None of the aboveAns: A
13. MPEG coders are used for
a. compression of audio
b. compression of text
c. compression of audio and video
d. None of the aboveAns: A
14. B-frame is also known as
 a. unidirectional.
b. B- De-compression technique
c. B- compression technique
d. bidirectional frame. Ans: D

15. I-frame is
a. It basically searches the frames.
b. It basically predicts the movement of objects.
c. It basically compress the movement of objects
d. None of the above.Ans: B

74. Video Coding consists of two process

a. Processing for reducing Temporal Redundancy.

b. Processing for reducing Spatial Redundancy

c. Both a and b

d. None of the above.Ans: C

16. H.261 is

a. compression of audio
b. De-compression of audio
c. Video Compression standard
d. None of the aboveAns: A
 Assignment

(1)Explain the terms (i) Self information (ii) Average information (iii) Mutual Information.
(2)Discuss the reason for using logarithmic measure for measuring the amount of information.
(3)Explain the concept of amount of information associated with message.

(4) A binary source emitting an independent sequence of 0’s and 1’s with probabilities p and (1p)
respectively. Plottheentropyofthesource.

(5) Explain the concept of information, average information, information rate and redundancy as
referred to information transmission.

(6)Let X represents the outcome of a single roll of a fair dice. What is the entropy of X?

(7)A code is composed of dots and dashes. Assume that the dash is 3 times as long as the dot and
has one-third the probability of occurrence. (i) Calculate the information in dot and that in a
dash; (ii) Calculate the average information in dot-dash code; and (iii) Assume that a dot lasts
for 10 ms and this same time interval is allowed between symbols. Calculate the average rate
of information transmission.
(8)What do you understand by the term extension of a discrete memory less source? Show that
the entropy of the nth extension of a DMS is n times the entropy of the original source.

(9)A card is drawn from a deck of playing cards. A) You are informed that the card you draw is
spade. How much information did you receive in bits? B) How much information did you
receive if you are told that the card you drew is an ace? C) How much information did you
receive if you are told that the card you drew is an ace of spades? Is the information content of
the message “ace of spades” the sum of the information contents of the messages ”spade” and
“ace”?

(10) The output of an information source consists OF 128 symbols, 16 of which occurs with
probability of 1/32 and remaining 112 occur with a probability of 1/224. The source emits
1000 symbols/sec. assuming that the symbols are chosen independently; find the rate of
information of the source.