Basic Information

  • Course Code: EE708
  • Course Name: Information Theory and Coding
  • Course Offered In: 2023-2024
  • Semester Season: Spring
  • Instructors: Prof. Bikash Kumar Dey
  • Prerequisites: EE325 (Probability and Random Processes), having done EE605 (Error Correcting Codes) might help in some topics
  • Difficulty (1 being easy and 5 being tough): 3

Course Content

  1. Introduction: Definition of information quantities with examples, log-sum inequality, Jensen’s inequality, Fano’s inequality, data processing inequality, sufficient statistics

  2. Source coding techniques: Types of codes – singular, non-singular, uniquely decodable, prefix-free codes; Huffman code, Arithmetic codes, Lempel-Ziv codes, Kraft inequality, Optimal codes, source coding theorem under zero error.

  3. Typicallity: Typical sequences – weak and strong typicallity, AEP, Counting results on typical sequences, source coding theorem under zero error and small error using AEP.

  4. Channel coding: Channels, capacity, and the channel coding theorem: Discrete memoryless channels, capacity calculations, jointly typical sequences and their counting, channel coding theorem with proof, average error versus maximum error – expurgation of codes, feedback does not improve capacity

  5. Differential entropy, multivariate Gaussian distribution, Gaussian distribution maximizes differential entropy for a given covariance matrix

  6. AWGN channels: Capacity calculation, entropy power inequality, parallel AWGN channels, power allocation – water pouring, Capacity of continuous time band-limited AWGN channels

  7. Multiple-access channels - joint source and channel coding for multiple sources (depending on time remaining)

Feedback on Lectures

80% compulsory attendance, else DX. The very start of the course and the last two or three weeks had the professor use the blackboard, but for the remaining semester he used slides.

He often went slow through the slides for making sure all the students were able to follow - which is essential for any class with >10 or 15 students, but it can become boring for those who quickly read and assimilate the math on the slides but then listen to it being explained for 5-10 minutes. Contrary to that, following through teaching on blackboard is much more engaging as the entire text doesn’t appear instantaneously but rather builds on.

The professor often used only order-wise math (exponential vs polynomial etc) which initially appeared hand-wavy, but later the proofs/bounds all depended on such similar methods.

Following through the lectures is essential as the content is very rich in math. A lot of it can be covered from his extensive slides alone + the reference book, but since attendance mandates going to classes anyways, one should make use of it by making notes and jotting down useful intuitions/ideas.

Feedback on Evaluations

2 Quizzes totalling 25%, Midsem 25%, Endsem 25%

The exams were usually on the easier side and were not very difficult to solve if you follow through the content regularly and solve the tutorial problems (and more from the reference book). Writing proofs clearly with all steps and reasons shown is essential for the TA to give full marks, as the course is almost entirely mathematical and solutions need to be detailed enough to show all concepts used.

P.S. - We were initially under the impression that the professor will do absolute grading (90+ AA, 80+ AB and so on) based on the previous year’s feedback, but it was a lot more lenient than that in the end.

Study Material and Resources

Elements of Information Theory by Cover and Thomas. It has all the concepts, solved examples and practice problems.

Tutorial problems are usually given from this book, which is a subset of all the problems and usually represents the level of questions to be expected in exams. A large part of the textbook also has problem solutions, whose pdf can be found on the internet with some hunting.

Follow-up Courses

EE 605 (Error Correcting Codes), runs in odd semesters

Final Takeaway

This course would be a great choice for students interested in the math/probability aspect of EE, specially in communications.

The lecture hours during the period when slides are used can get a bit monotonous, but covering the content (either in class by listening/making notes, or later from slides) should be in sync with the classes and not left for the end.

Solving tutorial problems (and better, the textbook problems) is absolutely essential for exams because the concepts developed in class alone are not sufficient for an idea of how questions can be framed from those concepts - solving problems is necessary for that.