Basic Information
 Course Code: EE708
 Course Name: Information Theory and Coding
 Course Offered In: 20232024
 Semester Season: Spring
 Instructors: Prof. Bikash Kumar Dey
 Prerequisites: EE325 (Probability and Random Processes), having done EE605 (Error Correcting Codes) might help in some topics
 Difficulty (1 being easy and 5 being tough): 3
Course Content

Introduction: Definition of information quantities with examples, logsum inequality, Jensen’s inequality, Fano’s inequality, data processing inequality, sufficient statistics

Source coding techniques: Types of codes – singular, nonsingular, uniquely decodable, prefixfree codes; Huffman code, Arithmetic codes, LempelZiv codes, Kraft inequality, Optimal codes, source coding theorem under zero error.

Typicallity: Typical sequences – weak and strong typicallity, AEP, Counting results on typical sequences, source coding theorem under zero error and small error using AEP.

Channel coding: Channels, capacity, and the channel coding theorem: Discrete memoryless channels, capacity calculations, jointly typical sequences and their counting, channel coding theorem with proof, average error versus maximum error – expurgation of codes, feedback does not improve capacity

Differential entropy, multivariate Gaussian distribution, Gaussian distribution maximizes differential entropy for a given covariance matrix

AWGN channels: Capacity calculation, entropy power inequality, parallel AWGN channels, power allocation – water pouring, Capacity of continuous time bandlimited AWGN channels

Multipleaccess channels  joint source and channel coding for multiple sources (depending on time remaining)
Feedback on Lectures
80% compulsory attendance, else DX. The very start of the course and the last two or three weeks had the professor use the blackboard, but for the remaining semester he used slides.
He often went slow through the slides for making sure all the students were able to follow  which is essential for any class with >10 or 15 students, but it can become boring for those who quickly read and assimilate the math on the slides but then listen to it being explained for 510 minutes. Contrary to that, following through teaching on blackboard is much more engaging as the entire text doesn’t appear instantaneously but rather builds on.
The professor often used only orderwise math (exponential vs polynomial etc) which initially appeared handwavy, but later the proofs/bounds all depended on such similar methods.
Following through the lectures is essential as the content is very rich in math. A lot of it can be covered from his extensive slides alone + the reference book, but since attendance mandates going to classes anyways, one should make use of it by making notes and jotting down useful intuitions/ideas.
Feedback on Evaluations
2 Quizzes totalling 25%, Midsem 25%, Endsem 25%
The exams were usually on the easier side and were not very difficult to solve if you follow through the content regularly and solve the tutorial problems (and more from the reference book). Writing proofs clearly with all steps and reasons shown is essential for the TA to give full marks, as the course is almost entirely mathematical and solutions need to be detailed enough to show all concepts used.
P.S.  We were initially under the impression that the professor will do absolute grading (90+ AA, 80+ AB and so on) based on the previous year’s feedback, but it was a lot more lenient than that in the end.
Study Material and Resources
Elements of Information Theory by Cover and Thomas. It has all the concepts, solved examples and practice problems.
Tutorial problems are usually given from this book, which is a subset of all the problems and usually represents the level of questions to be expected in exams. A large part of the textbook also has problem solutions, whose pdf can be found on the internet with some hunting.
Followup Courses
EE 605 (Error Correcting Codes), runs in odd semesters
Final Takeaway
This course would be a great choice for students interested in the math/probability aspect of EE, specially in communications.
The lecture hours during the period when slides are used can get a bit monotonous, but covering the content (either in class by listening/making notes, or later from slides) should be in sync with the classes and not left for the end.
Solving tutorial problems (and better, the textbook problems) is absolutely essential for exams because the concepts developed in class alone are not sufficient for an idea of how questions can be framed from those concepts  solving problems is necessary for that.