EE 708 – INFORMATION THEORY AND CODING
Course offered in:
Spring 2021
Instructors:
Bikash Dey
Course Content:
Information theory is an area of mathematics that electrical engineers find particularly neat because it provides answers to the most fundamental questions regarding the theoretical limits of reliable communication. But it finds applications in statistics, physics, economics, biology, computer science and even philosophy – ‘information’ is central to the way we perceive the world and understanding it helps unlock many doors. This course starts by defining the language of information theory – entropy, information, rate, channel, capacity, and so on. Then the emphasis shifts to understanding Shannon’s incredible theorems. Since this is an EE course, everything will be done with a focus on communication (refer to section on prerequisites). A few practical coding techniques may also be discussed, particularly for source coding. Finally, some information theoretic problem settings may be described, such as that of broadcast or multiple access channels, to provide a flavour of pursuing information theory further would be like. For an exact list of the (typical) topics covered, please refer to the website.
Prerequisites:
The only real pre-requisite is a probability course (such as EE 325). But to fully appreciate the material covered, it would help to have also done (or do in parallel) a course on digital communication. Sophomores are generally not allowed to take the course.
Feedback on Lectures:
Prof. Bikash has a very measured and fluid style of explanation which makes it easy to follow what is being taught. Since this was in an online semester, there were pre-recorded lectures totalling around 30 minutes that we were expected to watch in each lecture slot, and then attend a 10-minute discussion/doubt clearing session in the same slot.
Feedback on Tutorials, Assignments and Exams:
In tutorial sessions, TAs discussed solutions to assigned problems from the textbook (Cover and Thomas, see below). The exam questions were similar to these, save for a couple of tricky ones. There were no assignments, but there was a reading project where groups of two students were each allotted a topic to study and present.
Difficulty:
With this being an introductory course, the material is almost completely self-contained, except for basic probability. A certain degree of mathematical rigour is required, which might make the course seem harder than it is. As with all mathematical courses, lectures build on previous ones, which calls for regular effort. Overall, the course is easy.
Grading Statistics:
Study Material and References:
There are several classic textbooks, but here are my favourites:
- Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. (best place to learn the basics, has lots of questions at the end for practice and also add to the material covered)
- MacKay, David J. C. Information Theory, Inference and Learning Algorithms. Cambridge University Press, 2003. (beautifully written, offers a different perspective to the more business-like Cover and Thomas; source coding is best learnt from here)
- Abbas El Gamal and Young-Han Kim. 2012. Network Information Theory. Cambridge University Press, USA. (in general offers a more rigorous treatment of the subject, but the focus is on multi-user settings that are generally covered towards the end of the course)
Advanced courses that can be taken after this:
EE 756 – Network Information Theory [EE 708 mainly deals with setting up the language and structure of the subject – EE 756 is where it really gets to the heart of the problem (from a communication perspective, of course)],
EE 726 – Advanced Information Theory and Coding [name is self-explanatory].
(Neither of these courses have run in a while, so they may not be an option for you.)
Takeaways from the course:
This is an introductory course to the world of information theory. Anyone who is interested in digital communication should definitely do this course, along with EE 605 – Error Correcting Codes. Anyone interested in applied mathematics should also consider this course.
Review by – Adway Girish (adwaygirish@iitb.ac.in)
Sponsored Content