EE 708 - Information Theory and Coding

Course offered in:

Spring 2014-2015

Instructor:

Prof. Bikash Kumar Dey

Overview:

This course is an introduction to how messages and their content can be quantified. The course revolve around the two central tenets of source coding and channel coding. In simple English, the course tries to show how we can transmit a message from A to B (Which can be separated in space (communication) and/or time (storage)) efficiently and reliably. If time permits, compression algorithms used by computers are also discussed.

Warning: The course is mathematically heavy, and a certain level of rigour is expected as par for the course.

Pre-requistites:

The course is aimed at Electrical Engg. students who have completed a course in Probability and Random Processes (EE 325). This is not, however, a hard pre-requisite, though it is definitely advised, since it makes life a lot easier for the student in this course.

Course Contents:

The course begins by defining and trying to provide intuitive understanding of Shannon’s Entropy, and this is where the mathematical heaviness begins, as you are exposed to several related quantities and other mathematical tools. This buildup of a toolset lasts for a little over a month and it is this that allows you to understand Shannon’s beautiful theorems.

After this, one goes through Huffman Codes and their optimality (basically where you show that this is The best compression you can get for a given source of messages), Shannon Codes and also maybe Lempel-Ziv codes. The advantages and disadvantages of these codes vis-a-vis optimality and computational complexity are discussed.

The rest of the semester (about half of it) is given up to Source coding and channel coding and the basic proof technique of Random Codebooks.

Evaluation:

This is course is, as mentioned earlier, heavy on the math, and so are the assessments, which consist of two quizzes, one Midsem and one Endsem.

The assessments all have both Proof-based questions, and Solving-based questions.

Rigorous answers are expected, and lax answers are penalized.

Study Material:

Homework questions are assigned regularly, and ensuring you are up to date with these will be enough to get a good grade, since often enough homework questions have turned up on examinations. The Professor also hold a weekly optional tutorial where students can ask him questions on topics they have found difficult.

Elements of Information Theory by Cover and Thomas is the book followed by the Professor, and most homework questions are set from here. Using this to consolidate lectures is the best thing one can and should do, especially as for the first half semester there are no slides.

Grading and Difficulty:

It is not an easy course, but simply attending lectures, doing the homework and following the book can be enough to get a decent grade.

Attending lectures is not only beneficial academically, but it seems the Professor also reserves a certain part of the marks for attendance.

Reviewed by Jay Mardia (jay.mardia19@gmail.com)