EE 708 - Information Theory and Coding
Course offered in:
Spring 2013-2014
Instructor:
Prof. B. K. Dey
Motivation behind the course:
Information theory was developed to find fundamental limits on signal processing operations such as compressing data and on reliably storing and communicating data. Since its inception it has broadened to find applications in many other areas, including statistical inference, natural language processing, cryptography, neurobiology, quantum computing and other forms of data analysis. Applications of fundamental topics of information theory include lossless data compression (e.g. ZIP files), lossy data compression (e.g. MP3s and JPEGs), and channel coding (e.g. for Digital Subscriber Line (DSL)). Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the compact disc, the feasibility of mobile phones, the development of the Internet, the study of linguistics and of human perception, the understanding of black holes, and numerous other fields.
Course Content:
- Review of probability theory
- Entropy , Mutual information, Jensen’s Inequality
- Data compression, Huffman coding, Shannon-Fano-Elias Coding, Lempel-Ziv Coding
- Asymptotic Equipartition Property
- Channel capacity, Differential entropy, Gaussian Channels
- Universal source coding, Block codes and Convolutional codes
- Multiple-Access Channels
Pre requisites:
Basic knowledge of Probability Theory is sufficient.
Feedback on lectures:
Initially lectures were interesting but after source coding (Point 3 of course content) part the lectures became repetitive and slow. However the professor was enthusiastic in answering the doubts during the lectures and giving extra time if required.
Feedback on tutorials, assignments and exams:
Regular tutorials used to happen and many interesting puzzles were also given during these lectures (see, for eg, http://circuit.ucsd.edu/~yhk/ece87-aut11/handouts.html).
Difficulty Level:
Moderate
Grading:
Moderate
Study Material and References:
Textbook was followed throughout the course.
Textbook: Elements of Information Theory by Thomas & Cover
Advanced Courses that can be taken after completing this course:
EE726 Advanced Information Theory and Coding