Information theory

Information theory studies how information is represented and transmitted under specified conditions. A basic communication model has a source produce a message, an encoder turn it into a signal, a channel carry that signal, and a decoder reconstruct the message for its destination. Noise or other disturbances may alter a signal while it travels. By modelling sources and channels mathematically, the theory asks how efficiently messages can be encoded and what limits a channel places on reliable transmission. In Shannon's framework, the technical problem of transmitting signals is treated separately from the semantic question of what a message means.

A central measure in classical information theory is entropy: a quantity describing the average uncertainty of a probabilistic source. For a discrete source with outcomes of probabilities p₁, p₂, …, Shannon's expression is H = −∑ pᵢ log₂ pᵢ, measured in bits per outcome when base-two logarithms are used. When outcomes are more evenly likely, uncertainty is higher; when one outcome is nearly certain, it is lower. Entropy is a statistical measure, not a measure of a message's truth, importance or semantic richness. For lossless coding, source entropy also sets a theoretical limit on average code length, although a practical code may not attain that limit exactly.

Source coding uses patterns in a source to represent messages efficiently; in a simple variable-length code, more probable symbols can receive shorter codewords than rare ones. Data-compression methods apply these ideas to files and signals. Communication through a noisy channel presents a different challenge: error-detecting and error-correcting codes add structured redundancy so a receiver can detect or repair some errors. Shannon defined channel capacity as a theoretical upper limit on the rate of reliable information transfer for a channel. His noisy-channel theorem shows that, under the mathematical model's assumptions, coding can make the error probability arbitrarily small at rates below capacity, while reliable transmission at rates above capacity is not possible. The theorem identifies a limit; it does not promise that a practical system will reach it with a simple code.

Modern information theory grew from practical questions about the speed and reliability of telegraph and telephone systems. Work by Harry Nyquist on telegraph speed in 1924 and by Ralph V. L. Hartley on information in 1928 supplied important earlier foundations. In 1948, Bell Laboratories researcher Claude E. Shannon published “A Mathematical Theory of Communication” in the Bell System Technical Journal. The paper presented a general mathematical model of communication and developed measures of information and channel capacity that could be applied to both noiseless and noisy channels.

Information-theoretic ideas underpin important parts of digital communication, compression and error-control coding. They also provide tools used in areas such as statistics, physics and the study of biological or linguistic systems, though transferring the mathematical framework to a new field requires care about what is being measured. Shannon's theory deliberately abstracts away from a message's meaning: it can quantify uncertainty and transmission constraints without deciding whether a message is true, useful or understood by its recipient.