Information Theory Glossary
25 essential terms — because precise language is the foundation of clear thinking in Information Theory.
Showing 25 of 25 terms
A memoryless channel where each input bit is flipped independently with a fixed crossover probability p.
The basic unit of information, equal to the information content of a single binary choice between two equally likely outcomes.
A system that transmits information from a sender to a receiver, potentially introducing noise or distortion during transmission.
The tight upper bound on the rate of information that can be reliably transmitted over a channel, measured in bits per use.
The process of adding structured redundancy to transmitted data to enable error detection and correction at the receiver.
The expected amount of information needed to describe a random variable Y given that the value of another random variable X is known.
The average number of bits needed to identify an event from distribution P when using a coding scheme optimized for distribution Q.
The principle that no processing of data Y can increase the information it contains about a source X: post-processing can only lose information.
The average amount of information or surprise produced by a stochastic source, defined as H(X) = -sum p(x) log p(x).
A coding scheme that enables detection and correction of errors in transmitted data by introducing controlled redundancy.
A channel model where additive white Gaussian noise is added to the transmitted signal, characterized by bandwidth and signal-to-noise ratio.
An optimal prefix-free variable-length code that minimizes average codeword length for a known symbol probability distribution.
A quantifiable reduction in uncertainty about the state of a system, measured in bits (base-2 logarithm) or nats (natural logarithm).
The total entropy of a pair of random variables considered together, measuring the combined uncertainty.
Kullback-Leibler divergence: a non-symmetric measure of the difference between two probability distributions P and Q.
A necessary and sufficient condition for the existence of a prefix-free code: sum of 2^(-l_i) <= 1, where l_i are the codeword lengths.
Low-density parity-check code: a linear error-correcting code with a sparse parity-check matrix, offering near-capacity performance with iterative decoding.
The amount of information that one random variable provides about another, equal to the reduction in entropy of one variable given knowledge of the other.
A class of error-correcting codes, invented by Erdal Arikan, that provably achieve channel capacity with efficient encoding and successive cancellation decoding.
A code in which no codeword is a prefix of any other codeword, allowing unambiguous instantaneous decoding.
The von Neumann entropy S(rho) = -Tr(rho log rho), extending Shannon entropy to quantum states described by density matrices.
The branch of information theory that characterizes the minimum bit rate required for lossy compression at a given distortion level.
The portion of a message that is not essential for conveying information, representing the difference between maximum and actual entropy.
The process of compressing a source's output to approach its entropy rate, removing statistical redundancy.
The set of sequences whose empirical entropy is close to the true entropy of the source, containing approximately 2^(nH) sequences for length n.