Citation
Abstract
This article tabulates continuous probability density functions and discrete probability mass functions which maximize the differential entropy or absolute entropy, respectively, among all probability distributions with a given Lp-norm (i.e., a given pth absolute moment when p is a finite integer) and unconstrained or constrained value set. Expressions for the maximum entropy are evaluated as functions of the Lp-norm. The most interesting results are obtained and plotted for unconstrained (real-valued) continuous random variables and for integer-valued discrete random variables. The maximum entropy expressions are obtained in closed form for unconstrained continuous random variables, and in this case there is a simple straight-line relationship between the maximum differential entropy and the logarithm of the Ly-norm. Corresponding expressions for arbitrary discrete and constrained continuous random variables are given parametrically; closed-form expressions are available only for special cases. However, simpler alternative bounds on the maximum entropy of integer-valued discrete random variables are obtained by applying the differential entropy results to continuous random variables which approximate the integervalued random variables in a natural manner. Most of these results are not new, The purpose of this article is to present all the results in an integrated framework that includes continuous and discrete random variables, constraints on the permissible value set, and all possible values of p. Understanding such as this is useful in evaluating the performance of data compression schemes.
Details
- Volume
- 42-104
- Published
- February 15, 1991
- Pages
- 74–87
- File Size
- 498.2 KB