Previous |  Up |  Next


Title: Maximizing multi–information (English)
Author: Ay, Nihat
Author: Knauf, Andreas
Language: English
Journal: Kybernetika
ISSN: 0023-5954
Volume: 42
Issue: 5
Year: 2006
Pages: 517-538
Summary lang: English
Category: math
Summary: Stochastic interdependence of a probability distribution on a product space is measured by its Kullback–Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probability distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure. (English)
Keyword: multi-information
Keyword: exponential family
Keyword: relative entropy
Keyword: pair- interaction
Keyword: infomax principle
Keyword: Boltzmann machine
Keyword: neural networks
MSC: 60B10
MSC: 82C32
MSC: 92B20
MSC: 94A15
idZBL: Zbl 1249.82011
idMR: MR2283503
Date available: 2009-09-24T20:18:24Z
Last updated: 2015-03-29
Stable URL:
Reference: [1] Aarts E., Korst J.: Simulated Annealing and Boltzmann Machines.Wiley, New York 1989 Zbl 0674.90059, MR 0983115
Reference: [2] Ackley D. H., Hinton G. E., Sejnowski T. J.: A learning algorithm for Boltzmann machines.Cognitive Science 9 (1985), 147–169 10.1207/s15516709cog0901_7
Reference: [3] Aigner M.: Combinatorial Theory, Classics in Mathematics.Springer–Verlag, Berlin 1997 MR 1434477
Reference: [4] Amari S.: Information geometry on hierarchy of probability distributions.IEEE Trans. Inform. Theory 47 (2001), 1701–1711 Zbl 0997.94009, MR 1842511, 10.1109/18.930911
Reference: [5] Amari S., Kurata, K., Nagaoka H.: Information geometry of Boltzmann machines.IEEE Trans. Neural Networks 3 (1992), 2, 260–271 10.1109/72.125867
Reference: [6] Ay N.: An information-geometric approach to a theory of pragmatic structuring.Ann. Probab. 30 (2002), 416–436 Zbl 1010.62007, MR 1894113, 10.1214/aop/1020107773
Reference: [7] Ay N.: Locality of global stochastic interaction in directed acyclic networks.Neural Computation 14 (2002), 2959–2980 Zbl 1079.68582, 10.1162/089976602760805368
Reference: [8] Linsker R.: Self-organization in a perceptual network.IEEE Computer 21 (1988), 105–117 10.1109/2.36
Reference: [9] Matúš F., Ay N.: On maximization of the information divergence from an exponential family.In: Proc. WUPES’03 (J. Vejnarová, ed.), University of Economics, Prague 2003, pp. 199–204
Reference: [10] Shannon C. E.: A mathematical theory of communication.Bell System Tech. J. 27 (1948), 379–423, 623–656 Zbl 1154.94303, MR 0026286, 10.1002/j.1538-7305.1948.tb01338.x
Reference: [11] Tononi G., Sporns, O., Edelman G. M.: A measure for brain complexity: Relating functional segregation and integration in the nervous system.Proc. Nat. Acad. Sci. U. S. A. 91 (1994), 5033–5037 10.1073/pnas.91.11.5033


Files Size Format View
Kybernetika_42-2006-5_1.pdf 1.184Mb application/pdf View/Open
Back to standard record
Partner of
EuDML logo