论文标题
双重信息瓶颈
The Dual Information Bottleneck
论文作者
论文摘要
信息瓶颈（IB）框架是使用原则方法来平衡准确性和复杂性获得的最佳表示形式的一般表征。在这里，我们提出了一个新的框架，即双信息瓶颈（Dualib），该框架解决了IB的一些已知缺点。我们提供了对Dualib框架的理论分析； （i）解决其解决方案的结构（ii）在优化平均预测误差指数方面的优势和（iii）证明其保持原始分布的指数形式的能力。为了解决大规模的问题，我们提出了Dualib的新型变异表述，用于深度神经网络。在几个数据集的实验中，我们将其与IB的变异形式进行了比较。这揭示了Dualib的出色信息平面特性及其在改善误差方面的潜力。
The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity. Here we present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the IB. We provide a theoretical analysis of the dualIB framework; (i) solving for the structure of its solutions (ii) unraveling its superiority in optimizing the mean prediction error exponent and (iii) demonstrating its ability to preserve exponential forms of the original distribution. To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks. In experiments on several data-sets, we compare it to a variational form of the IB. This exposes superior Information Plane properties of the dualIB and its potential in improvement of the error.
