论文标题
使用神经odes建模样式的潜在动力学
Modelling Latent Dynamics of StyleGAN using Neural ODEs
论文作者
论文摘要
在本文中，我们建议通过从GAN中学习独立反向潜在代码的轨迹来对视频动态进行建模。整个序列被视为对初始潜在代码连续轨迹的离散时间观察，通过将每个潜在代码视为移动粒子，而潜在空间则是高维动态系统。因此，代表不同框架的潜在代码被重新重新构成初始框架的状态转换，可以通过神经常规微分方程对其进行建模。学习的连续轨迹使我们能够执行无限的框架插值和一致的视频操纵。后一个任务是重新引入的，用于视频编辑，其优势是需要仅在所有框架上保持时间一致性的同时将核心操作应用于第一帧。广泛的实验表明，我们的方法实现了最先进的性能，但计算却少得多。代码可从https://github.com/weihaox/dynode_release获得。
In this paper, we propose to model the video dynamics by learning the trajectory of independently inverted latent codes from GANs. The entire sequence is seen as discrete-time observations of a continuous trajectory of the initial latent code, by considering each latent code as a moving particle and the latent space as a high-dimensional dynamic system. The latent codes representing different frames are therefore reformulated as state transitions of the initial frame, which can be modeled by neural ordinary differential equations. The learned continuous trajectory allows us to perform infinite frame interpolation and consistent video manipulation. The latter task is reintroduced for video editing with the advantage of requiring the core operations to be applied to the first frame only while maintaining temporal consistency across all frames. Extensive experiments demonstrate that our method achieves state-of-the-art performance but with much less computation. Code is available at https://github.com/weihaox/dynode_released.
