论文标题
使用开源技术的健康数据协作云计算框架
Collaborative Cloud Computing Framework for Health Data with Open Source Technologies
论文作者
论文摘要
传感器技术的扩散和数据收集方法的进步已经使大量数据的积累积累。这些数据集越来越多地用于科学研究。但是，系统体系结构的设计是在并行化，查询处理时间，异质数据类型的聚合（例如，时间序列，图像，结构化数据等）以及重复科学研究的困难仍然是一个主要挑战。对于健康科学研究而言，这是完全正确的，在该研究中必须是i）易于使用，以在最细粒度操纵数据的灵活性，ii）编程语言内核的不可知论，iii）可扩展和iv）符合HIPAA隐私法。在本文中，我们回顾了有关健康科学科学研究的大数据系统的现有文献，并确定了当前系统局势的差距。我们在分布式环境中使用Apache Hadoop，Kubernetes和Jupyterhub等开源技术为软件硬件数据生态系统提供了一种新颖的体系结构。我们还使用6900万例患者的大量临床数据集评估了系统。
The proliferation of sensor technologies and advancements in data collection methods have enabled the accumulation of very large amounts of data. Increasingly, these datasets are considered for scientific research. However, the design of the system architecture to achieve high performance in terms of parallelization, query processing time, aggregation of heterogeneous data types (e.g., time series, images, structured data, among others), and difficulty in reproducing scientific research remain a major challenge. This is specifically true for health sciences research, where the systems must be i) easy to use with the flexibility to manipulate data at the most granular level, ii) agnostic of programming language kernel, iii) scalable, and iv) compliant with the HIPAA privacy law. In this paper, we review the existing literature for such big data systems for scientific research in health sciences and identify the gaps of the current system landscape. We propose a novel architecture for software-hardware-data ecosystem using open source technologies such as Apache Hadoop, Kubernetes and JupyterHub in a distributed environment. We also evaluate the system using a large clinical data set of 69M patients.
