论文标题
个性化策略实验设计的评估框架
An Evaluation Framework for Personalization Strategy Experiment Designs
论文作者
论文摘要
在线受控实验（OCE）是评估网站更改有效性的黄金标准。 OCE的一种重要类型评估了不同的个性化策略，这些策略在低测试能力方面提出了挑战，并且在小组分配中缺乏完全控制。我们认为，要获得正确的实验设置 - 用户对治疗/分析组的分配 - 应优先考虑降低事后差异技术，以实现实验数量的缩放。我们提出了一个评估框架，该框架以及一些简单的拇指规则，允许实验者快速比较哪种实验设置将导致在特定情况下检测治疗效果的最高概率。
Online Controlled Experiments (OCEs) are the gold standard in evaluating the effectiveness of changes to websites. An important type of OCE evaluates different personalization strategies, which present challenges in low test power and lack of full control in group assignment. We argue that getting the right experiment setup -- the allocation of users to treatment/analysis groups -- should take precedence of post-hoc variance reduction techniques in order to enable the scaling of the number of experiments. We present an evaluation framework that, along with a few simple rule of thumbs, allow experimenters to quickly compare which experiment setup will lead to the highest probability of detecting a treatment effect under their particular circumstance.
