Conference Video: Wen Qin - How to Run a Trustworthy Online Controlled Experiment and Get Insights?
This is Wen Qin’s tech talk from the WiDS Puget Sound Conference 2020. Enjoy!
ABSTRACT:
How much can a feature boost the revenue? Will a feature hurt product performance? Online controlled experiment (a.k.a. A/B Testing) helps answer the critical questions. However, doing it correctly is challenging. If you search on the internet or talk with an expert, you can find many tips about how to run an experiment. Experiment starters can easily get confused about what steps to take. Advanced experimenters may have a list to go through, but if there are critical check points missing, it can lead to invalid results and incorrect decisions. I will talk about the checklists for running trustworthy experiments. The work is based on the experience of my team with more than 10 years focusing on experimentation and collaborating with majority of Microsoft products to resolve real-world problems
Wen Qin is a Data Scientist on Microsoft's Analysis and Experimentation team for 2 years, focusing on A/B testing. She mainly works with Microsoft Teams on scaling trustworthy experiments to build the culture for experimentation. She also works on several areas to improve trustworthiness of experiment in general, such as checklists for experimentation, metric design, sample ratio mismatch. Prior to Microsoft, she spent half a year at Wayfair as a Data Scientist Intern, working on recommendation models for the personalization of marketing emails.