Taking Control of the Control Group: How to Collect Reliable Data

A key of conducting analytic research that generates worthy insights is comparing data collected from an experimental group to data from an accurate control group in order to ensure the variable in question is isolated.  

Sounds obvious, right?

Well, in fact, in most digital data-driven analytical research done today, generating the correct control group is trickier than one might think, leading to bias and inaccurate results. For example, one of the most common misconceptions is the belief that control groups should be generated randomly as opposed to selected and crafted with care. In fact, not being rigorous in the formation of the control group when trying to isolate the specific variable being studied leads to random results in each test conducted. In cases where the control group is not created carefully, the finished piece of research will offer a brand manager insights of little-to-no value.

There is a growing push for increased transparency and accountability in marketing as well as for development toward greater accuracy and reliability within the data ecosystem. Major marketers are demanding that the digital marketing and advertising community get a handle on fraud and brand safety and create a more rational supply chain. Indeed, it is high time for brands to add data integrity to their transparency to-do list.

However, many brand marketers have been slow to address this pain point, relying on black box third-party data providers and lacking the internal resources to vet and match data sets against each other. In some cases, marketers have not prioritized the importance of abandoning legacy practices.  

At the heart of the research methodology challenge is that data continues to be siloed. Data silos are a result of the fragmentation of the digital ecosystem, where multiple vendors are often involved in each campaign, resulting in a partial view into the data for all.

Third-party measurement companies commissioned to conduct analytic research are often at a disadvantage from the get-go because they aren’t given critical information such as gender and geo data, without which it’s virtually impossible to test correctly.  

All eligible devices and channels have to be accounted for in conducting research, but brands have been slow to create in-house data infrastructures that mitigate silos and integrate datasets from the myriad platforms they employ. Only in this manner will measurement vendors be able to effectively meet their goals. Having all of the data in a central repository will give them a complete view and allow them to truly leverage data in the form of actionable insights and avoid pitfalls such as biased or inaccurate control groups.

Here are the key basic practices that we encourage all marketers and their partners to apply to overcome these challenges:

• Select your control group in advance by choosing part of your target audience to be excluded from actual targeting and ad serving in order to ensure that the only difference between groups will be exposure to the ad. In this manner, you can benchmark the exposed group against the control group, which will meet the same prime prospect criteria except exposure to the campaign.  
• Demand transparency from your research partners. Even if exact methodology or panel make-up can’t be shared, insist on greater verification of the results and parameters (e.g., control group size).
• Since running multiple tests can be costly and difficult to execute, a simple workaround is to request two reports looking at different time slots. If this is done, trends and insights are more likely to be valid and useful.

In short, only when marketers account for the totality of their consumer data and account for it in an integrated way will the data actually offer visibility and control. By demanding higher levels of accountability and transparency from third-party vendors, brands will achieve better results.  

If marketers truly want to ascend to the next level of marketing and advertising ROI, they must embrace the technology and allocate the dollars and resources necessary to create a centralized data infrastructure. Data is messy because it’s disparate and fragmented, obtained from many different channels including cookies, device IDs, and set-top boxes. It’s hard work, but it’s imperative if we are to finally reach the era of transparency in marketing.

Ran is Ubimo’s CEO. Prior to Ubimo, Ran co-founded LabPixies Ltd (acquired by Google in 2010), a leading web and mobile app development company, bootstrapping the company and growing the business to reach tens of millions of users in four years. At Google, Ran continued as a product manager in search where he led and launched large-scale products. Ran holds a bachelor’s degree in computer science from the Hebrew University of Jerusalem, Israel.