Routing Bias

Olivier de Gaudemar and Scott Porter of OTX presented “Measuring Potential Bias Introduced by Routing” at the 2010 CASRO Panel Conference.

Routing, as defined by OTX, is “a selection mechanism by which an incoming respondent is assigned in real-time to one or multiple surveys, generally after a screening process.” Routing is used because it scales well for sample blending, supports the fielding of surveys to low-incidence populations, directs ready and willing respondents to an active survey and can be used for either panel surveys (driving panelists by email to a router) or for river sampling. Routing is only used by the largest panel providers, as it requires a regular supply of live surveys with a constant flow of willing survey takers.

Routing bias occurs when the output of the router for a particular survey differs from the input: in other words, when a pool of survey takers directed by the router to a survey differs from the untargeted survey takers that enter the router.

• For instance, imagine a scenario where two surveys are being handled by a router: Survey A requires people who watch music videos, and Survey B requires people who watch TV online. If everyone the router identifies as watching music videos is directed to Survey A, then Survey B’s sample will consist of only those respondents who watch TV online and do not watch music videos. In another example, if 60% of incoming respondents watch music videos, but only 40% of respondents who completed the survey do, the routing bias is 0.20 (0.60-0.40).
• When multiple surveys are running, rather than multiplying or adding the bias for each attribute, use the maximum bias for any attribute. If the bias is 0.20 for watching movies in a theater and 0.10 for watching music videos, the maximum routing bias is 0.20.

To explore issues introduced by random routers vs. priority routers, OTX ran different simulations. For 144 simulations of three-survey routing, unsurprisingly, the priority router filled the quota for its survey three times faster than the random router. In 180 simulations of two-survey routing, the high priority survey saw no routing bias, while the low priority survey saw routing bias ranging from > 0.20 to 1.00, rising with the correlation between screening criteria for the two surveys. For 144 simulations of 10-survey routing, with no correlation between surveys, the routing bias for the random router was < 0.10, while the routing bias for the priority router ranged from ~0.14 to ~0.21 for surveys 2 through 10 (no bias for the highest priority survey).

As a result of this research, OTX recommends that panel providers begin measuring and monitoring bias. Other recommendations: “control with quotas when predictable, consider clusters, standardize screeners.” Further, OTX notes that “similar bias also affects sampling in Access Panels, [which have] a finite pool of respondents [and] past participation rules.” In other words, the past participation rules (limit to how frequently a panelist can be invited to take a survey) introduce bias similar to that of a router.

As researchers attempt to better understand the reliability and replicability of research results from access panels, routing bias will be an important source of error to consider.