Measuring Deep Canvassing
Developing a deep canvass approach is a process. Independent quantitative measurement allows you to rigorously learn how far along the learning curve you are: have you developed a successful approach, or do you need to go back to the drawing board? In our experience, many encouraging anecdotes may give the impression a deep canvass program is working even when it is not, such as in the abortion experiment.
Conducting an experiment to assess a deep canvass can be done fairly cheaply and quickly using a method we have developed and refined to rigorously test whether a deep canvassing program works: repeated online panel experiments (ROPEs).
This approach only requires approximately 1,000 deep canvass conversations to yield a rigorous and precise answer about whether a deep canvass program works and whether its effects last. This means ROPEs allow groups to run rapid, small-scale, relatively inexpensive tests early on to rigorously evaluate new ideas for deep canvassing.
As described in our academic paper, ROPEs work by identifying voters likely to respond to surveys and gathering precise measures of their opinion both before and after a persuasive intervention with an experiment. Our first study using a ROPE to measure the effects of the transphobia deep canvassing program was covered in This American Life.
ROPEs can answer three kinds of questions:
- Does a persuasive approach work at all?
- Do its effects last over time? Which of two or more persuasive approaches works best?
- Who does a persuasive program affect most? I.e., what targets are most responsive?
A relatively precise ROPE measuring a deep canvass program could be able to detect an effect of 5 percentage points with around 1,000 persuasion conversations (e.g., 200 shifts with 5 conversations per shift) and survey costs of $50,000.
Those interested in further details can also explore our persuasion experiment design tool online or read the full academic paper.