AARP and All Our Ideas
Using Wiki Surveys for Identifying Problems at Scale
How Does It Work?
AARP, the nation’s largest nonprofit, and The Governance Lab (The GovLab) launched an online consultation to better understand AARP members’ concerns about “big health data.” They used a wiki survey tool, which tabulates the inputs even of tens of thousands of people without the need for those running the consultation to do extra work to extract meaning for citizen responses.
Big health data refers to the ability to gather and analyze large quantities of information about health, wellness and lifestyle. Big health data includes information from health care providers as well as data from sources such as apps that track our sleep and exercise habits and the purchases we make.
Doctors, healthcare organizations, insurance companies, financial service providers, product and service companies, and governments at every level are keen to use such data. Increasing the use and sharing of data could enable better diagnoses, more targeted prevention and treatments, faster research and cures, the creation of new tools to help us make healthier choices, and economic growth from the creation of health data businesses. At the same time, the collection, sharing, and use of big health data could reveal sensitive personal information over which we have little control. This data could be sold without our consent. It could be used by entities for surveillance or discrimination, rather than to promote well-being.
AARP and the GovLab started the Big Health Data public consultation in order to “tap into the previously untapped know-how in the population to help AARP make more informed recommendations to Congress about how health data should be treated” on the basis of people’s lived experience, family history and professional expertise. The project was designed to gauge AARP members’ concerns on which big health data issues were most important to them. Rather than asking participants an open-ended question like “What are your concerns regarding the use of big health data,” which would likely lead to off-topic answers and make it difficult to process the results, this project asked participants to help prioritize a pre-populated list of concerns.
In order to do so, The GovLab used All Our Ideas — a “ wiki survey” tool developed by researchers at Princeton University that can be used to help a community identify and prioritize problems as part of a law or policymaking process. (We also showcase how a wiki survey has been used for solving problems at scale as was the case of Governador Pergunta in the State of Rio del Sul, Brazil).
Rather than give respondents a lengthy and time-consuming survey, the wiki survey presents participants with two randomly selected items from the list and asks them to select the one which is of greater concern or importance to them. Thus, for example, AARP asked its Members: “Which is your greater concern regarding big health data?” The GovLab and AARP prepared 63 statements in response, such as: “Big health data may lead to the encouragement of self-diagnosis over seeing a doctor” or “ Companies can use big health data without having to tell anyone what they are doing with it or being accountable for it.”
Respondents answer as many or as few randomized pairings as they want. They can also choose to pick “I can’t decide.” Respondents are also allowed to submit their own answer choice, which, after approval by the administrator of the wiki survey, will be added to the list and displayed to future respondents.
Who participated?
This process, known as “pairwise voting,” is faster and easier than responding to a long survey. Voting remained open for three weeks in December 2019 and over 5,000 participants cast 67,000 votes. (In Brazilian case, described under Solution Identification, over 100,000 people participated.) However, no demographic information was collected about respondents.
Results
At the end of the consultation, AARP received a rank ordered list of problems automatically presented by the software, which offers multiple visualizations of the results, including the ordered list of responses, data on when people participated and how many. If participants are required to login, then the software can also provide data and visualizations about member locations. In this consultation, participants articulated that ensuring corporate accountability and preventing discrimination by insurers were their most pressing concerns regarding the use of their health data. AARP used the findings from the wiki survey to run a second consultation, asking people to identify novel solutions to the problems identified via All Our Ideas.
What does it cost?
All Our Ideas is a free, open source tool available at allourideas.org. Anyone can set up a wiki survey on the All Our Ideas website. In addition, the software can also be downloaded and used on one’s own website. The public version of the tool can be embedded in any website at no cost and with basic knowledge of HTML/CSS. The All Our Ideas code and API are available on Github at https://github.com/allourideas) and can be used, for free, to customize the tool’s look,feel, and features. These customizations require more advanced software programming knowledge.
What are the benefits?
To date, organizations around the world have created 16,922 wiki surveys and have amassed 29.7 million responses.
- Large scale participation: The All Our Ideas tool is easy to use and requires little instruction. As a result, thousands of participants in the Big Health Data consultation each cast several votes in a short period with little difficulty. Additionally, since participants are not required to log in or provide any other information before voting, the barrier to participation is very low.
- Ease of administration: A wiki survey can be set up by anyone with no technical knowledge via All Our Ideas. Since the software automatically tabulates results and visualizes them, there is no additional effort required to extract meaning from participant responses.
What are the risks?
- In order to get the most useful, actionable insights from a wiki survey, framing unambiguous problem statements is critical. Statements with technical jargon or generic sentences are likely to confuse participants and will fail to capture their real concerns. The AARP and the GovLab prepared many more negative concerns (ie. fear of use of big health data) rather than “positive” concerns (ie. fear of failure to use big health data) among the 63 statements. Thus, outputs were also heavily skewed toward the negative.
