Social Auditing

Policy evaluation is the process of “understanding how a policy or other intervention was implemented, what effects it had, for whom, how and why.”^[1] It serves as an important piece in the feedback loop to improve existing service delivery and inform future policy formulation. However, some of the oft-cited challenges to effective evaluation include scarcity of resources and access to relevant data.^[2] The Internet creates the opportunity for engagement by asking the public how to measure impact, what data to use for that purpose and enlisting people in the process of evidence gathering to support better evaluation and oversight. Such participation has the potential to enhance accountability and improve results. 

Below, we summarize three so-called social auditing (also called civic auditing) initiatives that have enabled greater citizen participation in monitoring government projects. Although only one of these involves the legislative branch of government, the success of these projects, coupled with the absence of many legislative examples (UK Evidence Checks and Chile’s Evaluación de Leyes are two notable exception), leads us to include them as worthy exemplars for legislatures looking to engage the public in oversight and evaluation.^[3] 

In late 2016, at an event to mark the launch of a new Brazilian government transparency portal, the director of a high school from the rural area of Gama, Brazil publicly rued the severe lack of resources dedicated by the government to his institution - a story that is not uncommon in Brazilian public schools:

“I bought the taps installed in the bathrooms. I turned my living room into a pantry for food. I am very sad about this situation” - Edgard Vasconcelos, director of CED Casa Grande

A decree passed in 2007 ensures that Brazil’s public schools have the autonomy needed to spend funds assigned to them by the federal district for maintaining and operating the school. The intention behind the decree (called Programa de Descentralização Administrativa e Financeira (PDAF)) was to help public school management respond in an agile manner to local needs, which they are best-suited to know.

Between 2007, when the decree was passed, and 2016, over R$ 445 million was provided to public schools for school maintenance and local repairs with R$ 84 million being provided in the year 2016 alone. Despite this, audits conducted in random municipalities by the nation’s Comptroller have shown that there are deficiencies in school infrastructure quality across the country. Studies have attributed these deficiencies to several causes, including lack of resources, corruption^[4] and student behavior, but there is less information available at granular levels to pinpoint issues such as those faced by the Casa Grande high school in Gama. School administrators have often complained that they are constrained by delays in funding transfers while government officials have passed the blame back to schools.

In 2016, the Comptroller General of the Federal District (CGDF) launched an initiative called the Projeto Controladoria na Escola (Controllership in Schools) to engage students in 10 public schools in Brazil in the process of auditing school infrastructure, mapping commonly raised issues and fostering civic education in schools.^[5] 

The initiative was in accordance with the National Social Participation Policy (PNPS) that aims to prevent corruption by ensuring that public resources are spent transparently and with effective participation of society. It was also one of the 22 projects^[6] selected from over 90 proposals submitted to the #TodosJuntosContraCorrupcao campaign, a national anti-corruption campaign by the “ENCCLA”, the National Strategy to Combat Corruption and Money Laundering.^[7] 

Pilot Phase

Projeto Controladoria na Escola involved asking students to collect data about their local school environments, report the major issues they faced, identify the root causes of those issues and propose ideas to fix them. In the pilot phase students from 10 schools (including Edgard’s CED Casa Grande) participated in the process. In total, they identified over 600 issues including burnt out light bulbs, missing fire extinguishers and broken chairs. The students, as well as teachers, were also surveyed on the mode of transportation they use to go to school and their opinions about the school on a wide range of issues, ranging from the quality of educational materials provided to them to the state of the sports arena and labs. The CGDF compiled the issues identified and survey responses from each school into a report which detailed the audit findings which included images, descriptions and deadlines, which were then presented to the department of education. The Comptroller General visited each school later that year to monitor the results of the project and to oversee the resolution of the issues. In one school alone, the students identified 115 issues and within just 3 months, 45% of the issues were fixed either by the department of education or, where possible, by the students and school management themselves.^[8] 

Institutional Impact

The success of the project was two-fold. It not only enhanced the CGDF to conduct detailed audits of every public school but also generated greater buy-in from the schools to identify, report and fix issues in their surroundings. The buy-in from school management was a critical takeaway for the CGDF. By allowing the schools themselves to identify the issues, the CGDF was able to perform a full audit of the schools and see how public funds were spent without the negative connotation associated with being “overseen or audited.” Rather, the schools were able to see for themselves how misusing funds or neglecting the upkeep of school property was creating several issues for the students and teachers and the department of education was made aware of the most urgent issues public schools in Brazil were facing.

“This is the best way to fight against corruption. When the citizen understands that the public good belongs to him, he takes care of it” said Ziller. “Controladoria na Escola involves students in identifying and solving the institution's problems. This makes them aware, for example, that if they vandalize a bathroom, they lose resources that could be invested in improving the college.”

Expansion: The school audit award (2017)

The project’s great success was also evidenced by the fact that the social (also called civic) audit model was replicated the following year (2017), this time in 104 schools with over 4,000 students competing for a R$ 140,000 ($43,000 USD) grant award. It was called the “school audit award” and by the end of the campaign, students had submitted around 7,500 responses^[9] to the survey.

In 2018, the program is set to expand to over 200 schools in the country.^[10]

How it worked

The scale of participation in the first school audit award was much larger than the pilot. Hence, manually compiling reports from the data collected by the students was infeasible. Instead, the CGDF deployed Promise Tracker - a data collection tool developed by the MIT Center for Civic Media  Promise Tracker is a mobile application which allows campaign organizers to create surveys for distribution in order to collect information in the form of pictures, text and location data.

The school audit award campaign consisted of 5 phases:^[11]

1. Training for teachers to guide students through the data collection process
2. Theatrical shows and debates to show the value of citizenship and public participation
3. Student-led evaluation of school infrastructure (using Promise Tracker)
4. Student-led assessment of the problems identified
5. Student-led development of solutions to fix them

Judges from the Comptroller General’s office scored each school based on its performance in each of the activities and the top 10 schools^[12] shared the R$ 140,000 ($43,000) grant award.^[13] Nearly 4,000 students from 104 public schools participated in the campaign and helped evaluate the state of classrooms, availability of Wi-Fi and computer labs, toilet paper in bathrooms and other issues by collecting evidence in response to a questionnaire administered through Promise Tracker. 

Using the information they gathered, the students then went through a process to determine the root causes (such as student behavior, lack of resources and administrative issues) of the most commonly reported issues and went on to propose projects to address the issues they felt they could have an influence on.

By the end of the competition, all 104 schools had not only proposed but also implemented at least one - if not more - student-designed initiatives even though only the top 10 teams stood a chance to win the grant award.^[14] Among other reasons, giving participants the ability to intervene in their local environments in order to affect real change was a critical factor in achieving such large-scale participation.^[15] 

Among the projects the students developed was the Monitoring My School app designed to monitor the cleanliness of classrooms and common areas. The app also allowed janitorial staff to provide feedback on how the students maintained the tidiness of the school. Another school launched a web-based radio station by renovating an old, out-of-use computer lab to motivate students and teachers to maintain the space well.^[16] 

Figure 1: The winners of the Civic audit received a certificate from the Comptroller-General, Henrique Ziller, Source: Andre Borges/Agência Brasília

Technology

The open source Promise Tracker Tool (URL:  https://monitor.promisetracker.org/)^[17]

How it works: Creating a campaign using Promise Tracker

Step 1: Describe Project: The tool asks the campaign organizer to describe the project and its targeted audience.

Step 2: Set up Survey: The tool then allows the organizer to create a survey which can be disseminated among the public for data collection. The organizer can ask users to respond with text, images and/or location information.

Step 3: Design Survey Page: Next, the organizer is asked to design the look and feel of the survey page. Once complete, the tool provides a QR code, a machine-readable code consisting of an array of black and white squares, used for storing URLs,, and links for sharing the survey with the public. If required, the organizer can choose to be anonymous.

Step 4: Test Survey: The organizer can test the final survey to make sure all the fields work as required prior to making it public.

Step 5: Collect Data: Once the survey is live, the organizer can view the results on a dashboard which visualizes the results, displaying graphs, maps and photos.

Figure 2: Promise Tracker Sample Dashboard. Source: monitor.promisetracker.org

How it works: Responding to the survey

Step 1: Download Mobile App: After downloading the Promise Tracker mobile app, the user can download the campaign survey using a 6-digit code shared by the organizer.

Step 2: Data Collection: The type of data a user must collect depends on the requirement of the campaign. This might include text, images or location information.

Learnings

Design the initiative in a way that generates buy-in:  Some schools were hesitant to take part in a campaign that was going to “monitor and audit” their activities. However, engaging students and teachers and giving them the power to create change persuaded schools that it would be a value-add to the school rather than become “monitoring” in a negative sense.

Technology is a small piece - focus on networks for engagement: Working with community groups on the ground who already have a certain network and issues they care about was critical to the success of the campaign. In this case, the comptroller-general

The civic audit model employed in Brazil is a great example of organizing citizen-led campaigns to foster civic education and to help government oversight agencies understand local issues in granular detail. It also helps build a sense of community and, when done right, motivates citizens to take action to fix those issues. It is still unclear, however, if the campaign improved educational outcomes and if the medium and long-term solutions were implemented.

It is important to ensure that issues reported by citizens are used to enhance government accountability and improve policy implementation and formulation. If citizens don’t see that the data they collect is being acted on, they are less likely to participate in subsequent iterations of the project. On the other hand, government can take action only if the information from citizens is routed to the appropriate departments. For instance, reporting a leaky roof in a school to the department of education is likely to be less impactful and slower (or entirely useless) if the relevant authority to fix the issue is actually the public works department. In other words, there needs to be a feedback loop which carries citizen input to the relevant authority and one that informs the citizen when their report has been acted upon.

An interesting example of a platform which attempts to do this comes from Ghana and a platform called TransGov Ghana.

Problem it solves

In Ghana, there are no official mechanisms to allow citizens to easily find details of local infrastructure projects. For over four years now, the Ghanaian parliament has deliberated on the passage of a right to Information bill but it is unclear when it will materialize. Ghana is part of the Open Government Partnership (OGP) and puts out open data on the national open data portal (data.gov.gh). But even the portal boasts of “133 datasets”, only 15 datasets (mostly census data) are available to view or download and they are rarely updated. Infact, the OGP end-of-term review report in 2017 found^[21] that Ghana had made “limited” progress in its commitment to make datasets publicly accessible on the portal. Without a right to information bill or a robust open data portal, Ghanaian citizens have to resort to speaking with the government officials in the relevant departments and requesting information from them.

This poses two challenges: 1) It is often unclear which government authority is responsible for executing a certain project; and 2) Citizens have to find the individual within the department who is responsible for the project or any related information. This lack of clarity and accountability often means that citizens’ complaints or requests for information remain unresolved and increase their apathy towards government.”

How it works:

TransGov serves two purposes: 1) Tracking the progress of public projects and sharing relevant details with citizens; and 2) Serving as a platform for citizens to report faults in service delivery (like broken pipes or potholes) and directing those complaints to the competent authority.

1. Monitoring the progress of public projects: TransGov provides a snapshot of important information related to local infrastructure projects. This information includes details like proposed completion dates, funding, contractor information and current status. The fields are populated by a combination of crowdsourced data collection (where users submit pictures and comment on the status of the project) and curated data collection from official sources and interviews with contractors and officials. TransGov also shares this data with Ghana’s national open data portal.^[22] 

Figure 3: Sample project on TransGov Ghana. Source: http://transgovgh.org

2. Reporting complaints: A registered user can post a complaint (including pictures and videos) on TransGov through the web platform or using the mobile app. The complaints are forwarded to the relevant department where officials view the complaints on a dashboard and take action. The department updates the status of the complaint when it is resolved and a notification is sent to the complainant to confirm if it was indeed resolved. To streamline the flow of information to government departments, every complaint is “tagged” and every department’s dashboard only displays the complaints tagged to them.

Institutional support:

The TransGov team’s first challenge was to identify individuals within government departments who had the vision to support TransGov’s efforts. In association with partners like the World Bank office in Ghana, Prince Amin and his team carried out a “power mapping” to strategically find the individuals and departments to work with and then narrowed down their focus to three government departments: 1) The ministry of finance; 2) The ministry of roads and housing; and 3) The Ghana water agency. While they work in tandem with these agencies, there is no dedicated unit or staff within government to deal with the complaints received through TransGov.

Communications strategy:

The second challenge was to inform citizens about TransGov and how it could be used. The team used a combination of online and offline engagement strategies to build the initial user base of TransGov. For example, the team used Facebook ads to spread the word online and laid the groundwork for further promotions. They also organized townhall meetings (often in the presence of local district heads) to educate people about their rights and to demonstrate the platform’s functioning. “A large part of the initial registrations for TransGov came through word of mouth advertising from friends and followers.” 

Impact:

Overcoming people’s apathy and fostering participation was a big challenge for the TransGov team as was getting government to respond more quickly. But TransGov’s success, according to Prince Anim, should be measured by two metrics: 1) Number of issues it resolves; and 2) The timeframe of responses to complaints. TransGov serves as a conduit between citizens and the concerned department. This helps improve interactions between the two but does little to improve the process or pace of resolving the actual complaint. TransGov has taken steps to improve that aspect as well. By posting complaints received on TransGov to social media sites,  the time taken to resolve complaints reduced by nearly 60% since public officials were subjected to the heightened scrutiny. On average it takes 3 days to fix a pothole and 48 hours to fix a burst pipe reported via TransGov compared with nearly a week to fix a pothole and more than 3 days to fix a pipe before TransGov’s social auditing process. There is still plenty of room for improvement.

Problem it solves

In Ghana, there are no official mechanisms to allow citizens to easily find details of local infrastructure projects. For over four years now, the Ghanaian parliament has deliberated on the passage of a right to Information bill but it is unclear when it will materialize. Ghana is part of the Open Government Partnership (OGP) and puts out open data on the national open data portal (data.gov.gh). But even the portal boasts of “133 datasets”, only 15 datasets (mostly census data) are available to view or download and they are rarely updated. Infact, the OGP end-of-term review report in 2017 found^[21] that Ghana had made “limited” progress in its commitment to make datasets publicly accessible on the portal. Without a right to information bill or a robust open data portal, Ghanaian citizens have to resort to speaking with the government officials in the relevant departments and requesting information from them.

This poses two challenges: 1) It is often unclear which government authority is responsible for executing a certain project; and 2) Citizens have to find the individual within the department who is responsible for the project or any related information. This lack of clarity and accountability often means that citizens’ complaints or requests for information remain unresolved and increase their apathy towards government.”

How it works:

TransGov serves two purposes: 1) Tracking the progress of public projects and sharing relevant details with citizens; and 2) Serving as a platform for citizens to report faults in service delivery (like broken pipes or potholes) and directing those complaints to the competent authority.

1. Monitoring the progress of public projects: TransGov provides a snapshot of important information related to local infrastructure projects. This information includes details like proposed completion dates, funding, contractor information and current status. The fields are populated by a combination of crowdsourced data collection (where users submit pictures and comment on the status of the project) and curated data collection from official sources and interviews with contractors and officials. TransGov also shares this data with Ghana’s national open data portal.^[22] 

Figure 3: Sample project on TransGov Ghana. Source: http://transgovgh.org

2. Reporting complaints: A registered user can post a complaint (including pictures and videos) on TransGov through the web platform or using the mobile app. The complaints are forwarded to the relevant department where officials view the complaints on a dashboard and take action. The department updates the status of the complaint when it is resolved and a notification is sent to the complainant to confirm if it was indeed resolved. To streamline the flow of information to government departments, every complaint is “tagged” and every department’s dashboard only displays the complaints tagged to them.

Institutional support:

The TransGov team’s first challenge was to identify individuals within government departments who had the vision to support TransGov’s efforts. In association with partners like the World Bank office in Ghana, Prince Amin and his team carried out a “power mapping” to strategically find the individuals and departments to work with and then narrowed down their focus to three government departments: 1) The ministry of finance; 2) The ministry of roads and housing; and 3) The Ghana water agency. While they work in tandem with these agencies, there is no dedicated unit or staff within government to deal with the complaints received through TransGov.

Communications strategy:

The second challenge was to inform citizens about TransGov and how it could be used. The team used a combination of online and offline engagement strategies to build the initial user base of TransGov. For example, the team used Facebook ads to spread the word online and laid the groundwork for further promotions. They also organized townhall meetings (often in the presence of local district heads) to educate people about their rights and to demonstrate the platform’s functioning. “A large part of the initial registrations for TransGov came through word of mouth advertising from friends and followers.” 

Impact:

Overcoming people’s apathy and fostering participation was a big challenge for the TransGov team as was getting government to respond more quickly. But TransGov’s success, according to Prince Anim, should be measured by two metrics: 1) Number of issues it resolves; and 2) The timeframe of responses to complaints. TransGov serves as a conduit between citizens and the concerned department. This helps improve interactions between the two but does little to improve the process or pace of resolving the actual complaint. TransGov has taken steps to improve that aspect as well. By posting complaints received on TransGov to social media sites,  the time taken to resolve complaints reduced by nearly 60% since public officials were subjected to the heightened scrutiny. On average it takes 3 days to fix a pothole and 48 hours to fix a burst pipe reported via TransGov compared with nearly a week to fix a pothole and more than 3 days to fix a pipe before TransGov’s social auditing process. There is still plenty of room for improvement.