Evaluación de la Ley

Evaluación de la Ley is an ex-post evaluation methodology for assessing the effectiveness of active laws led by the Law Evaluation Department in the Chilean Chamber of Deputies using off-line and online focus groups.

The bicameral Congress of Chile is the first national parliament in Latin America to implement a system for evaluating current laws and one of the first in the world to institutionalize an ex-post evaluation mechanism of the effects of legislation.

The Law Evaluation Department is a professional support unit within the administration of the Chamber of Deputies (lower house). It develops and publishes reports that analyze the impact that important laws have had after being in effect over a period of years. Their aim is to assess fulfillment of the objectives that the legislators sought when introducing a law, to detect unintended consequences, to have a record of the citizen's perception of the impact of the law on their lives, and to serve as useful input for updating and revising the evaluated laws.

Created in 2010 by officials from the Chamber of Deputies, the initiative received technical support from the OECD through a program of cooperation and exchange in matters of legal evaluation.^[1] In 2011, the new department developed a plan for how it would work and carried out a pilot to validate the proposed methodology and assess the  relevance of having an ex-post evaluation mechanism.

Although the Law Evaluation Department is part of the administrative structure of the Chamber of Deputies, the decision as to which laws to evaluate is made by the Law Evaluation Committee, a group comprising elected members of Congress from multiple parties that functions similarly to a standing committee and meets every two months.

The Committee prepares the list of potential laws to be evaluated based on the requests they receive from Deputies or other political bodies (standing committees, the Speaker Of The House, etc.). To select the laws chosen for evaluation, the Law Evaluation Department assesses the laws according to selection criteria such as technical feasibility, general applicability of the law, political neutrality, among others, and then presents the filtered list to the Committee for them to agree which law to evaluate next . Due to human resources constraints, the Committee selects no more than 1 or 2 laws each year.

Today the Evaluation Department employs a multidisciplinary team of four full-time professionals. These are trained researchers and facilitators with expertise in both quantitative and qualitative research methods and the moderation of group discussions.

They perform the evaluation of the laws by using a four-month process that unfolds in three stages:

1. Technical study of the law
2. Citizen perception study
3. Final report.

1. Technical Study (6 weeks) - The technical study includes a thorough investigation of the legislative history and parliamentary debate to identify the objectives of the law, the tools chosen by legislators to achieve its goals, the institutions involved in its implementation, and the interest groups and areas of society affected. In this stage, interviews are carried out with specialists and those responsible for the implementation of the law, and indicators for the quantitative measurement of the impact of the law are drawn up.

2. Citizen Perception Study (3 weeks) -To measure the citizen perception of the law, the technical study helps identify the groups of people who will be invited to participate and identifies the most appropriate methodologies according to the type of law: interviews, surveys, focus groups, workshops, citizen meetings, face-to-face forums,debates with experts and/or seminars. The convened participants can come from one or more of the following groups: frontline public officials from the public agencies responsible for the implementation of the law; groups of citizens affected by the law; intermediate organizations that bring together natural or legal persons affected by the law; civil society organizations whose beneficiaries are persons reached by the law; specialist practitioners or academics.

3. Final Report (11 weeks)- Finally, the technical report and citizen perception report are both compiled, and conclusions are drawn about the effects of the application of the regulation. Based on the conclusions, recommendations are made to improve the quality or effectiveness of the regulation, correct unexpected effects, cover legal gaps or expand its scope, among others. The report is delivered to the Law Evaluation Committee for review and the committee president reports on it in a plenary session. Then, it is distributed to the relevant standing committees and to the public and private entities that participated in the evaluation process.

The stages are subject to modification. Given the heterogeneity of the country’s laws, a flexible methodology was designed in which, for each stage, the team can select the activities that it considers most appropriate, according to the characteristics of each law to be evaluated. For example, for some laws it may be necessary to conduct an international law analysis; in cases where it is known in advance that the law is failing to meet its objectives, it is necessary to deepen the study into the possible causes.

When conducting its first pilot evaluation in 2011, the Law Evaluation Department implemented an online forum to collect opinions from self-selected citizens interested in volunteering to participate . However, with insufficient resources and technical support to keep it running and analyze the comments received, the online forum was disabled at the time.

Now, in 2019, the National Congress of Chile is promoting a bidding process to build an online platform called “Virtual Congress” that aims to facilitate interaction between members of the congress and their staff and citizens. One of the features of this future platform will allow the Law Evaluation Department to collect opinions, evidence and expertise from the public in response to targeted questions. The Evaluation Department’s main motivation for incorporating an online, crowdlaw component of this social auditing process is to be able to increase both, the territorial scope and the quantity of citizens consulted.

Between 2011 and 2018, 12 laws have been evaluated collaboratively, for which a total of 305 people were interviewed, 32 focus groups held (11 in the capital city of Santiago and 6 in Valparaíso, headquarters of the parliament and 19 in other cities) in which 391 citizens participated and 55 other citizens participated in workshops and panels with experts. In short, somewhere on the order of 800 people have participated in these activities.

There is no detailed demographic information, however about who took part in those focus groups but anecdotal reporting by the Evaluation Department staff suggests that, mainly white-collar professionals and administrative agency officials participated in interviews. No attempt was made to select representative samples of the population, but rather to gauge the opinion of well-informed people on the subject including experts in the area, members of the teams that implemented the law from the administrative agencies and citizens with knowledge based on lived experience related to the law being evaluated. 

The qualitative inputs provided by the citizen,experts and implementers of the law during these focus groups provide an on-the-ground perspective of the real world impact of legislation that would be missing based only on other instruments, such as opinion surveys and quantitative data analysis.

By contacting both the public agencies responsible for implementing the law and citizens and representatives of civil society affected by it, the Law Evaluation Department team has the opportunity to identify the points of the process in which the implementation of the law is inadequate. In addition, contact is made with a wider and more diverse array of people who may not have been consulted when the  the law was enacted.

All the reports presented contain recommendations for both the executive and legislative branches to improve and update the laws, some of which have been used as input for legal modifications. Eighteen  bills have been introduced, amending existing law and citing the work of the Law Evaluation Department. For example, the report on organ donation was used to prepare amendments to current law.  The report on the Private Data Protection Law was cited by the executive branch when it sought to introduce a bill updating the legislation.

• By contacting both the public agencies responsible for implementing the law and citizens and representatives of civil society affected by it, the team has the opportunity to identify the points of the process in which the implementation of the law is inadequate so that it can be improved.
• The multi-partisan political committee that selects the law to be evaluated from a list prepared by professionals according to technical criteria reduces bias in the selection and gives political support both to the evaluation process and to the reports produced.
• Giving the multi-partisan committee the power to select which law to evaluate and to oversee the Law Evaluation Department work, helped to overcome the initial resistance of some congress members who were afraid that this evaluation process could be aimed to audit or scrutinize their work.
• Interviewing frontline public workers and citizens affected  by the the law allows for an understanding of the "situation on the ground".
• The technical and financial capacity required to operate the process are both relatively low.

• The process does not yet take much advantage of the role that technology could play. Using technology could enable the team to gauge public opinion at a much larger scale and engage people who cannot currently participate.