Open Access to Public Policy Evaluations (FR0006)

STAKES Evaluation of public policies helps decision-makers make public action more relevant, effective and efficient and contributes to informing citizens. The stake of evaluation goes beyond those of its direct protagonists and concerns all citizens. Nevertheless, access to these various works remains complex due to the large number of players involved and the multitude of distribution media used.

CONTEXT & AIM The Prime Minister’s circular dated on February 23th, 1989, (10 May 1988/15 May 1991), on the renewal of public service, was an important step in the affirmation of public policy evaluations as a tool for government modernization. Evaluating a public policy means judging its value with regard to a set of criteria, in order to improve and inform decision-making. According to the decree of 18 November 1998, it means "assessing, within an interministerial framework, the effectiveness of this policy by comparing its results to objectives assigned and resources used". Today, public policy landscape is marked by numerous stakeholders: Parliament, Ministries, the Cour des comptes, the Economic, Social and Environmental Council (CESE), the Regional Economic, Social and Environmental Councils, local and regional authorities, State agencies, etc. The diversity of actors involved in evaluations and of media used for publication now raises questions about the legibility and impact of these evaluations in public decisionmaking. Furthermore, citizens aspire to get more involved in the evaluation and design of public policies. The government will undertake a process involving all these actors so that, gradually, citizens will have more visibility on the work carried out, reports will increasingly be made public and contribution of evaluations in public decision-making will be more legible. To do this, the government primarily relies on the best practices of its own approach. 59 evaluations have been launched since the end of 2012, covering numerous public policy fields (social affairs, national education, justice, ecology, housing, etc.). In June 2015, 43 reports were published on the web site of the Secretary-General for Government Modernization12 (SGMAP)(some evaluations are still ongoing). In the dedicated area, a gauge indicates the progress of each evaluation (by large milestones: launch, diagnostics, transformation scenarios, report, implementation), along with the related documentation. The methodology promoted by the SGMAP recommends to systematically involve users benefiting from the evaluated policy (through consultations, polls, ad hoc surveys and workshops), and for stakeholders to be invited to monitor the evaluation work as part of an "evaluation committee". The SGMAP also provided its support and contributed to funding the project run by the French Evaluation Society (SFE), which has been developing, since 2012, a Public Policies Evaluation Observatory. The SFE gathers 300 members (230 individual members and 70 institutional members: State services, local and regional authorities, public institutions, associations and consulting firms). This observatory has already listed more than 1600 references for evaluations made since 2000, and published summaries of corresponding reports for half of them. This initiative is a first step towards the creation of a resource center that will offer an easy access to all evaluation work, in order to help members of Parliament in their mission of drafting bills and controlling the government, as well as decision-makers, public administrators and citizens.

ROADMAP
• Encourage all actors involved in evaluations to make their work available in the Public Policies Evaluation Observatory, to improve its comprehensiveness and facilitate research through its database (access by keyword, etc.)
- Make the Public Policies Evaluation Observatory available to all, as the reference database for public policies evaluations
- Systematize the publication of public policies evaluations launched in the context of government modernization programs (when not prevented by legal considerations)
• Systematize and improve citizens' participation in evaluations coordinated by the SGMAP, for example through opinion surveys, workshops for citizens' juries, etc.
• Improve the traceability of impact on public action of public policies evaluations launched under government modernization programs

4. Open access to evaluations of public policies and to their conclusions

Commitment Text:

ROADMAP

• Encourage all actors involved in evaluations to make their work available in the Public Policies Evaluation Observatory, to improve its comprehensiveness and facilitate research through its database (access by keyword, etc.)

- Make the Public Policies Evaluation Observatory available to all, as the reference database for public policies evaluations

- Systematize the publication of public policies evaluations launched in the context of government modernization programs (when not prevented by legal considerations)

• Systematize and improve citizens' participation in evaluations coordinated by the SGMAP, for example through opinion surveys, workshops for citizens' juries, etc.

• Improve the traceability of impact on public action of public policies evaluations launched under government modernization programs

Editorial Note:This is a partial version of the commitment text. For the full commitment text please see France's national action plan: https://bit.ly/2MTYhsR.

Responsible Institution: Secretary-General for Government Modernisation (SGMAP) attached to the Prime Minister

Supporting Institution(s): N/A

Start Date: Not Specified 

End Date: Not Specified

Commitment Aim

This commitment sought to make public policy evaluation reports more accessible to the public. Public policy evaluations facilitate relevant and efficient public action, and contribute to better-informed citizens. However, access to evaluations is limited due to the number of actors involved and the different methods used. This commitment aimed to make evaluations accessible to the public, encourage the use of evaluation by public officials, and involve citizens in the evaluation of public policies.

To this end, the Secretary-General for Government Modernization (SGMAP) has set up a partnership with the French Evaluation Society (Société Française de l'Évaluation, or SFE), an association which maintains a repository for public policy evaluations. The repository (called the “Observatoire” in French) is a database of public policy evaluations from agencies at different government levels (state, local authorities, etc.). It also gives users the opportunity to submit a report themselves.

Status

Midterm: Limited

Completion of this commitment was limited at the midterm. There was a significant difference in progress between the commitment activities; the public policies evaluation repository was complete while the other two activities, involving citizens in public policy evaluation and encouraging officials' use of the evaluation repository, had not started.

The Public Policies Evaluation Repository was made accessible online, free of charge. An increasing number of evaluations were published during the first year of implementation. The midterm IRM report notes that the search filters were inadequate given the number of evaluations. Importantly, stakeholders signalled that the technology used in the repository was not open source as the software was proprietary. Lastly, the repository did not have a dashboard to follow the number of downloads and contributions.

The midterm IRM report indicates that there was no public evidence that the other two activities had started. For more information, please see the IRM midterm report.^{[Note48: Independent Reporting Mechanism, France Rapport D'ètape (OGP, 2017), https://www.opengovpartnership.org/wp-content/uploads/2001/01/France_Progress-Report_2015-2017.pdf.]}

End of Term: Limited

The government, with SGMAP in the lead, took some steps forward regarding the production and publication of public policy evaluations. However, the government self-assessment does not list any new developments since the midterm assessment. The overall objectives of the commitment were not achieved by the end of the implementation period. Of particular note is the fact that there is no evidence that citizens were given a chance to participate in public policy evaluations nor that the impact of public policy evaluations is more measurable as a result of this commitment.

Based on desk research, the IRM researcher notes that it is now possible to use multiple filters to search through evaluations. However, a significant number of evaluations are not linked to any document or report, making the listing of evaluations less relevant and useful.

Regarding the evaluations launched by the government in 2012 as part of its efforts to modernise public action, 80 evaluations are listed on the dedicated page on the SGMAP's website^{[Note49: Available at http://www.modernisation.gouv.fr/laction-publique-se-transforme/en-evaluant-ses-politiques-publiques/toutes-les-evaluations-de-politiques-publiques.]} and 76 feature a public report. The webpage also contains an explanation of the methodology used to evaluate policies, as well as examples of documents and instruments used during evaluations.

The government self-assessment indicates that an independent evaluation of SGMAP's methodology and assessment of public policies has been undertaken by the auditing firm KPMG and Quadrant Conseil between July 2016 and January 2017. The report is available on the SGMAP's website.^{[Note50: Available at http://www.modernisation.gouv.fr/laction-publique-se-transforme/en-evaluant-ses-politiques-publiques/levaluation-des-politiques-publiques-evaluee-a-son-tour (accessed 9 Jan. 2018).]} The report recommends that:

· transparency and stakeholder participation be reinforced;

· the methodology be more selective in the evaluation themes as not all policies can be evaluated the same way;

· the methodology become more flexible and less time-consuming; and

· that evaluations be combined with other methodologies.

The evaluation highlights the absence of a monitoring mechanism that would allow for an improved follow-up to policy evaluations. It also notes that innovative data collection techniques (e.g. big data, predictive analysis, or citizen consultations) are not used.^{[Note51: SGMAP, L'évaluation des politiques publiques évaluée à son tour : enseignements et perspectives, (SGMAP, 2017), http://www.modernisation.gouv.fr/laction-publique-se-transforme/en-evaluant-ses-politiques-publiques/levaluation-des-politiques-publiques-evaluee-a-son-tour (accessed 22 Sept. 2017); KMPG, Quandrant. Evaluation de la démarche globale d'évaluation des politiques publiques menée dans le cadre de la modernisation de l'action publique, (KMPG, 2017), http://www.modernisation.gouv.fr/sites/default/files/fichiers-attaches/evaluation_epp-map_rapport.pdf.]}

Finally, the IRM researcher did not find any public information that points to the completion of the activity seeking to make the impact of evaluations on policy more traceable.

Did It Open Government?

Access to Information: Marginal

Civic Participation: Did Not Change

The midterm IRM report considered the milestones of this commitment to have a negligible potential impact because a free library of public policy evaluation reports already exists (La documentation Française^{[Note52: Stéphane Saurel, “What is the Budget for the European Union after 2020?” (La Documentation Francaise, accessed 25 Mar. 2018), http://www.ladocumentationfrancaise.fr/.]}) and the activities for increasing citizen participation in SGMAP evaluations and monitoring the evaluations' impact are vague and lack measurable actions. SGMAP has published over 70 public policy evaluations on its website and has started a discussion on how to improve the traceability and impact of public policy evaluations. As such, the implementation of this commitment marginally opened government regarding access to information, given the availability of new information and improvements in the search functions of the Observatory during the second year of implementation.

However, civic participation showed no improvement. The evaluation produced by KPMG and Quadrant highlights the absence of public consultations. During Etalab's forum, civil society noted that public evaluations should gather feedback from different stakeholders using a variety of tools, while a participative evaluation should involve stakeholders starting from the very conception of the evaluation in a steering committee.^{[Note53: “Thomas” (comment), “Commitment 4: Open access to public policy evaluations and their conclusions: Mid-term self-evaluation report of the Action Plan for France 2015-2017: ‘For a transparent and collaborative public action,'' (Etalab, accessed 25 Mar. 2018), https://forum.etalab.gouv.fr/t/engagement-4-ouvrir-lacces-aux-evaluations-de-politiques-publiques-et-a-leurs-conclusions-rapport-dautoevaluation-a-mi-parcours-du-plan-daction-pour-la-france-2015-2017-pour-une-action-publique-transparente-et-collaborative/1905/3.]} He mentions Région Rhône-Alpes, Rennes Métropole and Nantes Métropole as local governments having implemented such methods. Civil society also recommends that evaluations not only be commissioned by public authorities but also by civil society, that evaluators be trained in public consultations to make sure both intermediary bodies and beneficiaries are involved in the process, that a digital evaluation method be piloted at the local level, and that all the data surrounding evaluations be opened.

Carried Forward?

This commitment was not carried over to the new action plan.