Developing the Piloting Framework (EE0061)

Brief description of the commitment Make piloting a natural part of policy-making by creating a comprehensive piloting framework (including the legal system, support measures, guidance materials, etc.).

Problem definition
What problem does the commitment aim to address? Countries around the world are looking for new solutions to major social challenges that have not been solved with existing means and tools (e.g. aging population, the climate crisis, etc.). As a result, various ideas and solution variants are increasingly piloted before their implementation. Piloting is a structured process of trying out new ideas on a limited scale, allowing for learning and refinement of ideas before wider implementation. The main reasons why piloting is increasingly being used: ● the desire to try new bold ideas. Piloting makes it possible to test bold ideas on a small scale, which, for example, seem too risky for nationwide implementation in Estonia. ● Better policy-making. Well-designed pilot projects provide insight into what works and what does not, as well as what is the impact of existing interventions. ● Cost savings. Thanks to the lessons learned during the piloting, major and costly mistakes can be avoided during the full-scale implementation of the solution. ● Making real progress. Piloting is one way to move forward with topics that otherwise do not progress beyond endless discussions. ● Increase dialogue and trust through more inclusive and transparent policy-making. Collaboratively designing, implementing, and learning from the pilot projects makes decisions more transparent. ● More agile policy-making. Piloting means step-by-step policy-making that makes it possible to learn from the process and immediately incorporate those lessons.

What are the causes of the problem? Innovative solutions are often not implemented because their full-scale implementation seems too resourceintensive based on the available information. In addition, rigid policy-making processes do not sufficiently support step-by-step and experience-based decision-making, thereby stifling innovation. In the Estonian public sector, the desire and willingness to pilot solutions has grown, but there is a lack of experience and knowledge. Piloting requires some transformation of the usual policy-making process and the ability to prepare and conduct the pilot project.

Commitment description
What has been done so far to solve the problem? So far, piloting has not been very common in the Estonian public sector. Instead, it has mostly been used in individual projects or in small teams. For example, the public sector innovation team applies behavioural scientific approaches, including nudging, which have also been applied in individual projects by the Ministry of Social Affairs, the Ministry of Justice, the Ministry of the Interior, the Tax and Customs Board, and others. Accelerate Estonia pilots private sector solutions that help solve public sector concerns.

What solution are you proposing? Piloting is a structured process of trying out new ideas and different possible solutions on a limited scale, allowing for learning and developing the solution before wider implementation. By enabling smarter choices to be made with limited resources, efforts are being made to find more flexible and open ways instead of rigid processes to solve increasingly complex public sector problems. Rather than implementing broad social reforms without knowing whether they work or not, piloting allows for learning as reforms are developed or implemented. Thus, piloting makes it possible to try bold ideas and balance the risks associated with their implementation by letting us know what expected and unexpected consequences the measures will bring.

What results do we want to achieve by implementing this commitment? Make piloting a natural part of policy-making by creating a comprehensive piloting framework (including the legal system, support measures, guidance materials, etc.). Piloting needs a supportive institutional framework and a culture that favours risk-taking, in addition to support structures and the ability to implement pilot projects. The institutional framework must ensure that the state always pilots the solutions legally, ethically, transparently, and methodically, and support the agile implementation of pilot projects. Therefore, we need a legal framework that supports piloting, as well as agreed principles of ethics and legality and a body assessing the legality and ethicality of pilot projects. We also need to introduce piloting as part of the impact assessment process.

Commitment analysis
How will the commitment promote transparency? Piloting creates new knowledge for both policy-makers and the people affected by a particular issue. The knowledge and data obtained as a result of the pilot project are generally public, and thus the transparency of policy decisions is improved.

How will the commitment help foster accountability? How will the commitment improve citizen participation in defining, implementing, and monitoring solutions? As the pilot projects are carried out with a limited target group, within a predetermined period of time, and in cooperation with important parties, it also increases the co-creation of policy making.

Commitment planning Milestones Expected outputs Expected completion date Stakeholders

Launching the support measure for pilot projects The grant conditions have been published and the acceptance of applications is open December 2022 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Joint organisation of Enterprise Estonia and KredEx

Description of piloting as one possible part of impact analysis methods Supplementing the supporting materials for impact assessment methodology December 2022 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Ministry of Justice

Piloting instruction material with examples The instructional material has been made public January 2023 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Ministry of Justice

Analysis and proposals: Memorandum for the cabinet meeting January 2023 Lead: Government Office 1. How to assess the ethics, legality, and quality of public sector pilot projects and under what conditions is such an assessment necessary? 2. On the need for changes in the legislative drafting process and/or practices so that piloting can be done in an agile, legitimate, and ethical manner. Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Ministry of Justice Estonian Research Council

The examples and lessons learned from Estonian pilot projects so far have been collected and published, including lessons learned from carrying out the piloting process on the example of seven projects in 2022 The instructional material has been updated December 2023 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Ministry of Economic Affairs and Communicati ons Joint organisation of Enterprise Estonia and KredEx

Introduction of piloting on a larger scale into the trainings of (future) top managers and into the competency model and trainings of middle managers The competency models have been updated December 2023 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Ministry of Finance

Action Plan Review

Commitment 2.1 Developing the Piloting Framework

● Verifiable: Yes

● Does it have an open government lens? Yes

● This commitment has been clustered as: Fostering evidence-based decision-making (activities 2.1 and 2.2 of the action plan)

● Potential for results: Substantial

Government Office, all ministries, Statistics Estonia, Data Protection Inspectorate

For a complete description of the activities included in this commitment, see activities 2.1 and 2.2 in the action plan here.

Context and objectives:

This commitment has two main drivers. First, the government believes that novel solutions to complex public policy problems are easier to implement if evidence of their impacts can be generated at a small scale before investing in large-scale implementation. [20] At the same time, the increasing datafication of society puts pressure on the government to use data to create public value. [21] The Government Office plans to support a shift to policy-making that relies less on decision makers’ subjective perceptions and more on data and evidence. [22] Although vast amounts of potentially useful data exist both in public databases and private sources, there are gaps in public officials’ data literacy [23] and the use of data and evidence to forecast the impacts of policies remains limited. [24] According to Estonia’s Digital Strategy for 2030, the public lacks information on the data and models used to make public policy decisions, which decreases the transparency of public governance and may fuel the spread of disinformation. The limited findability and uneven quality of the data stored in various databases further complicates the use of data in policy-making. [25]

This commitment consists of two activities that support evidence-based policy-making. Both activities could improve government transparency by enabling the public to see what evidence led the government to adopt certain decisions or policies.

The first (2.1) foresees the development of a policy framework to support the use of systematic experimentation and piloting in policy-making, i.e., testing policy solutions in small-scale pilots and documenting their impacts based on a clear methodology. [26] Specifically, it involves including piloting in the government’s methodological guidelines for regulatory impact assessment and launching a funding program with a budget of 60 million EUR to support policy experiments conducted with researchers. [27] It also foresees publishing guidelines with success and failure stories that organizations can learn from, analyzing measures to assess the lawfulness and ethical aspects of pilots, as well as analyzing the legal and procedural changes needed to enable widespread implementation of piloting in the public sector. The government also plans to integrate this topic in public service top and middle managers’ training programs.

The second activity (2.2) seeks to develop a digital tool that would perform automated analysis of the vast amounts of data that can inform policy, in particular to assist the preparation of government memoranda. [28] Such data includes public sector databases and document management systems, text corpora including meeting minutes and memos, public research data, and big data collected by private companies. [29] In the future, the automated analysis tool could be integrated with the government’s legislative drafting and co-creation tool. [30] The government is applying a step-by-step approach, starting from data and functionalities that are easiest to integrate. The milestones include delivering a roadmap for technical development, engaging CSOs to improve the solution, and implementing first steps of the roadmap. The plan is to continue the commitment in future action plans. According to the Government Office, the first prototype will likely include a search engine of publicly available data from various web and media sources to help map a topic of interest. [31]

Potential for results:Substantial

Although previous action plans have not included commitments to promote evidence-based policy-making, Estonia is not starting from scratch. Since its establishment in 2018, the government’s inter-departmental innovation unit has worked to develop a culture of experimentation in the public sector and has recently mapped more than 70 public sector-led initiatives that have involved some degree of piloting. For example, in 2019, the municipality of Saaremaa tested ways to nudge residents to sort packaging waste. [32] In three consecutive summers, the city of Tartu temporarily transformed its traffic-heavy central streets into car-free zones, measuring noise and traffic levels and observing people’s mobility patterns. [33] However, understanding of experimentation as a policy-making method is uneven across the public sector and organizations’ willingness to pilot innovative solutions depends on whether they have champions of piloting. [34]

Activity 2.1’s comprehensive approach to fostering the use of policy experiments can drive actual changes in policy-making practices. However, widespread adoption of experimentation will likely require the accumulation of positive experiences over time and a gradual change of organizational cultures to favor innovation over fear of failure. Nonetheless, the Government Office’s plan to present the results of the legal landscape analysis to government ministers will likely strengthen the impact of the commitment. According to the innovation unit, it is vital to engage political decision makers, so that they can initiate strategic policy experiments themselves. [35] Moreover, the Government Office notes that the size of the government’s funding program for financing the pilot implementation is notable, considering the size of Estonia. The Government Office aims to engage all ministries as well as more capable local municipalities with several large-scale policy experiments, because of which the Government Office expects permanent cultural change. [36]

Activity 2.2 is ambitious but somewhat techno-optimistic in its vision of data-driven decision-making and automated preparation of government decisions. While the Government Office’s long-term vision is to fully automate data collection, analysis, and preparation of proposals to the cabinet, they regard the activity as experimental in nature. [37] Since policy decisions often concern complex problems and making value choices, focusing on good data analytics may be a more realistic objective than expecting the tool to be able to suggest decisions based on data. Nevertheless, since no similar tools exist in the Estonian public sector, the activity will likely increase data-driven decision-making, even if its functionalities end up being limited to simpler search and analytics functions.

The IRM considers this commitment to have substantial potential results. This is because activity 2.1 includes a comprehensive set of measures to help institutionalize the use of experimentation in policy-making: a legal review, a generous funding program, guidelines and methodologies, and advice to implementers, However, the objective to shift to automated data-driven decision-making in the government (activity 2.2) raises ethical issues that warrant more thorough discussions with civil society and experts before large-scale application. Furthermore, the national statistical office has noted that the activity’s current scope is limited to the Government Office’s decision-making processes but does not include clear mechanisms to support data-driven decision making in other government institutions. [38] They are also concerned that using unstructured data of varying quality from diverse sources may complicate rather than simplify public decision-making processes. In the long-term, however, activities 2.1 and 2.2 could serve as important preliminary steps towards institutionalizing evidence-based policy-making in the public sector.

Opportunities, challenges, and recommendations during implementation

Regarding the institutionalization of experimenting and piloting as part of policy-making routines (activity 2.1), the main challenges are to ensure broad awareness of the method among public officials both in the central and local government and their capacity to carry out pilots. In certain policy areas, such as those involving social policy, minorities, and marginalized groups, experiments may also run into legal impediments. The IRM recommends the following to support successful implementation:

• Engage experts to develop guidelines and solutions for designing ethical experiments. As some experiments may affect people’s fundamental rights and equal treatment, strong ethical and legal guidance is needed to design experiments in a responsible way. The government is already planning to tap into the expertise that exists in universities’ research ethics committees and potentially use these committees to assess the ethical aspects of pilots before implementation. [39] When designing guidelines and instruments for ethical assessment, the government could also consult experts in human rights and administrative law to account for the public sector context. One of the experts working on the guidelines is an expert in human rights (who previously worked in the Chancellor of Justice). Also, the team is planning wider discussion on ethics as part of the process. [40]
• Allocate resources to active awareness raising and capacity building to ensure take-up of the results. The government plans to promote the guidelines among the applicants of the funding program for pilots. The guidelines could also be disseminated in public service trainings. Both government ministries and municipalities could benefit from structured experience-sharing with their peers and practical workshops where those with no prior experience could learn from others’ success and failure stories. The government could design a capacity-building and peer learning program to facilitate such exchange of experience. According to the Government Office, the necessary resources (budget, personnel, and public service training sessions) are allocated in 2023's work plan of the public sector innovation team and Strategy Unit at the Government Office. [41]

Data integration projects can be challenging due to problems with data quality and accuracy, lack of technical and semantic interoperability, legal barriers to data access and reuse, and transaction costs related to negotiating data access agreements with private data holders. Therefore, the digital decision support tool (activity 2.2) may face challenges that delay or limit its usefulness by excluding data that may be valuable but too complicated to integrate. The barriers may be even higher regarding the automated interpretation of the data. While AI-driven data processing and analytics technologies can make sense of diverse data, the challenge is to determine to what extent the results can be trusted as a basis of making public decisions, and who has the capacity to catch possible errors in the data or algorithms. When implementing this activity, the Government Office could consider the following recommendations:

• Plan thorough legal and feasibility analysesto anticipate possible legal and technical barriers. According to the Government Office, the roadmap that is currently being developed also involves a legal analysis. It is important to plan concrete actions to start addressing the identified barriers as soon as this analysis becomes available.
• Ensure the quality of the data used to inform public policy decisions. There is likely a trade-off between integrating as many data sources as possible and maintaining control over data quality. However, in policy issues of high importance or sensitivity, the latter may be more important. The government could also consider involving independent experts in assessing the quality of the algorithms used in the tool. Moreover, although the Government Office’s long-term goal is to automate the preparation of proposals to the cabinet, it will be important to maintain a level of human judgement in the decision-making process.
• Ensure public transparency of the data and AI are used to inform government decisions. The Government Office intends to make the tool at least partly open for public use. Whereas there may be legal impediments to public access to the data or technical limitations to the volume of simultaneous data requests that the system can handle, the search engine can be made accessible to anyone. [42] The government could also aim to open the datasets integrated to the tool to the extent legally possible and make it clear to the public when AI has been used to inform government decisions. The government could create an obligation that all memoranda presented to the government include an overview of the data used to prepare them. Since the memoranda discussed in the cabinet meetings are not public by law, the government could analyze if the memoranda that do not concern sensitive issues could be made fully or partly public.
• Engage CSOs and experts on AI ethics to develop the tool. The action plan foresees the engagement of CSO stakeholders in discussing the roadmap to identify their needs and possible problems. It could also be useful to engage researchers and experts on ethical and explainable AI to discuss ways of ensuring the transparency and public understandability of the models and algorithms used for automated data analysis. In addition, the government could develop a mechanism for CSOs and the public to raise concerns about government decisions that were informed by data analysis and AI.