Supporting Data-Driven Decision-Making In Policy-Making (EE0062)

Brief description of the commitment Create a technological solution that supports data-based decision-making integrated into decision-making processes, so that the use of data in policy-making is transparent, simple, and fast.

Problem definition
What problem does the commitment aim to address? In policy-making, the use of all available information related to the issue is difficult and time-consuming in the preparation of decisions. Having to take into account a large amount of information is overwhelming for the decision-makers. In practice, only information which was known and available to the drafter, which they knew how to use, and which they considered important is presented as the background and justification of the decision proposal. Even if subjective factors have not significantly influenced the presented background and arguments, it is often not possible to make sure of this at the time of making a decision or even later. Therefore, it is possible that a decision is made in good faith and based on the information provided, which in reality does not take into account all important aspects. Sometimes, the decision is criticised even if all important aspects have been taken into account. The commitment focuses on increasing the data-basedness of the final stage of the decision-making process of the Government of the Republic. Approximately 1,500 decisions are made annually at government cabinet meetings and sessions. The preparation of draft decisions is time-consuming, but a significant part of the preparation may not be used in the decision-making process. Studying a highly detailed document requires an excessive amount of time from a member of the government while a document with a high level of generalisation does not allow for a sufficient perception of the problem. As different ministers are interested in details in different aspects, even in those that the author of the document may not consider important, the only possible solution is to dynamically change the information and the analysis underlying it.

What are the causes of the problem? At the proposal of the Government Office, the Government Data-Driven Decision-Making Framework Implementation project was initiated in 2022. Its external consultant PwC conducted interviews with ministries, Statistics Estonia, the Estonian Data Protection Inspectorate, and data specialists in the first half of the year. Based on the interviews, the following reasons were identified as bottlenecks affecting the effectiveness, transparency, and speed of decision-making in policy-making:  skills in identifying, collecting, processing, and analysing relevant data are rather poor;  access to registers and databases is limited; 26  data that is known and accessible is used;  private sector data is difficult to use;  data quality is variable and in some cases poor;  it is difficult to combine different data sets for analysis purposes;  getting an overview of the background of the topic (memorandums, studies, analyses) is difficult and timeconsuming. Based on the above-mentioned mapping, the problem description and scope of this commitment have been formulated. The final results of the analysis of Government Data-Driven Decision-Making Framework Implementation will be completed in February 2023, and then, it is planned to specify the next steps in cooperation with ministries and partners (if necessary, to adapt and/or expand the terms of reference). When planning and implementing this commitment, both the broader view of the data-based decision-making process and other main activities in the field are kept in mind throughout to ensure their interconnection (including technological solutions).

Commitment description
What has been done so far to solve the problem? So far, the main focus has been on the more thoughtful organisation of data management and state databases. Attention has also been paid to increasing the competence of officials who prepare decision proposals. What has been done so far has the potential to solve the described problems only partially. For example, it does not allow handling unstructured data sets, systematic search and presentation of information from them, does not allow ensuring a significant increase in the speed of background information mapping, the integrity of the created background, etc. A situation where all the basic data is in order and all the people dealing with the subject are sufficiently competent is not likely. The continuation of what has been done so far and the planned commitment are therefore not in competition. Instead, they support and empower each other. Better organisation of basic data and increasing the competence of users allow for more effective use of potential technological opportunities for databased decision-making. The creation of an output, the operation of which the officials themselves are interested in, in turn motivates them to solve issues related to data management in a more purposeful and sustainable manner.

What solution are you proposing? As a solution, we plan to automate the preparation of draft decisions (especially memoranda submitted to the Government of the Republic, as well as others) in a way where related data and information are aggregated, preprocessed, and visualised in a way that is easier for people to understand. As a result, the first draft of the decision is prepared without significant user intervention. For example, the data underlying a memorandum is aggregated and the results of its visualised analysis are presented automatically. The user will review the document and, if necessary, correct and supplement it. The proposed solution allows to speed up the decision-making process, reduce the unintentional or intentional subjectivity of the preparation of draft decisions, and increase the transparency of the decision-making process. For example, the automated solution enables government cabinet meetings and sessions to receive data-based answers to questions that arise on an ongoing basis. These questions may require separate preparation time and may therefore be postponed to the future. The ideal solution is complex and it is reasonable to develop it in stages. The mentioned problems can be significantly alleviated already in the first stages, i.e. with a solution for the automatic aggregation, systematisation, and visualisation of data. This solution is a combination of the results of initiatives implemented today (data management, open data information gateway, development of language technology, etc.). The goal is not necessarily to create a new user environment, especially if the goals set in the road map completed in February are achieved by adding individual additional functions to some existing or in-development technological solutions (e.g. the data reuse environment of Statistics Estonia).

What results do we want to achieve by implementing this commitment? Drafters of draft decisions have at their disposal a working tool that significantly reduces the time spent searching for information related to the preparation of draft legislation, supports the inclusion of all relevant data, makes it more transparent what information was used and what information was left unused, and creates decision alternatives that are based only on rational considerations. When developing the tool, it is taken into account that all technological platforms used in policy-making and proceedings (including the co-creation workspace, session information system, etc.) must be connected to each other as much as possible.

Commitment analysis
How will the commitment promote transparency? The information that has been scattered so far is aggregated automatically, which makes information that decision-makers, implementers, and other interested parties were not aware of or could not find available to them. The information is made easier for humans to perceive automatically, which allows meaningful reading of large data sets and data with more complex relationships. In addition, understanding information in this way is less affected by the level of human data processing skills. This approach also ensures a visible trace of the data used for the preparation of the decision, related options, etc., on the basis of which it is possible to assess whether all relevant information was taken into account during the decision-making or later.

How will the commitment help foster accountability? The basic data related to the decision and the broader background of the draft decision as well as the decision proposal based on rational considerations are presented. The possibility that some relevant data will not be used only for subjective reasons (e.g. the decision-makers were not aware of their existence) is reduced. Possible contradictions between decisions and background data stand out more clearly and it is possible to take this into account more easily when making a decision.

How will the commitment improve citizen participation in defining, implementing, and monitoring solutions? The basic data of the decision and the connection of the decision therewith becomes observable and understandable even if the person interested in the topic lacks the knowledge and skills to search for data and process it as information. This enables a databased view of the problem and makes it easier to present alternative proposals based on rational arguments.

Commitment planning Milestones Expected outputs Expected Stakeholders 28 completion date

Development of a solution supporting data-based decision-making A description of the future version of the solution, a roadmap to get there, and a prototype March 2023 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) All ministries, Statistics Estonia, Data Protection Inspectorate OECD, European Commission

A seminar or other engagement format for supplementing the developed solution with non-governmental organisations An enhanced version of the solution supporting data-driven decision-making June 2023 Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) Network of Estonian Nonprofit Organisation s, eGovernance Academy

Development of a solution based on the roadmap in accordance with the prescribed stages Those stages of the solution based on the roadmap which are feasible with the available resources have been implemented June 2024 (may continue in the next action plan) Lead: Government Office Supporting stakeholders Government CSOs Others (e.g. parliament, private sector, etc.) All ministries Network of 29 Estonian Nonprofit Organisation s, eGovernance Academy

Action Plan Review

Commitment 2.2 Supporting Data-Driven Decision-Making In Policy-Making

● Verifiable: Yes

● Does it have an open government lens? Yes

● This commitment has been clustered as: Fostering evidence-based decision-making (activities 2.1 and 2.2 of the action plan)

● Potential for results: Substantial

Government Office, all ministries, Statistics Estonia, Data Protection Inspectorate

For a complete description of the activities included in this commitment, see activities 2.1 and 2.2 in the action plan here.

Context and objectives:

This commitment has two main drivers. First, the government believes that novel solutions to complex public policy problems are easier to implement if evidence of their impacts can be generated at a small scale before investing in large-scale implementation. [20] At the same time, the increasing datafication of society puts pressure on the government to use data to create public value. [21] The Government Office plans to support a shift to policy-making that relies less on decision makers’ subjective perceptions and more on data and evidence. [22] Although vast amounts of potentially useful data exist both in public databases and private sources, there are gaps in public officials’ data literacy [23] and the use of data and evidence to forecast the impacts of policies remains limited. [24] According to Estonia’s Digital Strategy for 2030, the public lacks information on the data and models used to make public policy decisions, which decreases the transparency of public governance and may fuel the spread of disinformation. The limited findability and uneven quality of the data stored in various databases further complicates the use of data in policy-making. [25]

This commitment consists of two activities that support evidence-based policy-making. Both activities could improve government transparency by enabling the public to see what evidence led the government to adopt certain decisions or policies.

The first (2.1) foresees the development of a policy framework to support the use of systematic experimentation and piloting in policy-making, i.e., testing policy solutions in small-scale pilots and documenting their impacts based on a clear methodology. [26] Specifically, it involves including piloting in the government’s methodological guidelines for regulatory impact assessment and launching a funding program with a budget of 60 million EUR to support policy experiments conducted with researchers. [27] It also foresees publishing guidelines with success and failure stories that organizations can learn from, analyzing measures to assess the lawfulness and ethical aspects of pilots, as well as analyzing the legal and procedural changes needed to enable widespread implementation of piloting in the public sector. The government also plans to integrate this topic in public service top and middle managers’ training programs.

The second activity (2.2) seeks to develop a digital tool that would perform automated analysis of the vast amounts of data that can inform policy, in particular to assist the preparation of government memoranda. [28] Such data includes public sector databases and document management systems, text corpora including meeting minutes and memos, public research data, and big data collected by private companies. [29] In the future, the automated analysis tool could be integrated with the government’s legislative drafting and co-creation tool. [30] The government is applying a step-by-step approach, starting from data and functionalities that are easiest to integrate. The milestones include delivering a roadmap for technical development, engaging CSOs to improve the solution, and implementing first steps of the roadmap. The plan is to continue the commitment in future action plans. According to the Government Office, the first prototype will likely include a search engine of publicly available data from various web and media sources to help map a topic of interest. [31]

Potential for results:Substantial

Although previous action plans have not included commitments to promote evidence-based policy-making, Estonia is not starting from scratch. Since its establishment in 2018, the government’s inter-departmental innovation unit has worked to develop a culture of experimentation in the public sector and has recently mapped more than 70 public sector-led initiatives that have involved some degree of piloting. For example, in 2019, the municipality of Saaremaa tested ways to nudge residents to sort packaging waste. [32] In three consecutive summers, the city of Tartu temporarily transformed its traffic-heavy central streets into car-free zones, measuring noise and traffic levels and observing people’s mobility patterns. [33] However, understanding of experimentation as a policy-making method is uneven across the public sector and organizations’ willingness to pilot innovative solutions depends on whether they have champions of piloting. [34]

Activity 2.1’s comprehensive approach to fostering the use of policy experiments can drive actual changes in policy-making practices. However, widespread adoption of experimentation will likely require the accumulation of positive experiences over time and a gradual change of organizational cultures to favor innovation over fear of failure. Nonetheless, the Government Office’s plan to present the results of the legal landscape analysis to government ministers will likely strengthen the impact of the commitment. According to the innovation unit, it is vital to engage political decision makers, so that they can initiate strategic policy experiments themselves. [35] Moreover, the Government Office notes that the size of the government’s funding program for financing the pilot implementation is notable, considering the size of Estonia. The Government Office aims to engage all ministries as well as more capable local municipalities with several large-scale policy experiments, because of which the Government Office expects permanent cultural change. [36]

Activity 2.2 is ambitious but somewhat techno-optimistic in its vision of data-driven decision-making and automated preparation of government decisions. While the Government Office’s long-term vision is to fully automate data collection, analysis, and preparation of proposals to the cabinet, they regard the activity as experimental in nature. [37] Since policy decisions often concern complex problems and making value choices, focusing on good data analytics may be a more realistic objective than expecting the tool to be able to suggest decisions based on data. Nevertheless, since no similar tools exist in the Estonian public sector, the activity will likely increase data-driven decision-making, even if its functionalities end up being limited to simpler search and analytics functions.

The IRM considers this commitment to have substantial potential results. This is because activity 2.1 includes a comprehensive set of measures to help institutionalize the use of experimentation in policy-making: a legal review, a generous funding program, guidelines and methodologies, and advice to implementers, However, the objective to shift to automated data-driven decision-making in the government (activity 2.2) raises ethical issues that warrant more thorough discussions with civil society and experts before large-scale application. Furthermore, the national statistical office has noted that the activity’s current scope is limited to the Government Office’s decision-making processes but does not include clear mechanisms to support data-driven decision making in other government institutions. [38] They are also concerned that using unstructured data of varying quality from diverse sources may complicate rather than simplify public decision-making processes. In the long-term, however, activities 2.1 and 2.2 could serve as important preliminary steps towards institutionalizing evidence-based policy-making in the public sector.

Opportunities, challenges, and recommendations during implementation

Regarding the institutionalization of experimenting and piloting as part of policy-making routines (activity 2.1), the main challenges are to ensure broad awareness of the method among public officials both in the central and local government and their capacity to carry out pilots. In certain policy areas, such as those involving social policy, minorities, and marginalized groups, experiments may also run into legal impediments. The IRM recommends the following to support successful implementation:

• Engage experts to develop guidelines and solutions for designing ethical experiments. As some experiments may affect people’s fundamental rights and equal treatment, strong ethical and legal guidance is needed to design experiments in a responsible way. The government is already planning to tap into the expertise that exists in universities’ research ethics committees and potentially use these committees to assess the ethical aspects of pilots before implementation. [39] When designing guidelines and instruments for ethical assessment, the government could also consult experts in human rights and administrative law to account for the public sector context. One of the experts working on the guidelines is an expert in human rights (who previously worked in the Chancellor of Justice). Also, the team is planning wider discussion on ethics as part of the process. [40]
• Allocate resources to active awareness raising and capacity building to ensure take-up of the results. The government plans to promote the guidelines among the applicants of the funding program for pilots. The guidelines could also be disseminated in public service trainings. Both government ministries and municipalities could benefit from structured experience-sharing with their peers and practical workshops where those with no prior experience could learn from others’ success and failure stories. The government could design a capacity-building and peer learning program to facilitate such exchange of experience. According to the Government Office, the necessary resources (budget, personnel, and public service training sessions) are allocated in 2023's work plan of the public sector innovation team and Strategy Unit at the Government Office. [41]

Data integration projects can be challenging due to problems with data quality and accuracy, lack of technical and semantic interoperability, legal barriers to data access and reuse, and transaction costs related to negotiating data access agreements with private data holders. Therefore, the digital decision support tool (activity 2.2) may face challenges that delay or limit its usefulness by excluding data that may be valuable but too complicated to integrate. The barriers may be even higher regarding the automated interpretation of the data. While AI-driven data processing and analytics technologies can make sense of diverse data, the challenge is to determine to what extent the results can be trusted as a basis of making public decisions, and who has the capacity to catch possible errors in the data or algorithms. When implementing this activity, the Government Office could consider the following recommendations:

• Plan thorough legal and feasibility analysesto anticipate possible legal and technical barriers. According to the Government Office, the roadmap that is currently being developed also involves a legal analysis. It is important to plan concrete actions to start addressing the identified barriers as soon as this analysis becomes available.
• Ensure the quality of the data used to inform public policy decisions. There is likely a trade-off between integrating as many data sources as possible and maintaining control over data quality. However, in policy issues of high importance or sensitivity, the latter may be more important. The government could also consider involving independent experts in assessing the quality of the algorithms used in the tool. Moreover, although the Government Office’s long-term goal is to automate the preparation of proposals to the cabinet, it will be important to maintain a level of human judgement in the decision-making process.
• Ensure public transparency of the data and AI are used to inform government decisions. The Government Office intends to make the tool at least partly open for public use. Whereas there may be legal impediments to public access to the data or technical limitations to the volume of simultaneous data requests that the system can handle, the search engine can be made accessible to anyone. [42] The government could also aim to open the datasets integrated to the tool to the extent legally possible and make it clear to the public when AI has been used to inform government decisions. The government could create an obligation that all memoranda presented to the government include an overview of the data used to prepare them. Since the memoranda discussed in the cabinet meetings are not public by law, the government could analyze if the memoranda that do not concern sensitive issues could be made fully or partly public.
• Engage CSOs and experts on AI ethics to develop the tool. The action plan foresees the engagement of CSO stakeholders in discussing the roadmap to identify their needs and possible problems. It could also be useful to engage researchers and experts on ethical and explainable AI to discuss ways of ensuring the transparency and public understandability of the models and algorithms used for automated data analysis. In addition, the government could develop a mechanism for CSOs and the public to raise concerns about government decisions that were informed by data analysis and AI.