Define an Approach for Measuring Open Government Performance (CA0046)

Define an Approach for Measuring Open Government Performance Why do this: Open government is a relatively new way of doing things, and there is no single, common, and agreed-upon methodology to measure progress on all aspects that are important to Canada. As Canada defines its goals, it will also have to define how it wants to measure them, identifying interim steps towards longer term objectives. How will it be done: Government departments and agencies are publishing their plans for releasing data and information. The Government will use these plans to help measure progress on improving openness and transparency. Further analysis will be completed as part of the development of an overall Performance Management Framework for government-wide information management. Work will also be done to better measure progress on broad open government efforts.

5. Define an Approach for Measuring Open Government Performance

Commitment Text:

The Government of Canada will integrate performance indicators for openness and transparency into a Performance Management Framework for Open Government.

Milestones:

5.1. Integrate key performance indicators related to openness and transparency as part of a Performance Framework for managing data and information government-wide.

5.2. Measure and report publically on annual departmental progress on implementation of the Directive on Open Government.

5.3. Work on developing a performance management framework and indicators that can better measure a wider breadth of Open Government efforts and outcomes.

Responsible institution: Treasury Board of Canada Secretariat

Supporting institution(s): N/A

Start date: Not specified

End date: Not specified

Editorial Note: The text of the commitment was abridged for formatting reasons. For full commitment text, visit: http://www.opengovpartnership.org/wp-content/uploads/2001/01/Canada_AP3.pdf.

Context and Objectives

This commitment aims to address a lack of unified and objective indicators for measuring open government progress in Canada by creating a performance management framework for open government. It is easy for governments to point to thousands of datasets which are being released as proof of expanding openness, but more difficult to consider the overall impact of their efforts. Both milestones 5.1 and 5.2 are essentially duplicative to existing reporting requirements under the OGP, though Milestone 5.1 provides a stronger model for implementation of this responsibility. Milestone 5.3 could have a more substantial potential impact, since it could allow for a thorough and contextual assessment of the impacts of openness. However, all three milestones, and in particular Milestone 5.3, are phrased relatively vaguely, leaving room for interpretation as to how the reporting will take place, and what level of implementation is anticipated under this action plan. Moreover, in order to have relevance to OGP values, and in particular the value of public accountability, the indicators and reporting contained in this commitment need to be both publicly facing and responsive to public inputs. As a consequence, the commitment is currently marked as unclear relevance, though this is not to deny its potential value as a mechanism for promoting progressive improvement. The lack of clarity, as well as the relatively duplicative nature of milestones 5.1 and 5.2, lead to a scoring of only minor potential for impact overall.

Completion

In relation to Milestone 5.1, the Treasury Board of Canada Secretariat has developed and published an online progress tracker for its OGP commitments.[Note55: Available at: http://open.canada.ca/en/content/progress-tracker-third-biennial-plan-open-government-partnership.] The self-assessment implies that further progress will be limited to quarterly updates of the tracker, meaning that the overall impact of this milestone will be only marginally beyond the government’s existing reporting requirements under the OGP. Under this reading, the commitment is on schedule.

Milestone 5.2 also appears duplicative, as the language in the government’s self-assessment is almost identical to the reporting under Milestone 3.1. As a result, the second milestone is indeed on schedule, but the progress appears to be duplicative of other milestones.

According to the government’s self-assessment, as well as consultations with lead officials on this commitment, Milestone 5.3 is on schedule.[Note56: The draft self-assessment is available at: http://open.canada.ca/en/mtsar/draft-consultation-mid-term-self-assessment-third-biennial-plan-open-government-partnership.] The government has researched existing global indexes and developed a draft framework for the work. It is currently working to collaborate through the Open Government Working Group, as well as engaging with other governments, and the OECD, to discuss better practices for measuring impact. A work plan has also been developed and published.[Note57: The working plan is available for download at: https://gccollab.ca/file/view/62638/en1-pager-on-performance-measurementfr1-pager-on-performance-measurement (registration required).]

Next Steps

At a stakeholder consultation, a representative of Open North noted the importance of measuring impacts and outcomes, rather than purely looking at outputs.[Note58: Montreal consultation, 20 September 2017.] Supplying data is just the starting point, they noted, the real question is whether governments are adding value down the production line. Milestone 5.3 of this commitment is of substantial importance in this regard. Although the current phrasing in the action plan is somewhat vague regarding the level of progress which is anticipated under the current cycle, the IRM researcher recommends carrying this project through to the final development of a performance management framework, indicators and their application to the government’s actual progress in parallel to OGP reporting.

5. Define an Approach for Measuring Open Government Performance Commitment

Commitment Text: The Government of Canada will integrate performance indicators for openness and transparency into a Performance Management Framework for Open Government.

Milestones:

5.1. Integrate key performance indicators related to openness and transparency as part of a Performance Framework for managing data and information government-wide.

5.2. Measure and report publically on annual departmental progress on implementation of the Directive on Open Government.

5.3. Work on developing a performance management framework and indicators that can better measure a wider breadth of Open Government efforts and outcomes.

Responsible institution: Treasury Board of Canada Secretariat

Supporting institutions: N/A

Start Date: Not specified

End Date: Not specified

Editorial Note: The text of the commitment was abridged for formatting reasons. For full commitment text, visit http://www.opengovpartnership.org/wp-content/uploads/2001/01/Canada_AP3.pdf.

Commitment Aim

This commitment aimed to address a lack of unified and objective indicators for measuring open government progress in Canada. It called for the creation of a performance management framework. The specific milestones included developing a performance framework featuring key indicators, measuring implementation of the “Directive on Open Government,” and developing a performance management framework and indicators to assess impacts and outcomes from open government initiatives.

Status

Midterm: Substantial

In June 2017, the midterm assessment rated the overall progress on this commitment substantial. That progress includes the publication of an online progress tracker for OGP commitments (Milestone 5.1),[Note40: Available at “Progress Tracker for the Third Biennial Plan to the Open Government Partnership,” Government of Canada, http://open.canada.ca/en/content/progress-tracker-third-biennial-plan-open-government-partnership.] the publication of open data inventories (Milestone 5.2), and development of a draft performance management framework and working plan for implementation (Milestone 5.3).[Note41: The working plan is available for download at https://gccollab.ca/file/view/62638/en1-pager-on-performance-measurementfr1-pager-on-performance-measurement (registration required).]

End of term: Complete

All three milestones have been completed. However, as noted in the midterm assessment, all three were phrased in relatively open-ended ways. This phrasing made it difficult to conclusively assess the level of progress which was envisioned when the action plan was created.

The online progress tracker for OGP commitments remains online (Milestone 5.1). However, it was updated only once between January and August 2018. This update schedule fell behind the commitment's intentions to carry out quarterly updates. The end-of-term self-assessment reports that the Treasury Board of Canada Secretariat (TBS) also posted a draft logic model for open government on GCcollab, a cloud-based collaboration platform hosted by the government of Canada. In addition, TBS provided continued reporting through the Management Accountability Framework.[Note42: See https://gccollab.ca/file/view/171990/draft-logic-model-for-the-government-of-canadas-work-on-open-government (registration required); and “Management Accountability Framework,” Government of Canada, https://www.canada.ca/en/treasury-board-secretariat/services/management-accountability-framework.html.] Under Milestone 5.2, the TBS released its Progress Tracker for Open Data dataset on 14 June 2018. The tracker captures the number of datasets released by departments over the previous year and estimates the number that will be released by November 2018 and March 2019.[Note43: See “Progress Tracker for Open Data,” Government of Canada, https://open.canada.ca/data/en/dataset/b94ff2b5-9adc-4951-ab07-c7430dee01d2.]

TBS worked with the SecDev Foundation, a Canadian think tank, to develop a proposed performance management framework with related indicators (Milestone 5.3).[Note44: See “Open Government Performance: Measuring Impact,” Government of Canada, https://open.canada.ca/ckan/en/dataset/f637580f-e0f7-5939-bf3f-ded35ce72d2a.] The government self-assessment reports that this framework will be used to facilitate a conversation with stakeholders about how to structure a framework for impact.[Note45: The self-assessment is available at https://open.canada.ca/data/en/dataset/9da9faf5-deb1-48db-8f16-91055d942d65.]

Did It Open Government?

Access to Information: Marginal

This commitment aimed to improve the evaluation and assessment of open government progress. This area has considerable scope for improvement both in Canada and, indeed, around the world. However, evidence demonstrates only marginal progress directly relevant to OGP values. As written, the commitment is assessed as having no relevance to OGP values, due to the vagueness of its milestones. In order to have relevance to OGP values, and in particular the value of public accountability, the indicators and reporting contained in this commitment need to be both publicly facing and responsive to public inputs. However, its implementation resulted in marginal improvement of access to information. While Milestone 5.1 is mostly duplicative of existing OGP reporting requirements, Milestone 5.2 provides a single additional dataset to the public. Milestone 5.3 targets a more important action area: developing a meaningful framework for assessing the value of particular programming. However, Milestone 5.3 was phrased vaguely regarding its expected progress. Further, its implementation and application remain in their early stages. Nonetheless, the development of this framework and indicators could deliver benefits in the future, even if it is too early to assess its results as having substantially contributed to open government.

Carried Forward?

Canada's fourth action plan includes a commitment to publish data on service delivery performance at open.canada.ca (4.5). The plan also includes a commitment to use international events to drive peer learning in this area and to measure government and open data impact internationally (10.3).