Public Sector Performance Measurement and Reporting

Background

An effective system of performance measurement and reporting is critical if government is to achieve its policy goals in a way that is transparent and accountable.

In Victoria, the performance measurement and reporting system aims to:

• fully inform government's resource allocation to best achieve its policy goals
• help departments to understand how well they are meeting government's performance expectations, and provide the basis for them to continually improve their performance
• enable Parliament and the community to understand the challenges facing government, and its achievements in addressing these and areas for further focus.

The government expanded the framework to transform it from measuring only outputs to one that also measured the impact of those outputs on its intended outcomes for the community. It also introduced requirements for medium-term and long-term planning so it could anticipate and prepare for the fiscal and performance challenges likely to affect Victoria in the future.

Applying the system as intended would represent better practice performance measurement and reporting because government, Parliament and the community would understand whether:

• departments were effectively and efficiently delivering funded outputs
• these outputs resulted in outcomes government and the community wanted
• performance could be sustained within reasonable funding requirements and what reforms were necessary to achieve acceptable performance under expected resourcing and demand.

While we found elements of better practice in other jurisdictions, such as Western Australia's more focused approach on outcomes, we found no other jurisdiction in Australia or overseas that had achieved these performance measurement goals in their entirety.

Implementing this form of transformational change to effectively and fully communicate performance to government, Parliament and the wider community is very challenging. It requires:

• the commitment of government
• proactive central agency leadership, providing clear guidance, effective support, close monitoring of progress and the ability overcome implementation barriers
• committed and capable departments, with the conceptual and technical strength to redefine approaches to measuring and reporting performance
• the flexibility to consider how the existing reporting structure needs to be changed to achieve better practice measurement and reporting.

The objective of this audit was to assess the effectiveness of the Department of Premier and Cabinet's (DPC), Department of Health's (DH) and Department of Transport, Planning and Local Infrastructure's (DTPLI) performance measurement and reporting and the Department of Treasury and Finance's (DTF) oversight, by examining whether:

• sampled departments are applying the government's performance measurement and reporting system to measure and communicate how well they are achieving outcomes consistent with government's policy objectives
• DTF is effectively guiding, supporting and overseeing the development and application of an updated performance measurement and reporting system.

Conclusions

The departments we examined are not effectively applying the government's performance measurement and reporting system.

While Budget Papers and annual reports include many output performance measures, weaknesses in defining objectives and linking them to outputs, the absence of robust and logical outcome measures and meaningful commentary mean they are impenetrable to the reader.

The analysis provided directly to government is more extensive but again is solely focused on the output measures that over time have been included in Budget Papers.

The progress towards developing corporate and long-term plans that fully address the government's requirements has been slow. Draft plans for DH, DTPLI and DPC fall well short of the government's minimum requirements and the rate of progress does not suggest that agencies are close to addressing this.

DTF's oversight of the performance measurement and reporting system has been partly effective. DTF applied the required processes and provided guidance and support to departments and reviewed their progress.

However, DTF needs to improve the quality and depth of its guidance material, the rigour of its reviews and clarity of its reporting. But above all it needs to identify and address the persistent barriers that are hampering progress, and to work with departments to overcome these.

Findings

Improving performance measurement

Budget Papers and annual reports fall well short of providing the information needed to understand departments' effectiveness and efficiency in delivering outputs and intended outcomes for the community. Progress to address this has been slow and:

• output measures rarely provide sufficient information to understand how effectively and efficiently departments are delivering on their funded outputs
• published information does not adequately explain outcome effectiveness because objectives are inconsistent and objective indicators do not provide clear, quantified measures linking departments' outputs to objectives.

Of most concern is the lack of any meaningful departmental commentary explaining performance. The patchy coverage and inconsistent quality of performance measures and the absence of documented analysis of their meaning and significance mean these documents are of minimal value in explaining departments' performance.

Departments do analyse and comment on changes in output measures for government. DTF collates and comments on this material. However, these analyses do not address the structural weaknesses in departments' performance frameworks nor link output performance with the achievement of government objectives.

The departments in this audit identified barriers in the way output reporting is applied, that impede effective outcome measurement and reporting. These included:

• difficulties in linking specific outputs and objectives
• perceptions about the restrictive nature of Budget Papers and annual reports to house more extensive performance reporting
• difficulties in adequately defining objectives and objective indicators
• challenges in getting ministerial buy-in to significant changes in performance measurement.

DTF needs to work with departments to identify where and why they are falling short of government's requirements and recommend how best to overcome these barriers to change.

Improving planning

The latest draft corporate and long-term plans for DH, DTPLI and DPC do not meet the government's minimum requirements.

The corporate plans for these departments did not fully describe the service delivery risks departments face, nor provide the information needed to understand the performance and budgetary implications of the options for addressing these.

Across all departments, the depth and quality of corporate plans improved after their introduction in 2012 but remain variable and, for the three departments we examined, their latest plans do not fully address government's requirements.

The preparation of long-term plans was paused after departments submitted their first versions to government in March 2013 because of their inconsistent quality and the varying levels of commitment across portfolios to their timely submission.

Department of Treasury and Finance oversight

DTF's oversight of the performance measurement and reporting system has been partly effective. Its efforts to guide, support and check on departments' progress have been visible but not adequate and it needs to improve its performance.

DTF has:

• been proactive in communicating the changing measurement system requirements to departments and supporting them in applying these through guidance notes, support groups and providing day-to-day advice
• tested departments' compliance and fed the results back to departments and reported progress to government.

However, DTF needs to improve by better guiding departments with better-practice examples, more rigorously assessing and reporting on departments' performance, and better understanding and addressing the barriers to progress.

We found that weaknesses in departments' frameworks had not been consistently identified and reported back to departments or communicated to government.

The three agencies in this audit and the working group DTF set up to implement the framework identified barriers to implementation that were not effectively addressed.

The recommendations from this audit directed at DPC, DH and DTPLI are highly relevant to all departments that are subject to the performance measurement and reporting requirements set out in the budget and financial management guidance.

Recommendations

1. That the Department of Health, the Department of Transport, Planning and Local Infrastructure and the Department of Premier and Cabinet design—without being constrained by past performance measures—a performance measurement and reporting system that fully meets the government's requirements.
2. That the Department of Treasury and Finance and portfolio departments work together to:

• understand the barriers that hinder departments from fully applying the government's performance measurement and reporting system
• assess the options for overcoming these barriers including whether performance against outcomes should complement and be reported separately from Budget Papers, annual reports and corporate plans
• agree and apply a strategy to transition departments to a performance measurement and reporting system that meets government's requirements.

4. That the Department of Treasury and Finance and portfolio departments work together to determine how to deliver 2015–16 plans that are fit for purpose, by identifying deficiencies in current plans and agreeing how to overcome the barriers to addressing these.

That the Department of Treasury and Finance:

5. completes its work on the medium- to long-term challenges facing Victoria—by modelling the likely impacts of future fiscal, economic and demographic trends—so that it is ready to improve long-term planning once government decides how to proceed
6. improves the guidance material on performance measurement to include more practical examples to help departments measure efficiency and effectiveness and link outputs to departmental objectives
7. more rigorously and consistently assesses and communicates performance back to portfolio departments and government.

Submissions and comments received

Throughout the course of the audit we have professionally engaged with the:

• Department of Premier and Cabinet
• Department of Health
• Department of Transport, Planning and Local Infrastructure
• Department of Treasury and Finance.

In accordance with section 16(3) of the Audit Act 1994 we provided a copy of this report, or part of this report, to those agencies and requested their submissions or comments.

We have considered those views in reaching our audit conclusions and have represented them to the extent relevant and warranted. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

An effective system of performance measurement and reporting is critical if government is to achieve its policy goals in a way that is transparent and accountable.

In Victoria, the performance measurement and reporting system aims to:

• fully inform government's resource allocation to best achieve its policy goals
• help departments to understand how well they are meeting government's performance expectations and provide the basis for them to continually improve their performance
• enable Parliament and the community to understand the challenges facing government, and its achievements in addressing these.

The system requirements are communicated through a series of guides that are part of the government's budget and financial management guidance (BFMG) issued by the Department of Treasury and Finance (DTF).

These requirements apply to all government departments.

The effective implementation of the performance measurement and reporting system is critical if government is to be fully accountable for past performance and prepared to meet the significant performance challenges facing the state.

To date performance in terms of community outcomes has been unclear and departments have not fully informed government about the fiscal and performance impacts of emerging trends. VAGO's 2013 audit on the government's risk management framework identified that the scale of the emerging challenges is likely to be too great to maintain current living standards using a business-as-usual approach.

1.2 Past audits and reviews

VAGO's audits in 2001, 2003 and 2010, and the Public Accounts and Estimates Committee's (PAEC) reviews found significant shortcomings in DTF's oversight and implementation of the performance measurement and reporting system by departments.

1.2.1 VAGO's findings

The 2010 audit of departments' performance measurement and reporting, Performance Reporting by Departments, found that:

• Victoria's approach compared unfavourably to the acknowledged better-practice jurisdictions of New Zealand, Canada, United Kingdom and Western Australia
• departments did not consistently measure nor clearly report how well they were achieving outcomes consistent with government's policy objectives
• stronger central agency leadership was needed because there had been little progress in measuring and communicating outcomes over the previous decade.

The audit recommended that DTF:

• establishes, and departments apply, a framework measuring performance against government's strategic outcomes and embed this into the planning cycles
• reviews, consolidates, updates and communicates better-practice requirements to departments as a foundation for improving performance reporting.

VAGO's Reflections on audits 2006–12: Lessons from the past, challenges for the future summarised repeated and significant weaknesses including:

• not using appropriate measures of performance
• failing to measure outcomes
• insufficient guidance, advice and oversight by central agencies to support departments to implement the performance measurement system.

1.2.2 Public Accounts and Estimate Committee reviews

PAEC reports on the Budget Estimates covering the years 2011–12 to 2013–14 highlighted systemic weaknesses in Victoria's performance measurement system, its application by agencies and its oversight by DTF. PAEC noted the absence of a central, consolidated depository of the legislative and policy requirements and guidelines underpinning the system and made recommendations for change.

In response to VAGO's 2010 audit and PAEC's report on the 2011–12 Budget Estimates, the government implemented a number of reforms to enhance performance reporting. These changes have been reflected in DTF amending the BFMG.

PAEC's March 2014 Review of the Performance Measurement and Reporting System report acknowledged the progress made by departments in applying the upgraded requirements—described in Section 1.3 of this report. PAEC found examples of good practice, but identified that departments should strengthen their systems by ensuring that:

• departmental objectives meet government's criteria, focusing on the outcomes achieved for the community rather than goods and services delivered
• objective indicators cover all key aspects of objectives
• performance measures are clear, meaningful and robust
• performance measure targets are challenging but achievable
• explanations for changes to targets and variances between targets and results are sufficiently informative to provide a better understanding of performance.

1.3 Victoria's performance measurement system

The essential components of Victoria's current system date back to 1998–99. This involved transitioning from the government funding departmental inputs to it funding departments to deliver outputs to specific standards, as the means of achieving policy goals.

Departments had to assess their effectiveness and efficiency in delivering these outputs by measuring their quantity, quality, timeliness and cost.

The transition to output-based budgeting, while improving aspects of the system, did not fully address the system goals described above. Parliamentary inquiries and VAGO audits repeatedly identified the absence of comprehensive information on outcomes because of weaknesses in how agencies defined and linked objectives and output measures and targets.

In 2011, the government amended the framework by requiring departments to:

• from 2011–12, include objectives—intended outcomes—in Budget Papers to better show how departmental outputs will benefit the community
• from 2012–13, clearly show how outputs influence or link to objectives
• from 2013–14, include objective indicators as statistical and rigorous measures of the contribution of outputs to achieving specific objectives
• by 2013–14, develop four-year corporate and 10-year long-term plans to inform Budget decisions, by analysing challenges, and proposing mitigation strategies.

In Victoria the policy framework and requirements for performance measurement and reporting are set out in the BFMG.

Figure 1A shows Victoria's current system. We note that the process for developing long-term plans beyond initial versions was suspended in late 2013 pending a government review.

Figure 1A

Victoria's performance measurement and reporting system

Source: Victorian Auditor-General's Office based on data from the Public Accounts and Estimates Committee.

1.3.1 Expectations of a mature measurement system

Figure 1B describes the design of Victoria's current performance measurement and reporting system and how the components are meant to combine to measure effectiveness and efficiency.

Figure 1B

Components of a mature performance measurement system

Source: Public Accounts and Estimates Committee, based on the Victorian Auditor-General's Office's data.

According to government-endorsed guidelines, State Budget Papers and departments' annual reports, corporate and long-term plans need to include performance information and commentary that clearly explains to government, Parliament and the wider community how departments' performance is:

• effective—by:
  • delivering budgeted outputs according to prescribed quantity, quality, timing and cost targets
  • demonstrating how well departments' activities and outputs are contributing to government's intended outcomes for the community through quantitative, objective indicators—for example measuring improved safety, health, etc.
• efficient—by explaining how and why quality-adjusted outputs per dollar are changing over time, and how these compare when benchmarked against other departments and jurisdictions
• sustainable—based on a longer-term appreciation of future challenges, their performance implications and the type of reforms necessary to sustain and improve performance.

The BFMG requires departments to measure and benchmark efficiency by:

• setting output performance measures that demonstrate the efficiency and effectiveness of output delivery and value for money—BFMG guides 2 and 9
• enabling 'meaningful comparison and benchmarking over time, across departments and against other jurisdictions'—BFMG guide 9.

VAGO's expectation is that departments would have made significant progress since 2011–12 in transforming performance measurement and reporting:

• from an output-focused system without connection to government's policy objectives nor meaningful commentary on performance
• to an integrated system of outputs and outcomes with sufficient commentary to explain performance and flag emerging risks and the strategies that are likely to drive improved outcomes in the medium to long term.

Implementing this form of transformational change is very challenging. It requires:

• the commitment of government
• proactive central agency leadership, providing clear guidance, effective support and the capability to monitor progress and overcome implementation barriers
• committed and capable departments, with the conceptual and technical strength to redefine approaches to measuring and reporting performance.

1.3.2 Illustrating what better practice means

Victoria's updated system now aims to comprehensively and reliably communicate how the delivery of outputs has changed the outcomes for the community by:

• comprehensively measuring the efficiency and effectiveness of output delivery
• directly and specifically linking departments' activities and output performance to government's intended outcomes through quantitative, objective indicators
• adequately explaining output and outcome performance, the relationship between these, the reasons for significant change and departments' responses to these
• providing a longer-term context for current funding decisions through corporate and long-term plans that describe the performance challenges and longer-term strategies for addressing these.

Delivering on these aims represents a significant challenge that requires transformational change. It will require the capability to define meaningful objectives, objective indicators that measure the achievement of intended outcomes and the flexibility to adapt output measures to link to these indicators and objectives.

Figure 1C is a VAGO illustration of what better practice performance measurement might look like for one output designed to reduce preventable disease.

While the expenditure of all public funds should be subject to evaluation, departments have to determine the depth and scope of performance measurement based on the amount of money spent and the importance of the output to the achievement of its outcomes.

Not all outputs might command this depth of analysis. However, in reading Budget Papers and annual reports for the three departments we examined, we did not find any examples of this type of integrated and connected performance measurement.

Figure 1C

Hypothetical example of better practice performance measurement

Source: Victorian Auditor-General's Office.

1.4 Comparing Victoria with other jurisdictions

In the past Victoria's performance management system focused on measuring agency performance in delivering funded outputs without a clear connection to intended outcomes or a longer-term context within which to frame current funding decisions.

Current Victorian requirements, if effectively applied, would address these weaknesses and connect outputs to intended outcomes while also providing a longer-term context for decision-making. This represents better practice.

Our review of performance measurement and reporting in other Australian states and territories, and overseas, did not find any jurisdiction that had applied the better practices that Victoria wants to achieve.

Instead we found examples of jurisdictions that had applied aspects of better practice or were developing systems that had potential to effectively measure and report on performance. For example:

• Western Australia's approach is more focused on outcomes and this together with a greater commentary on the rationale and trends in measures are relative strengths. But it does not have the same depth of output performance information as Victoria, nor provide sufficient information to understand efficiency trends and explain how performance indicators adequately measure outcomes.
• The New Zealand Ministry of Health's 2013 annual report uses a framework where high-level outcomes are assessed using a range of measures that are reported with detailed commentary. While not fully linking outputs to clearly articulated outcomes, the framework has more information on outcomes and describes and comments on health spending and productivity trends. It also provides useful commentary on past trends for a range of relevant risk factors.
• The Commonwealth Department of Health and Ageing provides numerous key performance indicators and deliverables under 14 outcomes, together with extensive commentary. However, most of what it reports relates to outputs without adequately linking these to outcomes.
• South Australia is developing more comprehensive health performance reporting and this is still a work in progress.
• The UK is piloting the use of Patient Reported Outcomes Measures measuring patient health gains covering four common elective procedures and comparing the health gains with the cost of treatment.

These jurisdictions, while showing aspects of better practice, did not fully embody the practices that Victoria is aiming to apply as better practices. We also found no evidence that these jurisdictions had articulated better practice goals that replicated what Victoria is trying to achieve.

We found that useful performance information is collected and published by Commonwealth agencies but not used as part of Victoria's performance measurement and reporting framework, including:

• the Productivity Commission's annual report on government services measures the equity, productive efficiency and cost effectiveness of services
• the Australian Institute of Health and Welfare's National Health Performance Framework reports on health status, its determinants and system performance.

Victoria does not make full use of this information in its performance reporting.

1.5 Audit objective and scope

The audit objective was to assess the effectiveness of the Department of Premier and Cabinet's (DPC), Department of Health's (DH) and Department of Transport, Planning and Local Infrastructure's (DTPLI) performance measurement and reporting and DTF's oversight, by examining whether:

• the government's performance measurement and reporting system reflects better practice
• the sampled departments are applying the government's performance measurement and reporting system to adequately measure and communicate how well they are achieving outcomes consistent with government's policy objectives
• DTF is effectively guiding, supporting and overseeing the development and application of an updated performance measurement and reporting system.

We selected these departments because:

• DH and DTPLI which are together responsible for $20 billion of output expenditure and for outcomes that are of significant interest to the community
• DPC, in contrast, has lower expenditure but sees its mission as 'displaying exemplary leadership and innovation to support the Victorian Government in achieving strong public policy and service delivery outcomes for all Victorians'—this means it is involved in programs that influence the effectiveness and efficiency of public sector's systems and workforce and for whole-of-government programs.

1.6 Audit method and cost

We reviewed publicly available documents on performance measurement and requested access to agencies' records to understand how they had measured and reported on performance. We also accessed DTF's records of the guidance and support it provided to agencies together with evidence of its review and government reporting activities.

We used this information to address our audit criteria and followed up with departments where the information was insufficient or unclear.

The audit was conducted in accordance with section 15 of the Audit Act 1994 and Australian Auditing and Assurance Standards.

Pursuant to section 20(3) of the Audit Act 1994, any persons named in this report are not the subject of adverse comment or opinion.

1.7 Structure of the report

The report has three further parts:

• Part 2 examines the progress of three selected agencies in updating their performance frameworks to better measure and report on performance
• Part 3 assesses these agencies' progress in applying the government's enhanced planning requirements
• Part 4 assesses DTF's performance in guiding, supporting and overseeing agencies in applying these expanded requirements and overcoming the barriers to change.

2.1 Introduction

If properly applied, Victoria's updated performance measurement and reporting requirements will represent better practice.

Our review of performance measurement and reporting in other Australian states and territories and overseas did not find any jurisdictions that had applied the better practices that Victoria wants to achieve. Instead we found examples of jurisdictions that had applied aspects of better practice or were developing systems that had the potential to effectively measure and report on performance.

In this Part we assess the Department of Health's (DH), the Department of Transport, Planning and Local Infrastructure's (DTPLI) and the Department of Premier and Cabinet's (DPC) progress in measuring and communicating:

• information on performance by defining and reliably measuring appropriate indicators to understand:
• output efficiency and effectiveness in terms of quantity, quality, timeliness and cost
• outcome effectiveness by defining objective indicators that quantify the impacts of departmental outputs on government's intended outcomes
• longer-term threats to performance and strategies for addressing these through four-year corporate plans and 10-year long-term plans.

2.2 Conclusion

Published papers, and annual reports and progress reports to government show audited departments attempted to, but fell well short of providing the information needed to adequately understand their performance.

Progress to address this has been slow and:

• output measures rarely provide sufficient information to understand how effectively and efficiently departments are delivering on their funded outputs
• published information does not adequately explain outcome effectiveness because objectives are inconsistent and objective indicators do not provide clear, quantified measures linking departments' outputs to objectives.

Of most concern is the absence of any meaningful departmental commentary explaining performance. The patchy coverage and inconsistent quality of performance measures and the absence of documented analysis of their meaning and significance mean these documents are of minimal value in explaining departments' performance.

Departments analyse and comment on changes in output measures for government and the Department of Treasury and Finance (DTF) collates and comments on this material. However, these analyses do not address the weaknesses in departments' performance frameworks or link output performance with the achievement of government objectives.

Part 4 of this report examines the barriers to change and how well DTF and portfolio departments are addressing these.

2.3 Common findings across three departments

The information published by the three departments we examined clearly falls short of the government's performance measurement requirements. While the measures reported provide some valuable information about output quantities, quality, timeliness and costs, the coverage is patchy and does not adequately cover performance.

The coverage of output effectiveness is partial and no departments analyse output efficiency. Departments recently added objectives and objective indicators to their measurement frameworks but these do not consistently meet government's requirements and fail to effectively link outputs to outcomes.

The absence of meaningful commentary around the measures which departments do include make it difficult for a reader to understand what these say about performance.

The three departments we examined all identified the existing output-based framework and the reluctance to radically change this and the scope and content of the current reporting architecture, of Budget Papers and annual reports, as constraints on them achieving the requirements of an outcome-focused system. We examine these barriers and how well they have been addressed in more detail in Part 4 of the report.

DTF needs to work with departments to understand these constraints and work out how to deliver an annual departmental report on performance by:

• setting fit-for-purpose objectives and objective indicators based on reliable data
• reviewing and, if necessary, modifying:
• output measures and objective indicators to adequately measure output effectiveness and efficiency and to clearly link to departments' objectives
• the current reporting architecture—for example, by considering whether a separate report is warranted that describes and explains performance
• providing the analysis, benchmarking and commentary needed for government, Parliament and the community to clearly understand performance.

The remainder of this Part describes how departments' current performance measurement frameworks do not fully meet government's requirements.

2.4 Department of Health

DH is the lead agency overseeing all health services, mental health, ageing and aged care, and preventive health. It is responsible for planning, policy development, funding and regulation of health service providers and activities that promote and protect Victorians' health.

To carry out these responsibilities DH employs approximately 1 300 full-time equivalent staff and in 2014–15 is funded to oversee the production of outputs costing approximately $10 billion in appropriations funding.

Figure 2A shows the performance framework used in DH's 2012–13 annual report, and 2013–14 and 2014–15 Budget Papers including its objectives, objective indicators and groups of outputs.

2.4.1 Output measures

DH does not adequately link specific output measures to specific departmental objectives that reflect government policy. Figure 2A shows it has linked all output groups and outputs with all objectives and objective indicators. In addition it is not clear that the scope and coverage of output performance measures are sufficient to explain whether DH has efficiently and effectively delivered these outputs.

Figure 2A

Department of Health measurement framework, 2014–15

Source: Victorian Auditor-General's Office based on 2014–15 Budget Papers.

While the measures include costs and quantities, there is no analysis of changes in efficiency or productivity which is potentially misleading. For example, according to the Budget Papers time series data, and based on our own calculations, between 2007–08 (actual) and 2013–14 (expected), emergency presentations increased by 15 per cent while the costs more than doubled—a 106 per cent increase including inflation. The purpose of this example is not to show that efficiency is diminished—this would require further analysis on changes in quality and the other outputs in this group—rather it illustrates the complete absence of this type of analysis for all three departments examined in this audit.

Understanding these trends and having DH's insight into what they say about efficiency and productivity are critical if government and the community are to understand progress against the third departmental objective around financial sustainability—this gap needs to be addressed.

In addition, we are not assured that the output measures are adequate to understand effectiveness. For example, the Life! Diabetes and Cardiovascular Disease Prevention Program is critical for addressing risk factors that make people vulnerable to these conditions. Currently, the only output measures are the actual and target numbers of people taking the program.

This is insufficient to understand the program's effectiveness. Figure 1C in Part 1 of this report showed that a better practice approach would also measure participants' satisfaction and intentions at the end of the course and involve following up a sample of participants to measure sustained and significant changes in behaviour and targeted risk factors.

2.4.2 Outcome measures

The objectives only partly meet DTF's requirements:

• The intended high-level outcomes are clear―to reduce preventable disease, improve the effectiveness/efficiency of health care and improve system productivity and sustainability—these cover, at a high level, government's relevant policy goals, but do not exactly align with DH's 2012–22 long-term plan, the Victorian Health Priorities Framework—as shown in Figure 2B. This is designed so that the DH's objectives reflect but are not precisely the same as government's long-term goals so they are more likely to endure if there is a change in government.
• The target beneficiaries are all Victorians and DH accepts that within this objective there is likely to be a need to focus on specific groups in the community at a program level.
• The standard of service, quantifiable goals and clear medium-term time frames for achieving these objectives are not—as recommended by DTF—clearly defined.

Figure 2B

Comparing Budget Paper objectives and framework outcomes

Source: Victorian Auditor-General's Office based on Victorian Health Priorities Framework 2012–2022.

Objective indicators are meant to clearly link specific outputs and output groups with specific objectives. The framework fails this requirement because all outputs and approximately 195 output measures have been linked through all the objective indicators to all objectives. This goes against DTF's requirement for specific linkages so readers can understand.

DH has defined but has not yet quantified its objective indicators.

There are, in addition, several design weaknesses with the objective indicators shown in Figure 2A:

• The first two indicators under Objective 1 lack specificity―which chronic disease factors should be measured and which health and social outcomes should be tracked and for which disadvantaged groups?
• Several indicators measure outputs rather than outcomes―for example, the third indicator under Objective 1 measures immunisation rates not disease incidence and the second indicator under Objective 2 measures the number of people treated
• It is unclear what is being measured under the final two indicators for Objective 2 and the single indicator under Objective 3. Measures of improved efficiency and productivity are more appropriate for the third objective.

DH accepted that these findings represent a reasonable commentary on the way it has implemented the performance measurement framework.

DH produces publicly available reports and web-based data, internal management reports and information used in Commonwealth performance reporting that include valuable performance information. However, this information is not integrated into its annual performance reporting to form a clear and comprehensive picture of performance outcomes that is accessible to Victorians.

This information includes hospital performance data on the Commonwealth's MyHospitals website, periodic state reports on the status of Victorian's health and the Productivity Commission's annual report on costs and efficiency. DH also raised a number of practical barriers to change that are discussed in Part 4.

2.5 Department of Transport, Planning and Local Infrastructure

DTPLI's purpose is to create a liveable and prosperous Victoria. It will do this by providing connected transport, land-use planning and infrastructure services.

To carry out these responsibilities DTPLI employs approximately 1 130 full-time equivalent staff and in 2014–15 is funded to produce outputs costing $6.3 billion.

Figure 2C shows the performance framework used in DTPLI's 2012–13 annual report and the 2014–15 Budget Papers. DTPLI has updated the objectives and objective indicators used in its 2014–18 draft corporate plan. However, its output measures in the 2014–18 draft corporate plan remain unchanged.

Figure 2C

Department of Transport, Planning and Local Infrastructure measurement framework, 2014–15

Source: 2014–15 Budget Papers.

2.5.1 2014–15 performance measurement framework

Output measures

As for DH, DTPLI's performance management framework does not link outputs consistently with objective indicators and objectives.

For example, for the second last objective in Figure 2C—safer transport services and infrastructure—there are 53 output measures within the two output groups, transport safety regulations and investigations, and transport safety and security management. These output groups are meant to link to the objective through a single objective indicator―'Fatalities and serious injuries on the road network reduced through a strategic approach aimed at road user and vehicle regulation, road user education, safer road network operation and improving road infrastructure'.

We could not see how two-thirds of the output measures were relevant to the single objective indicator and therefore could not understand how they were linked to the objective. For example, the measure 'the proportion of reported marine pollution incidents…' is not relevant to this objective indicator and this is typical of the irrelevant measures we identified. However, we note that the draft DTPLI 2014–18 corporate plan has expanded the road fatalities objective indicator to also include public transport fatalities.

Outcome measures

The objectives only partly meet DTF's requirements:

• The intended outcomes are clear but high level for three of the eight objectives—the ones covering transport, the first and two last objectives—the remaining objectives do not describe clear outcomes, for example the second objective, 'deliver effective reform and governance of local government'.
• The target beneficiaries are by default all Victorians and again this is too coarse, given that some areas of action are likely to target specific groups.
• The standard of service, quantifiable goals and clear medium-term time frames for achieving these objectives are not defined.

Our initial assessment of the objective indicators is mixed:

• The stronger indicators are under the three transport objectives where on-time running, public transport passenger satisfaction and road fatalities and injuries provide reasonable links between DTPLI's outputs and objectives. There are, however, areas of weakness—for example, there is insufficient measurement around traffic congestion, journey reliability and road user satisfaction.
• The remaining non-transport indicators are much weaker. It is unclear from the papers what these mean—for example, the indicators for the objective 'plan for the future growth and transformation of cities and regions' are:
• 'the vision for Victoria is reflected in the State Planning System
• satisfaction of key stakeholders with State Planning Strategies
• reforms…to increase the efficiency of Victoria's planning, building…system'.
• Indicators are too vague to measure progress towards this objective. DTPLI needs to define specific, measurable indicators of liveability and satisfaction that can be forecast, tracked and managed, including journey time reliability for common journeys and liveability measures.

Overall DTPLI agreed with these findings. It advised that the pace of effective change within the department had been slow due to the timing and impact of the Machinery of Government (MoG) changes in April 2013, which meant the new department could do little else than combine the components of the two merged departments for the 2013–14 Budget.

DTPLI also provided useful commentary on the practical barriers that it would need to overcome in achieving the government's performance measurement and reporting goals and these are examined in Part 4 of the report.

2.5.2 Department of Transport, Planning and Local Infrastructure updated objectives and indicators

DTPLI's 2014–18 draft corporate plan includes revised objectives and objective indicators tailored for the combined planning, transport and local infrastructure department. These will be applied in the 2015–16 Budget Papers. DTPLI thinks these address some of the identified weaknesses.

While the revised framework better integrates DTPLI's functions, it does not fully address our criticisms. The objectives are not clear and specific in terms of the intended outcomes for the community.

In addition, the objective indicators do not clearly connect unchanged outputs to objectives to provide a comprehensive and understandable picture of performance.

2.6 Department of Premier and Cabinet

DPC's mission is to display exemplary leadership and innovation to support the Victorian Government in achieving strong public policy and service delivery outcomes for all Victorians.

DPC is involved in programs that influence the effectiveness and efficiency of the public sector's systems and workforce and also in whole-of-government programs.

It delivers on this agenda with just over 300 full-time equivalent staff and with an output budget of $676 million for 2014–15.

Figure 2D shows the performance framework used in DPC's 2012–13 annual report and the 2014–15 Budget Papers.

Figure 2D

Department of Premier and Cabinet measurement framework, 2014–15

Source: 2014–15 Budget Papers.

2.6.1 Output measures

In terms of the output measures we again could not, as DTF requires, see clear and specific linkages between output measures and the objectives the quality and timely delivery of these outputs is meant to drive.

For example, one output group under the objective 'supporting high-quality government decision-making and implementation' is 'government-wide leadership and implementation'. In the Budget Papers the description of this output group is to 'Monitor the implementation and delivery of the government's decisions and projects and lead effective whole-of-government responses to significant identified issues'.

We are interested in this area because of repeated findings about the problems with whole-of-government initiatives. Our review of some 200 audits—Reflections on audits 2006–2012: Lessons from the past, challenges for the future, found 'joined-up arrangements between agencies were often not coordinated sufficiently…meaning that agencies with aligned priorities and goals worked in isolation…'

In our view the outputs and measures under this group do not provide information to fully understand performance in relation to this objective. They focus on the number of briefings, forums facilitated and the timely delivery of events. The outputs provide no useful information on the successful delivery or otherwise of government's decisions, projects and whole-of-government responses.

2.6.2 Outcome measures

The objectives only partly meet DTF's requirements:

• They state high-level intentions―to support high‑quality government decision‑making and implementation, develop and promote a thriving Victorian arts and cultural sector, support and promote full participation in strong and vibrant communities, and promote an effective, accountable and professional public administration―but it's difficult to clearly see what achieving these objectives would mean.
• The target beneficiaries are not specified except for the third objective which identifies veterans, aboriginal and culturally, linguistically and religiously diverse communities—the other objectives by default apply to all Victorians.
• As was the case for DH and DTPLI, the standard of service, quantifiable goals and clear medium-term time frames for achieving these objectives are not defined.

DPC has objectives that are particularly challenging to specify in measurable terms but we do not think it has made sufficient progress in applying DTF's guidelines for defining objectives.

Our assessment is that DPC's objective indicators have not been developed to the point where it is clear how DPC will collect and use data to link specific outputs to outcomes and measure progress towards meeting objectives. They have not been applied to date and their state of development means it is unclear how they could be quantified to measure progress towards staged objectives.

For example we would expect success in Objective 4 'promoting an effective, accountable and professional public administration' would be seen in:

• the perceptions of public servants about their work and workplaces
• community members' views as customers of the public sector and taxpayers
• changes in the efficiency of the public sector showing the impacts of improved approaches to recruitment, retention and training.

DPC's current objective indicators don't provide a clear insight into how progress towards this objective will be measured and include:

• 'The Governor is supported effectively in the exercising of his functions and powers'—how will the effectiveness of this support be measured?
• 'A centre for excellence that fosters an efficient, ethical and responsible public sector'—where are the measures of efficiency and impact?
• 'Fairness, integrity and respect for human rights and administrative excellence in the Victorian public sector are effectively promoted'—how will effective promotion be measured?

DPC needs to rethink its measurement framework, starting with the intended outcome and then defining what it should measure to understand changes in these outcomes and how this is driven by the outputs it is responsible for.

DPC accepts that there is room for improvement and, like DH and DTPLI, identified issues that present significant challenges to departments successfully transitioning to an outcome-focused performance framework. We summarise these barriers and consider ways of overcoming them in Part 4 of the report.

Recommendations

1. That the Department of Health, the Department of Transport, Planning and Local Infrastructure and the Department of Premier and Cabinet design—without being constrained by past performance measures—a performance measurement and reporting system that fully meets the government's requirements.
2. That the Department of Treasury and Finance and portfolio departments work together to:

• understand the barriers that hinder departments from fully applying the government's performance measurement and reporting system
• assess the options for overcoming these barriers including whether performance against outcomes should complement and be reported separately from Budget Papers, annual reports and corporate plans
• agree and apply a strategy to transition departments to a performance measurement and reporting system that meets government's requirements.

3.1 Introduction

This Part of the report examines departments' progress in upgrading their medium- to long-term plans. Comprehensive and rigorous plans are critical if government is to be adequately prepared to meet the medium- to long-term challenges facing the state.

The government upgraded its planning framework in 2011 to support its strategic decision-making by providing the information it needs to better understand:

• performance and the likely impact of future challenges and risks on the outcomes it wants to achieve for Victorians
• the options for reform to meet these performance challenges.

Four-year corporate plans and 10-year long-term plans:

• have to be endorsed by portfolio ministers
• explain how resources will be used to deliver agreed outputs and infrastructure
• provide explicit links between departmental short-, medium- and long-term priorities and clearly articulated and measurable levels of performance.

Figure 3A shows the attributes VAGO sees as critical if departments' long-term plans are to identify significant internal and external factors which may affect performance and determine how their impact will be minimised. These attributes are consistent with the government's requirements and guidelines.

Figure 3A

Longer-term plans—required attributes

Source: Victorian Auditor-General's Office.

Departments were required to:

• submit corporate plans by 31 August 2012, provide a 2013–14 update by July2013 and provide a 2014–15 version in July 2014, made up of a main body that would be published and a section confidential to government—none of the audited departments has published these updated plans
• finalise a first version of their long-term plans by 31 October 2012 and then update this by the 31 August in subsequent years—long-term plans were not intended for dissemination beyond government.

3.2 Conclusion

The progress towards developing corporate and long-term plans that fully address the government's requirements has been slow. The latest corporate and long-term plans for Department of Health (DH), Department of Transport, Planning and Local Infrastructure (DTPLI) and Department of Premier and Cabinet (DPC) fall well short of the government's minimum requirements, and the rate of progress does not suggest that agencies are close to addressing this shortfall.

The corporate plans for these departments did not fully describe the service delivery risks departments' face, or provide the information needed to understand the performance and budgetary implications of the options for addressing these.

Across all departments, the depth and quality of corporate plans improved after their introduction in 2012. However, the corporate plans of the audited departments do not adequately address government's requirements.

The preparation of long-term plans was paused after departments submitted their first versions to government in March 2013 because of their inconsistent quality and poor timeliness in submitting plans to the Department of Treasury and Finance (DTF).

3.3 Corporate plans

The corporate plans reviewed for the three audited departments provided useful information on the challenges and pressures departments face over a four-year period. However, critical gaps in coverage and a lack of depth and detail compromised the usefulness of these plans. None have been published.

The corporate plans for DH, DTPLI and DPC described:

• their mission, values and objectives
• at a high level, the challenges they faced in the short to medium term
• their short-term priorities
• what they were doing in terms of current and planned actions.

These plans described, at a high level, their progress in applying cost savings, the budget impact if lapsing programs were not renewed, and current and proposed reforms to improve efficiency and effectiveness.

In terms of the critical gaps and weaknesses, plans did not:

• adequately describe current performance in achieving departmental objectives
• explain how emerging risks affect performance under a business as usual scenario
• provide sufficient detail about current and planned actions and reforms, or explain how these are likely to work together to change performance
• adequately test and convey the key budgetary and performance impacts of a reasonable range of options for addressing emerging risks.

The lack of connection to performance is a concern because it does not adequately explain how proposed measures will affect performance. The plans do not give government the information it needs to make decisions about the real and significant challenges the state faces.

The parts of the plans intended for publication will provide even less information to the community, which means that readers must trust that departments will effectively address performance issues without giving them the information required to understand the issues.

The detail within these plans for both DH and DTPLI depict challenges similar to those described in a recent VAGO audit and in other government publications.

The Commonwealth's 2010 report Australia to 2050: future challenges, highlighted similar issues to those presented by the departments we examined in their draft corporate plans.

The key conclusion of this report for Australia as a whole also applies to Victoria—with the state facing a complex mix of challenges including an ageing and growing population and escalating pressure on the health system and the capacity of the transport system to cope with growth.

VAGO's 2013 audit on the Implementation of the Government Risk Management Framework concluded that Victoria is vulnerable to a range of statewide risks where significant consequences are expected to materialise in the medium to long term. These risks include the growth in lifestyle-related chronic disease and the impacts of growth on transport and congestion.

The report concluded that the state is not well prepared to effectively manage these risks and the consequences of not addressing this shortfall are likely to be significant.

The nature and likely significance of these future challenges make it imperative that departments address the weaknesses identified in their corporate plans.

3.4 Long-term plans

Long-term plans are critical if government is to make fully informed decisions about how to address emerging risks. The preparation of long-term plans was paused after departments submitted their first versions to government because of their inconsistent quality and examples of poor timeliness in submitting plans to DTF.

DTF encountered the following significant issues when implementing this:

• the absence of an overarching and agreed fiscal outlook covering the 10‑year period meant departments were not clear about the intended scope and extent of reform options
• some departments did not lodge a final long-term plan because the nature of the analysis presented policy challenges they could not resolve
• some departments viewed long-term planning as a compliance exercise with no clear objective or incentive to develop reform options
• plans varied in terms of their quality because the capability to model demand and test long-term alternative scenarios varied across departments.

DTF views long-term planning as critical if government is to effectively manage the significant economic and fiscal threats to service delivery and performance. The government has flagged that these plans will not be published.

DTF therefore proposed an alternative approach to achieving the goals of long-term planning. DTF is preparing forecasts of the likely impacts of fiscal, economic and demographic trends for Victoria. The emerging findings are that:

• population growth and ageing and cost pressures mean the gap between the demand for services and the resources available to meet these will grow
• planning beyond the four-year forward estimates period is essential to ensure the long-term financial sustainability of service delivery
• meeting these challenges will require substantial reform solutions and a whole‑of‑government approach.

DTF has proposed an approach which means it will drive long-term planning by:

• using the work on forecasts of the likely impacts of fiscal, economic and demographic trends to describe the funding and resourcing outlook, key challenges, opportunities and service delivery pressure point, and reform options and their impacts
• disseminating the results to portfolio departments for their review and response to refine the analysis and the reform options
• presenting the reform options for government's review and decisions on which ones should proceed.

This approach is being considered by government.

In terms of the three departments we examined, all three had similar weaknesses in their corporate plans with insufficient information on the performance implications of projected trends. DH's plan was clearly at a less advanced state of preparation than the other departments and was in the form of three A3 diagrams of its intentions rather than a fully documented plan.

Recommendations

1. That the Department of Treasury and Finance and portfolio departments work together to determine how to deliver 2015–16 plans that are fit for purpose, by identifying deficiencies in current plans and agreeing how to overcome the barriers to addressing these.
2. That the Department of Treasury and Finance completes its work on the medium‑ to long-term challenges facing Victoria—by modelling the likely impacts of future fiscal, economic and demographic trends—so that it is ready to improve long‑term planning once government decides how to proceed.

4.1 Introduction

This part assesses the Department of Treasury and Finance's (DTF) oversight of the performance measurement and reporting system on behalf of the Minister for Finance by examining whether DTF has:

• clearly and comprehensively communicated the government's performance system requirements to departments
• provided departments with the support they need to fully understand and apply government's requirements
• adequately tested departments' compliance with government's requirements and acted to address noncompliance and inferior practice
• fully informed government about departments' progress and performance in applying the performance measurement and reporting system.

4.2 Conclusion

DTF's oversight of the performance measurement and reporting system has been partly effective. Its efforts to guide, support and check on departments' progress have been visible but not adequate, and it needs to improve its performance.

DTF has:

• been proactive in communicating the changing measurement system requirements to departments and supporting them in applying these through guidance notes, support groups and providing day-to-day advice
• tested departments' compliance and fed the results back to departments and reported progress to government.

However, DTF needs to improve by better guiding departments with better practice examples, more rigorously assessing and reporting on departments' performance, and better understanding and addressing the barriers to progress.

We found that weaknesses in departments' frameworks had not been consistently identified and reported back to departments and communicated to government.

The three agencies in this audit and the working group DTF set up to implement the framework identified barriers to implementation that were not effectively addressed.

4.3 Communicating government's requirements and supporting departments

DTF has clearly communicated government's requirements for performance measurement and reporting through guides written under the budget and financial management guidance (BFMG) series.

These guides clearly describe the requirements for output and outcome performance measurement, and for the content of corporate and long-term plans.

While there is clarity around the requirements—for example, that departments need to measure the effectiveness and efficiency of output delivery—we found a lack of examples to help departments understand how to successfully apply these requirements. This type of transformational change requires extensive guidance about what success looks like.

DTF should use examples more extensively to illustrate what better practice application means.

DTF provided evidence that it had supported departments by:

• setting up a performance management working group with representatives from all departments that met 10 times between August 2011 and February 2013—the aim of this group was to support the key contacts who were responsible for upgrading each department's performance measurement system
• assigning a DTF contact to each department to liaise with and support key contacts in modifying performance systems to meet DTF's requirements.

The working group raised barriers and difficulties in terms of fully applying the government's performance framework, and the outcomes we observed lead us to the finding that DTF and departments were not able to overcome these. The following section describes these barriers based on information from the working group and the responses to our findings from audited agencies.

4.3.1 Effectiveness in addressing the barriers to progress

Taken together, departments identified the following barriers to change:

• the feasibility of effectively linking specific outputs and objectives when departments perceive that substantial output restructuring would be subject to review and probable rejection—the requirement to aggregate outputs added to the challenge of effectively linking and overlaying an outcome measurement framework
• the restrictive nature of Budget Papers and annual reports makes it difficult to use these as vehicles for reporting fully on outcomes
• difficulties in defining objectives and objective indicators, and the absence of detailed examples to guide departments—for example, in how to measure and report on efficiency
• challenges in getting ministerial buy-in to performance framework changes.

VAGO acknowledges these constraints and notes that they are a common theme across the departments examined and that the type of transformation requires clear direction and time.

Our concern is that we have not seen evidence that the barriers to progress identified by departments are being addressed. Instead the departments we examined are technically complying with government's requirements without achieving their intent and this does not assure us that significant, further progress will occur under this approach.

For example, the current public reporting architecture comprising corporate plans, Budget Papers and annual reports is clearly not achieving its intended performance reporting purpose because:

• Budget Papers remain focused on reporting output measures and the description of objectives and objective indicators adds little to a reader's understanding of outcomes and their connection to outputs. Even if objective indicators were improved and reported, the absence of insightful commentary on performance does not assure us that the Budget Papers would achieve the intended goal of fully communicating performance.
• Annual reports replicate the performance measures reported in the Budget Papers and add little to a readers understanding of what they mean. Commentary focuses on objectives, priorities and programs without effectively linking these to performance.

In responding to these barriers, DTF:

• has identified the trade-off between making improvements and government's requirement that performance measures should remain consistent over time wherever possible to ensure that useful information can be tracked
• considers that the enhanced performance reporting recommended in this report should not be included in Budget Papers because their primary purpose is to support the passage of the Appropriation Bill through Parliament. The Budget Papers are forward looking and enhanced performance information is more appropriately included in annual reports.

DTF needs to work with departments to determine how best to achieve government's goals—taking account of the need for continuity—and whether these forms of reporting can be transformed to achieve government's objectives for improved reporting or, whether a new type of report is required.

DTF performance measurement working group

DTF set up a performance measurement working group with representatives from all departments to help guide and support the implementation of governments updated performance measurement and reporting requirements. It met 10 times between August 2011 and February 2013.

In reviewing the minutes agencies raised concerns about:

• the feasibility of establishing one-to-one linkages between output measures and objectives without substantial restructuring of output measures
• proposed changes to output measures being rejected through the review process
• difficulties in defining objectives, and objective indicators meeting the framework's requirements.

Agencies responses to VAGO's early findings

VAGO provided agencies with an early briefing on the emerging findings and has found no further evidence to significantly alter these. Agencies provided comments and also identified the barriers they perceive as constraints on them fully achieving the government's requirements.

Department of Health response

The Department of Health (DH) accepted that VAGO's findings represented a reasonable commentary on the way it has implemented the performance measurement framework.

DH asked that the report recognise:

• the wide range of transparent reporting that it contributes in the public domain outside of the Budget Papers and annual report
• the restrictive format of the Budget Papers and annual report to achieve the reporting on outcomes of the health portfolio—DH contended that health outcomes depended on activities that went beyond the boundaries of a single portfolio and reporting on outcomes within the confines of the Budget Papers is difficult
• furthermore DH and the Department of Transport, Planning and Local Infrastructure (DTPLI) noted the barriers to significantly changing the output performance structure because of the benefit of longitudinal reporting.

DH suggested that achieving the government's intended performance measurement and reporting outcomes requires a different vehicle to the Budget Papers—such as a whole-of-government reporting framework.

The department agrees with our view that the requirements for objectives and objective indicators have been overlaid on a structure that was built for a different purpose—to acquit the delivery of funded outputs. Changes to the existing output group structure were limited by:

• retaining the ability to compare performance over time
• inconsistent advice from DTF about the acceptable amount of change
• rejection of what DH considered appropriate changes.

These responses do not alter our core findings but raise material issues about the barriers to departments applying the government's performance measurement and reporting requirements. These issues need to be raised with DTF and other departments with a view to working out how to overcome them.

DTPLI response

Overall DTPLI agreed with these findings but asked us to acknowledge the impact of Machinery of Government (MoG) changes and the amendments made to its objectives and objective indicators in the draft 2014–18 corporate plan. DTPLI also provided useful commentary on the practical barriers that it would need to overcome to achieve the government's performance measurement and reporting goals.

While DTPLI agreed that progress in applying government's requirements had been slow, and that the pace of effective change needs to accelerate, part of the explanation for this was that the timing and impact of the MoG change in April 2013 meant the new department could do little else than combine the components of the two merged departments for the 2013–14 Budget.

The 2014–15 draft plan includes revised objectives and objective indicators tailored for the combined planning, transport and local infrastructure department. Figure 2C in Part 2 of this report shows the changes, and DTPLI thinks it addresses some of VAGO's earlier findings.

DTPLI raised the following practical challenges in applying current requirements and:

• agreed that the objectives in the 2014–15 Budget Papers only partially met DTF's requirements but considers that these should focus on the intended outcomes—describing target beneficiaries, service standards and time frames are more appropriately addressed through the objective indicators
• noted that the need to retain measures to provide continuity of performance reporting, MoG changes and the changes to government or amended policy objectives could affect the ability and pace of change.

These comments did not change our core findings around the inadequacy of current frameworks and the overall slow pace of change, but provided valuable insights into the reasons for this and the challenges around addressing these issues.

Department of Premier and Cabinet response to findings and VAGO's conclusions

The Department of Premier and Cabinet (DPC) identified issues that present significant challenges to departments successfully transitioning to an outcome-focused performance framework:

• DPC acknowledged that the ideal foundation for this type of framework is where well-defined outcomes drive what should be measured. However, performance reporting and revenue certification remain output-driven and this presents a significant challenge in transitioning to an outcome-focused framework.
• The requirement to aggregate services and programs into larger output groups, while limiting the number of objectives, made it challenging to link specific outputs to what are more generic objectives. The current output-driven framework restricts the extent that departments can connect the specific activities supporting a particular outcome.

DPC also noted that it would reasonably take time for an improved system to reach maturity, especially since solid examples of best practice are not readily available. Departments will publish their first report of progress against objective indicators in their 2013–14 annual reports and DPC suggest it will be difficult to improve on existing indicators until a full reporting cycle has been completed.

4.4 Reviewing compliance and performance issues

4.4.1 Meeting government's measurement requirements

Departments complied with the process requirements around updating and submitting revised performance frameworks except for the submission of:

• 2013–17 corporate plans where, by the agreed deadline, four departments had lodged final plans—including DPC and DTPLI—four departments had lodged draft plans and DH needed more time to prepare a plan
• long-term plans where four departments—including DH, DPC and DTPLI—only submitted draft not final plans.

While departments complied with processes by defining objectives and objective indicators and associating these with output groups, our review of three departments shows they did not fully meet government's requirements. Part 2 of this report highlights these substantial deviations which meant they did not:

• adequately measure and explain output efficiency and effectiveness
• set objectives and objective indicators that meet government's requirements
• consistently link outputs with objectives and objective indicators
• provide meaningful commentary on performance, which makes it difficult to understand how departments have performed
• deliver corporate and long-term plans that meet government's requirements.

We have seen evidence that DTF reviewed and reported on departments' updated performance measurement frameworks and their application in mid-year and end‑of‑year assessments and advised government on instances where departments should:

• increase the number of performance measures to enhance transparency
• disaggregate performance measures
• adjust objectives and objective indicators
• advise on the measurability of objective indicators
• advise on new and discontinued measures.

While these assessments analysed output trends and highlighted important issues, they did not consistently identify the weaknesses and flaws in departments' performance measurement frameworks.

For example, DTF has never raised the absence of information about efficiency, the absence of commentary on the achievement of outcomes, nor clearly communicated where objectives and objective indicators did not meet government's requirements—we highlighted these flaws in Part 2 of the report.

DTF needs to be more rigorous in identifying weaknesses and communicating these to departments so they can be addressed.

4.4.2 Reliability of the data used for performance measurement

The BFMG states that departments and DTF should:

• regularly review information collection systems and processes to ensure that data sets are useful and relevant for both internal and external needs
• jointly review departmental outputs and the performance measures used to evaluate service delivery annually for their continuing relevance and robustness.

The clear structural weaknesses in departments' performance measurement means we have focused our effort on understanding these issues and what has caused them.

In terms of reviewing the robustness and rigour of the data, we requested evidence of adequate review from departments.

While we found that DTF and departments reviewed performance measures each year, the evidence of adequate review of the data underpinning these measures was unconvincing:

• DTPLI required each business group responsible for providing performance measures to complete a one page self-attestation describing the calculation method and compliance with its data policy. We did not find evidence of periodic or sampled audits to verify this information.
• DH provided three reports from July 2013 to January 2014 that reviewed elective surgery, admitted patient and emergency admission data. These cover a portion of the reported Budget output measures and vary in quality, with the admitted patient report stating that it did not meet auditing standards.
• DPC provided us with details of the source data used in its performance measures but no evidence that its reliability had been verified.

From the evidence we have seen we cannot verify that departments have applied the BFMG business rule that 'The data and methodology underpinning performance measures should be auditable and verified for accuracy…'—BFMG 9 business rule 5.

4.5 Informing government about progress and performance

DTF has regularly briefed the Minister for Finance on departments' progress in applying government's performance reporting requirements. It also provides government with a mid-year briefing on agencies achievements against the targets in the Budget Papers and the major, common issues emerging from this analysis.

DTF and portfolio departments also analysed and reported on changes in output measures. This produced some useful information especially where important areas of service delivery were falling short of output targets. However, its value is diminished by the structural weaknesses in current performance measurement frameworks.

These analyses should be expanded to report on outcomes, but the ability to do this is undermined by gaps and weaknesses in departments' objective definitions and outcome measurement.

While DTF's advice on the implementation of government's corporate and long-term planning requirements accurately reflected the lack of substantial progress, the same is not true for the implementation of objectives and objective indicators.

In our view DTF's analysis did not highlight what we consider to be clear shortfalls in departments' performance and consequent slow progress.

Recommendations

That the Department of Treasury and Finance:

1. improves the guidance material on performance measurement to include more practical examples to help departments measure efficiency and effectiveness and link outputs to departmental objectives
2. more rigorously and consistently assesses and communicates performance back to portfolio departments and government.

RESPONSE provided by the Secretary, Department of Premier and Cabinet

RESPONSE provided by the Secretary, Department of Health

RESPONSE provided by the Secretary, Department of Transport, Planning and Local Infrastructure

RESPONSE provided by the Secretary, Department of Treasury and Finance