Environment and Sustainability Sector: Performance Reporting

Background

Our natural environment underpins our economic and social opportunities and is an asset for all generations.

Government activities to achieve a healthy and sustainable natural environment are delivered through a range of public sector entities including the Department of Environment and Primary Industries (DEPI)—formerly the Department of Sustainability and Environment—the Environment Protection Authority (EPA), Parks Victoria, Sustainability Victoria, water authorities, catchment management authorities and waste management groups.

Performance measurement and reporting are crucial for determining whether the public sector is operating effectively, efficiently and in compliance with laws and policies. Previous audits across government have repeatedly identified shortcomings in agency performance reporting. Measuring and communicating performance is one of the most common persistent and recurring challenges revealed in our audits.

Reporting on performance has often been found to be limited in its usefulness and reliability. This reduces confidence about program effectiveness and quality of decision-making.

In the past three years, audits that have considered performance reporting in the sustainability and environment sector include Performance Reporting by Departments, Environmental Management of Marine Protected Areas, Control of Invasive Plants and Animals in Victoria's Parks, and Hazardous Waste Management. These audits have influenced the subsequent development of performance and reporting frameworks and controls in this sector.

The objective of this audit was to assess the effectiveness of public performance reporting by three of these agencies:

• DEPI—Victoria's lead government agency for the sustainable management of the natural environment, responsible for coordination of activities across the sustainability and environment agencies
• EPA—a statutory authority with independent powers to make regulatory decisions under the Environment Protection Act 1970
• Parks Victoria—a statutory authority responsible for managing a diverse park estate covering 17 per cent of Victoria, including terrestrial and marine parks, under an agreement with DEPI.

Conclusions

DEPI, EPA and Parks Victoria have clearly defined objectives and outputs. However, the selection of indicators and measures used to demonstrate performance is not always clear and not all indicators and measures for which data are collected are included in public reports. There continues to be an emphasis on reporting outputs and activities rather than outcomes.

DEPI, EPA and Parks Victoria have each demonstrated improvements in the practices used to collect data and report performance information. However, the quality and clarity of these processes varies across the agencies. The data selection and management, and the reporting controls and processes across the sustainability and environment portfolio as a whole do not provide sufficient confidence in the consistency and reliability of reported performance information.

Findings

Performance reporting framework

Good performance reporting provides sufficient information to reliably, objectively and clearly reflect an agency's performance.

Performance reporting in the audited agencies is mandatory. They must report on their performance in their annual reports, and departmental performance statements published in the government's Budget Paper No. 3–Service Delivery (BP3). Agencies may also produce other reports on performance for specific purposes.

DEPI, EPA and Parks Victoria have clearly defined and well integrated objectives and outputs for both BP3 reporting and broader agency operations. Their annual reports meet the minimum information requirements of the Financial Management Act 1994—including relevant Standing Directions.

Outside the BP3 framework, organisational objectives are well documented in corporate plans, long-term strategic and work plans and public reporting documents for each agency. The objectives are clearly defined, results-based and measurable, and adequately cover strategic priorities, key programs and areas of responsibility.

Performance indicators and output measures are generally well integrated across agencies' program planning and delivery. However, the design and selection process for performance indicators and output measures has not been adequately documented.

While EPA and Parks Victoria publicly report on a wide range of performance indicators and output measures, DEPI maintains that it is not required to report publicly on performance measures outside of those established in the State Budget process.

Despite establishing and maintaining data against a more extensive range of performance indicators and output measures, DEPI reports only on those measures contained in BP3. It may be appropriate for DEPI to publish this information if it provides a more comprehensive view of its overall performance.

Although DEPI has made some progress toward reporting performance more broadly across the environment and sustainability portfolio, this is not supported by a clearly articulated and structured performance reporting plan or strategy. Given DEPI's expanded portfolio coverage, which now includes primary industries, this work needs to be undertaken as soon as possible. This would allow DEPI to provide a more complete and reliable picture of its performance, consistent with clear and focused strategic and corporate plans.

Selecting and documenting indicators and measures

There is inadequate documentation about the rationale behind the selection of performance indicators and output measures, and a lack of complete or centrally controlled records that include agreed measurement methodologies.

Adequate documentation outlining the selection of indicators and measures helps to inform the choice of data sources so that reported data best reflects agency performance. It also allows for the appropriateness of measures to be reviewed. Neither DEPI nor Parks Victoria adequately document the information and processes used to select performance indicators and output measures. The identification of data sources and definition of data methodologies is not adequately built into the selection process, which has implications for the consistent collection and reporting of data.

DEPI does not have centrally developed and monitored standards for how program areas record and manage detailed information about their indicators and measures. This leads to a variety of approaches with inconsistencies in the quality of information recorded about the measures used.

Parks Victoria uses a range of data sources to inform the measurement of indicators and output measures. However, it has not demonstrated the basis for choosing each dataset, and how each is to be collected. Parks Victoria has recently commenced recording more comprehensive information about performance measures in information sheets.

EPA's highly integrated system clearly documents and defines all relevant information for its performance indicators and output measures, and related governance structures. Operational data and externally sourced scientific data is clearly and consistently justified and defined.

Both Parks Victoria and EPA have recently sought to modify their BP3 output measures to better reflect their current strategic priorities and objectives. A number of these changes have not yet been agreed to by DEPI or Department of Treasury and Finance. However, mechanisms are now being put in place to address this issue.

Data collection, management and reporting

Performance reporting must be based on reliable data that properly reflects the agency's performance. Agencies should have systems in place to ensure robust data collection and performance reporting that are well documented, consistently applied and effectively controlled.

All three audited agencies have centralised reporting systems that collate the data used for reporting. However, the controls over data collection and processing leading up to their collation vary markedly. Confidence in the accuracy of performance data, and the reliability of performance reports, is undermined by a lack of consistent standards and processes used for the collection, recording and validation of the data used to measure performance.

All agencies have established documented processes for each stage of report development and have identified those responsible for its authorisation. However, clarity of tasks and quality assurance processes for report development among the three agencies are considerably different.

EPA's approach to data specification, data collection and assurance, and report development represents best practice.

Continuous improvement

Good performance reporting approaches embed review processes and facilitate continuous improvement. Systems should be in place to routinely identify performance framework and system improvement needs, and to plan specific actions to implement them.

While DEPI and Parks Victoria have undertaken reviews of parts of their performance reporting systems and made improvements in recent years, these have been ad hoc or in response to external reviews. DEPI and Parks Victoria do not have their own internal mechanisms to review and improve their performance reporting processes.

In contrast, EPA's planning and reporting framework includes mechanisms that are used to track the ongoing development of its performance reporting, annual review of performance indicators, and assessments of data collection processes.

Recent machinery of government changes provide DEPI with both a challenge and an opportunity to review the nature of performance indicators and measures reported publicly across the environment, sustainability and primary industry portfolio.

Recommendations

1. The Department of Environment and Primary Industries should critically assess whether there is value in reporting on performance indicators and output measures beyond those in Budget Paper No. 3 in its annual report.

The Department of Environment and Primary Industries, the Environment Protection Authority and Parks Victoria should:

2. review their processes for developing, selecting and reviewing publicly-reported performance indicators and output measures, and more effectively collaborate across agencies
3. improve the documentation of the information and processes used to select performance indicators and output measures
4. assess annually how performance reporting practices comply with organisational standards, and conduct detailed biennial reviews to evaluate the adequacy of key performance indicators and output measure controls, data systems processes and controls, and development of performance reports.

The Department of Environment and Primary Industries and Parks Victoria should:

5. improve the documentation of standardised and centrally managed records to include more compete descriptions, definitions, and rationale of data for all publicly‑reported indicators and output measures
6. develop minimum standards for performance data collection, management, analytical processes and systems, and monitor compliance with these standards
7. develop and document detailed quality assurance standards for performance data collection and collation processes, and monitor compliance with these standards
8. clarify the specific tasks and responsibilities of staff at all levels responsible for report development roles, and specify the checks and approvals that are required.

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report, or relevant extracts from the report, was provided to the Department of Environment and Primary Industries, the Environment Protection Authority and Parks Victoria with a request for submissions or comments.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix B.

1.1 Introduction

A healthy and sustainable natural environment is essential for Victoria's economic and social prosperity. In 2012–13, the Victorian Government plans to provide over $1.4 billion in funding towards achieving its desired environmental outcomes.

With such a level of funding, it is important to know how government agencies are delivering environmental, social and economic outcomes. Various audiences want to know if agencies are doing what they are supposed to be doing, and whether they are doing it effectively and efficiently. They expect information about agency performance to be available, to be easy to understand, and to be accurate.

Shortcomings in performance reporting have repeatedly been identified in government departments and agencies. Reporting on performance has often been found to be limited in its usefulness and reliability, with weaknesses in building credible performance measurements into program activities and in measuring outcomes. This reduces confidence in program effectiveness and in the quality of decision-making.

The environment and sustainability portfolio comprises a range of public sector entities including the Department of Environment and Primary Industries (DEPI)—formerly the Department of Sustainability and Environment—the Environment Protection Authority (EPA), Parks Victoria, Sustainability Victoria, water authorities, catchment management authorities and waste management groups. This audit looked at performance reporting by three of these agencies—DEPI, EPA and Parks Victoria.

VAGO audits in this portfolio in the past three years that have considered performance reporting include: Performance Reporting by Departments, Environmental Management of Marine Protected Areas, Control of Invasive Plants and Animals in Victoria's Parks, and Hazardous Waste Management. These audits have influenced the subsequent development of DEPI's, EPA's and Parks Victoria's performance reporting frameworks and controls.

1.2 Assessing agency performance

1.3 Reporting requirements

1.4 Roles and responsibilities

1.5 Audit objective

The audit was conducted under section 15 of the Audit Act 1994 and in accordance with the Australian Auditing and Assurance Standards. Pursuant to section 20(3) of the Audit Act 1994, unless otherwise indicated, any persons named in this report are not the subject of adverse comment or opinion.

The total cost of this audit was $350 000.

1.7 Structure of the report

2.1 Introduction

Good performance reporting provides sufficient information to reliably, objectively and clearly reflect an agency's performance.

Effective performance measurement and reporting systems use robust and well‑developed performance indicators and output measures. They provide insights into the extent to which departments and agencies are achieving their objectives and program deliverables.

Performance indicators and output measures should be carefully selected to demonstrate the effectiveness and efficiency of an agency's activities in meeting the desired outcome.

2.2 Conclusion

The Department of Environment and Primary Industries (DEPI), the Environment Protection Authority (EPA) and Parks Victoria have clearly defined and well integrated objectives and outputs, for both Budget Paper No. 3 (BP3) reporting and broader agency operations.

However, there is inadequate documentation about the rationale behind the selection of performance indicators and output measures. Because of this, these agencies cannot demonstrate the logic and assumptions involved in setting the indicators and measures, or assure that they are the most appropriate to reflect performance. Lack of documentation also reduces confidence in the choice of data sources and methodologies used to report on performance.

While EPA and Parks Victoria publicly report on BP3 performance measures as well as a range of other indicators and measures, DEPI reports only on the output measures contained in its BP3 report. This means that its performance reporting may not give a complete picture of how it is performing against its corporate and business plans, and gives limited information about progress towards achieving departmental objectives.

DEPI and Parks Victoria have made some improvements in their performance reporting frameworks in recent years. However, they do not have adequate mechanisms for reviewing and improving their performance reporting processes.

2.3 Objectives and outputs

2.4 Publicly-reported performance information

Victorian public sector agencies are required to report performance against the BP3 output measures as agreed through the Budget process. This occurs in consolidated form for all outputs within the sustainability and environment portfolio, in the Budget Papers and in DEPI's annual report. While its annual report does not identify which agencies within the portfolio are responsible for individual outputs or performance measures, DEPI expects the individual agencies' annual reports to include reporting against the relevant BP3 performance measures—which they do.

DEPI's, EPA's and Parks Victoria's annual reports meet the minimum requirements of the Financial Management Act 1994—including relevant Standing Directions.

However, agencies are able to include additional performance reporting in their annual reports, or in other publications. This is particularly important if activities are not directly related to the use of funds provided through the Budget process, but instead relate to, for example, how an agency is exercising its legislative powers to make decisions, or to particular programs or initiatives.

EPA and Parks Victoria publicly report on a broader range of indicators and measures than required under the BP3 reporting requirements.

EPA has three strategic priorities to deal with past pollution, tackle current environmental issues, and shape the future environment. Ten objectives sit under these priorities—such as tackle pollution, ensure compliance using a risk-based approach, and support the conversion of contaminated environments to assets. To report progress against these objectives EPA uses 17 performance indicators and its eight BP3 output measures. Performance indicators include notices issued for particular activities and compliance of audits.

EPA's annual report is a highly detailed and informative public document that informs its readers about relevant context and significance, and provides useful comparisons. This is, in part, driven by a stronger focus by EPA's board and management on the need to be more community and client focused and clearly assert its strategic relevance as a modern regulator.

EPA produces an annual corporate plan that informs the public of its planned and proactive compliance activities over the coming year. The plan includes a range of targets, against which EPA reports progress quarterly. The plan provides a more detailed account of the targets and activities identified in EPA's annual report and articulates how its five-year plan priorities are delivered each year.

Parks Victoria's 2011–12 Business Plan and 2011–14 Corporate Plan set out five areas of work and five related strategic, long-term goals or objectives. Its objectives include conservation of park and waterway ecosystems, protection of culture and heritage in parks, and delivering opportunities for people to use and enjoy their parks and waterways. Under these objectives are 22 performance indicators and five BP3 output measures.

Parks Victoria's State of the Parks is a separate non-mandated reporting process that focuses on the effectiveness of its park management activities across the state. The report includes a systematic evaluation of the outcomes of park management programs, and the extent to which park management objectives are being met. It also highlights current challenges, identifies emerging issues, and is used to inform and modify management programs.

While Parks Victoria annual reports include a wider range of indicators than the BP3 output measures, this reporting lacks sufficient detail and analysis of performance. Parks Victoria is currently addressing this through a new reporting style and templates.

DEPI only provides performance data on the BP3 output measures as required under Standing Direction 4.2. This limits the transparency and comprehensiveness of its reporting. Further, while DEPI annual reports do include explanations of variances against targets, there is no discussion of trends, longer-term issues, context, or comparative data.

Performance indicators and output measures—including BP3 output measures—that are currently publicly reported are listed in Appendix A.

2.4.1 What other performance information could be reported?

There are additional data that agencies collect that may assist users of performance information.

While EPA and Parks Victoria report a range of measures in addition to those in BP3, DEPI does not. DEPI has established and collects data against a more extensive range of performance indicators and output measures than those currently reported publicly, which are used only for internal management purposes.

DEPI has identified 12 departmental objective indicators as its performance indicators to measure progress in achieving its four organisational objectives. Data is being collected for these indicators, but not reported externally. These departmental objective indicators include items such as the proportion of new homes connected to alternative water sources, and the number of community members who visit the public land managed by portfolio agencies.

Additional output measures monitored by DEPI, but not reported, reflect the quantity and timeliness of many of its core activities.

These departmental objective indicators and additional output measures are not required to be reported publicly as they are not currently part of the BP3 reporting framework, however, DEPI could explore reporting against them in future annual reports.

While not formally required, publication of such data would be in the public interest if it promotes an open and transparent government, assists accountabilities and supports ongoing policy development. It also provides a more complete picture of an agency's performance that may not be reflected from a Budget-specific perspective.

In principle, if an agency collects performance data for internal reporting, it should also consider using it for public reporting on its performance, subject to consideration of its reliability and accuracy. DEPI's annual reports are based on compliance with the formal requirements for BP3 reporting.

Although DEPI has made some progress toward reporting performance more broadly across the environment and sustainability portfolio, this is not supported by a clearly articulated and structured performance reporting plan or strategy. Given DEPI's expanded portfolio coverage, which now includes primary industries, this work needs to be undertaken as soon as possible. This would enable DEPI to provide a more complete and reliable picture of its performance, consistent with clear and focused strategic and corporate plans.

2.5 Performance indicators and output measures

Each agency has a generally well integrated planning process to identify relevant performance indicators for each of their objectives. However, there have been some gaps in these processes including:

• documentation of the rationale for selection of indicators and measures
• inconsistent methodologies and definitions for the data needed to measure performance
• unsatisfactory processes for the review and revision of measures.

2.5.1 Performance indicators and output measures used by agencies

Performance indicators

DEPI's 2012–23 Long-Term Plan lists 12 departmental objective indicators. The plan explains the alignment of each indicator to its respective objective and how they collectively cover the full suite of strategic priorities.

The objective indicators contained in DEPI's proposed 2012–16 corporate plan are included in the 2013–14 BP3 reporting framework, however, no targets or results have been included. They are documented internally to support strategic, business and operational planning, and the delivery and reporting of performance across all levels of the department. Reporting against these indicators is, therefore, highly relevant to inform an assessment of performance.

The indicators used by EPA to publicly report performance against its corporate objectives adequately cover its new strategic objectives, and the range of activities used to achieve its corporate plan and its compliance and enforcement plan. For each identified objective, indicators are clearly linked to activities and have appropriate targets. These enable EPA to report on how effectively it has been achieving its chosen objectives.

Parks Victoria has recently committed to a new series of long-term outcomes and determined what will be required for their achievement. It has selected indicators for public reporting that are logically aligned and relevant to the objectives and outcomes contained in its corporate and business plans.

Appropriate indicators have also been identified for Parks Victoria's State of the Parks report, which it intends to publish every three years. These indicators were developed following extensive consultation and engagement with the public and relevant government stakeholders. The indicators adequately cover all of Parks Victoria's major park management activities.

Output measures

DEPI reports against 71 performance measures to reflect its achievement of outputs. These measures align with DEPI's outputs and cover the key activities included in its business plan.

Parks Victoria monitors and reports internally on a total of 29 performance measures. Only five of these are included in the BP3. Parks Victoria has acknowledged that the performance measures included in the BP3 are not suitable and should be changed to more accurately align with current organisational objectives and plans. The current performance measures:

• are not clearly expressed and therefore are difficult to understand
• are often statements of fact rather than targets for performance
• do not reflect Parks Victoria's performance achievements.

Parks Victoria requests to DEPI and DTF to modify its BP3 output measures have not resulted in these being changed. However, DEPI is developing a new management agreement with Parks Victoria to create a schedule that includes the review of the performance measures.

EPA measures a total of 61 output measures through its corporate scorecards, which align with its objectives and directly link to the activities and targets under its corporate and business plans. EPA's measures are organised against each of its program areas. Only eight of these measures are included in BP3.

EPA has acknowledged that its performance measures reported in BP3, measure only some of its activities and do not closely reflect its five-year plan. Over recent years, EPA has sought changes to its BP3 output measures. While BP3 measures have been modified in three instances there are a further five changes that have not been agreed between DEPI and EPA. The changes wanted by EPA would provide flexibility to report program delivery more clearly and accurately, and measure additional activities related to the delivery of its objectives. DEPI has recently established a senior management coordination working group that has, as one of its tasks, to review and provide advice on the requests from EPA for these changes.

2.5.2 Processes for selecting indicators and measures

There is inadequate documentation about the rationale behind the selection of performance indicators and output measures. There is also a lack of complete or centrally controlled records that include agreed measurement methodologies and data sources.

Having the right performance indicators and output measures is only part of the story. The selection of indicators and measures should be adequately documented to:

• demonstrate the logic and assumptions involved in setting the indicators and measures
• inform the choice of data sources so that reported data best reflects achievement of targets
• allow reviews to determine if the measures remain appropriate.

Figure 2B

Appropriate documentation of performance indicators and output measures

Source: Victorian Auditor-General's Office.

Program logics and documented descriptions of the use of individual indicators and measures are elements of best practice in performance reporting. They allow an agency to record the nature of the performance indicators used—for example, based on survey or administrative data, sample-based or exhaustive—their rationale, the definition of all major components of the indicator, who collects it and who provides the data. They need to record the scope of the data including the frequency of data collection or provision, and whether information from different data sources will be combined or aggregated in some manner for that indicator. They should facilitate an agency-wide understanding of how the organisation's objectives link to chosen performance indicators, and outputs are linked to output measures.

Neither DEPI nor Parks Victoria adequately document the information and processes used to select performance indicators and output measures. They do not adequately identify data sources and define data methodologies as part of the selection process. This has implications for data collection and reporting—this is discussed further in Part 3.

2.6 Continuous improvement

Performance reporting frameworks and systems require ongoing review and improvement to ensure that they can provide the most reliable assessment of an agency's performance.

DEPI and Parks Victoria have undertaken reviews of parts of their performance reporting systems and made improvements in recent years.

DEPI is making significant progress in reforming its reporting following VAGO audits and other reviews by the Parliamentary Accounts and Estimate Committee and DTF. Since 2010, DEPI has:

• developed the Interplan planning and reporting tool
• recommenced development of corporate plans—including the development of departmental objectives
• developed draft performance indicators
• reviewed its BP3 measures against DTF guidelines.

DEPI has established a performance reporting reference group, which meets periodically and provides feedback about the general efficiency of performance reporting processes and reports.

Parks Victoria has also improved its performance reporting in response to internal reviews of its strategic planning frameworks, and improvements to some administrative practices. Parks Victoria's Corporate Strategy Directorate documents the improvements identified throughout the business plan cycle. Where applicable, enhancements are implemented immediately if they improve the current planning cycle.

These improvements have been ad hoc or in response to external reviews. DEPI and Parks Victoria do not have adequately structured mechanisms for review and improvement built into their performance reporting processes, or clear strategies that guide the development of their performance management systems.

EPA has enhanced its corporate planning processes and undertaken a business systems reform program to re-engineer its operations platform. This has resulted in a clear vision, objectives, milestones and review processes. These processes are clearly resourced and their implementation is planned and monitored.

EPA's planning and reporting framework includes annual review mechanisms that track the planned developments of its performance reporting, performance indicators and data collection processes.

Recent machinery of government changes provide DEPI with both a challenge and an opportunity to review the nature of performance indicators and measures reported publicly across the environment, sustainability and primary industry portfolio.

Recommendations

1. The Department of Environment and Primary Industries should critically assess whether there is value in reporting on performance indicators and output measures beyond those in Budget Paper No. 3 in its annual report.

The Department of Environment and Primary Industries, the Environment Protection Authority and Parks Victoria should:

2. review their processes for developing, selecting and reviewing publicly-reported performance indicators and output measures, and more effectively collaborate across agencies
3. improve the documentation of the information and processes used to select performance indicators and output measures
4. assess annually how performance reporting practices comply with organisational standards, and conduct detailed biennial reviews to evaluate the adequacy of key performance indicators and output measure controls, data systems processes and controls, and development of performance reports.

3.1 Introduction

Performance reporting must be based on reliable data that accurately reflects agency performance. Agencies should have effective systems to support robust data collection and collation, and performance reporting processes that are well documented, consistently applied and effectively controlled.

Appropriate data management practices are vital to support performance management and reporting programs. Effective data collection and management practices—including storage, access and analysis—assist with data reliability and the production of timely and consistent performance reports.

Agencies should systematically evaluate and select the most appropriate type and quality of data to inform performance indicators and output measures in a relevant, appropriate and representative way.

3.2 Conclusion

The Department of Environment and Primary Industries (DEPI), the Environment Protection Authority (EPA) and Parks Victoria have improved the practices they use to collect data and report performance information.

However, the quality and clarity of these processes varies across the agencies. The data selection and management, and the reporting controls and processes across the sustainability and environment portfolio as a whole do not provide sufficient confidence in the consistency and reliability of reported performance information.

3.3 Data specification and sources

Adequate documentation outlining the selection of performance indicators and output measures helps to inform the choice of data sources so that reported data best reflects performance.

The processes for selecting performance indicators and output measures at DEPI is inconsistent and often poorly documented.

There are no centrally developed and monitored standards for how program areas record and manage detailed information about their indicators and measures. This leads to a variety of approaches, with inconsistencies in the quality of information recorded about the measures used. DEPI's central planning and reporting system, Interplan, is not used to centrally register sufficient information about performance indicators and output measures, or as a mechanism to standardise what information should be captured.

DEPI does not have adequate centrally controlled records that consistently and clearly define and guide the data used for its performance indicators or output measures. In some instances data definitions and controls are comprehensive and align well with indicators and measures, but in other cases data sets are poorly or incompletely defined.

DEPI has prepared performance indicator templates for 10 of its 12 departmental objective indicators. However, the information in these documents is inconsistent in terms of scope, quality, clarity and completeness. In many cases they do not adequately define or sufficiently cover the rationale of the indicator they relate to. There are similar shortcomings in DEPI's documentation of its Budget Paper No. 3 (BP3) output measures.

Parks Victoria uses a range of data sources to inform the measurement of indicators and output measures, however, it has not demonstrated the basis for choosing each dataset and how each is to be collected. Parks Victoria has recently commenced recording more comprehensive information about performance measures in information sheets—2012–13 is the first year it has used this process and it plans to test and refine the information during the coming financial year.

EPA has developed a highly integrated system that combines operational data with externally sourced scientific data, each of which is clearly and consistently justified and defined. The system has a central log that includes all relevant information for its performance indicators and output measures, and related governance structures.

The indicator sheets used by EPA record the scope of each measure, its origins, the intent and purpose, and what the outcome of measuring this actually means for EPA's delivery of its strategic priorities. They also specify the source of data that will be used for each measure.

3.4 Performance data collection, validation and management

Agencies need established systems and controls to collect, validate and assure the consistency and accuracy of data collation, and to provide consistent reporting over time. Effective controls over data collection ensure that the data provided for reporting is complete, accurate, consistent, and correctly recorded and processed.

All three audited agencies have centralised reporting systems that collate the data used for reporting. However, the controls over data collection and processing leading up to their collation vary markedly.

3.4.1 Environment Protection Authority

EPA has established clear operating procedures and detailed indicator sheets for its performance measures and provides guidance to staff involved in collection and collating data.

The data collection controls, delivered using its new Integrated Business Information System (IBIS), reflect better practice. The controls provide EPA with a high level of confidence about the accuracy of data used in reporting. IBIS is automated, centralised, secured and provides consistent processes and controls for data collection.

EPA has a well-controlled, systematic and consistent approach to quality assurance for its performance data.

IBIS has an automated centralised data depository. There is an information management system strategy and associated plan, which provide guidance to staff on procedures. Clear controls exist around data input procedures to facilitate consistency, and an extensive training program has been implemented to assist staff using the system.

EPA has nominated data stewards who are appointed for each data set and a scorecard owner for each performance indicator and output measure. The data stewards undertake scheduled reviews of the veracity of data content. They are responsible for overseeing the data collection processes, including monitoring controls and staff training. The stewards group meets regularly and reports to a data owners committee. Actions identified in these reviews are included in IBIS's implementation action plan, which identifies various continuous improvement activities.

Figure 3A

How the Environment Protection Authority demonstrates better practice principles

Source: Victorian Auditor-General’s Office.

3.4.2 Parks Victoria

Parks Victoria has made progress in improving its processes and controls for the collection and management of its performance data. However, these are yet to be adequately documented or fully embedded into consistent practice.

A strategic planner informally monitors processes and intervenes with improvements from time to time, but these processes are insufficient to ensure that all risks to poor reporting are identified or acted upon.

There is reliance on manual data collection and although staff roles and authorisations are well established, they are not yet supported by rigorous and well documented quality assurance processes. The centralised data management system lacks standard operating procedures for manual data collection and collation processes. The use of old and unreviewed data guidance and protocols further undermines Parks Victoria's data management practices.

Parks Victoria has not reviewed the adequacy of its data collection and management practices to assure that:

• data management systems are appropriately documented, governed and secured
• data collection processes are precise and adequately documented
• responsible officers are appropriately qualified and aware of these processes
• quality assurance processes are appropriate and clearly defined.

Data collation involves staff manually re-inputting collected data into spreadsheets. This process does not include central control or oversight to check and assess the accuracy of information. This process creates a risk for the reliability of the data provided, which could undermine the usefulness of performance reporting.

Data collected for the State of the Parks report is more adequately controlled, which is reflected in a high degree of consistency in the quality of information collected and reported.

3.4.3 Department of Environment and Primary Industries

DEPI's larger, more complex organisational structure and the diversity of its programs has resulted in a large number of decentralised data management systems and processes used to collect data for program-based performance measures.

There are no common standards or approaches to ensure adequate checking and validation of data collection processes across the department. DEPI officers who collect performance data do not have access to clear and consistent definitions about the performance indicators and output measures used, or the specific data to be collected.

DEPI has not assessed the adequacy of its data collection processes across the department and does not have any plans to do so. It has not identified or formally examined the risks that may lead to poor data collection. While the centralisation of report development using the Interplan process has greatly reduced some risks, significant variability remains in many of the decentralised data collection processes and controls creating the potential for human error or poor quality reporting.

DEPI has few well-defined and systematic quality assurance processes to validate its collected data. Checking processes are informal and often based on an individual officer's familiarity with the data or a sense of whether it looked right. There were few examples of crosschecking of data with a second source or verifying compliance with specific data collection processes.

Only two of six indicators reviewed during this audit showed they were supported by controlled and well-documented data systems. The other four had significant flaws in the definitions of the data collected, or the required methodologies for data processing, including definitions that were:

• too general to be useful
• missing important information about how data is aggregated
• confusing or used variable methods of calculation
• missing sufficient explanation of how the data chosen is aligned to the indicator description.

Two out of three BP3 output measures examined in this audit had poor data collection controls that led to poor understanding and application of indicator definition, and poor collection and checking processes. This creates a risk that unreliable data could be provided for Budget reporting purposes.

DEPI has established a performance reporting reference group, which includes a range of divisional/regional business managers and DEPI's corporate planning and reporting area. The group meets periodically to provide and consider feedback about the general efficiency of performance reporting processes and reports. However, it does not undertake structured reviews that test the quality of data collection, or report development standards and practices.

3.5 Report development processes

All agencies have established documented processes for each stage of report development and have identified those responsible for its authorisation. However, clarity of tasks and quality assurance processes for report development among the three agencies are considerably different.

Users of performance reports expect published performance information to be accurate—whether in agency annual reports, Budget Papers, or stand-alone publications. The development of departmental performance reports generally involves a wide range of officers and executives from different program areas—each contributing, verifying and approving relevant information and analysis to be used in public reports. Central coordination and oversight is therefore important to ensure that the standards and processes applied during report development provide the necessary assurance about the quality and accuracy of the report information prior to its publication.

EPA has defined clear processes and requirements for developing reports that are aligned with its governance structures and quality assurance practices. This enables it to have confidence that reported information has been reviewed and is accurate. Quality assurance is automated through IBIS and incorporated into executive level reviews.

The documentation available for DEPI and Parks Victoria staff does not clearly articulate the responsibilities and tasks of officers involved in report development. The processes used by these agencies rely on the perceptions, experience and initiative of the individual officers involved.

DEPI uses standardised reporting templates and guidance materials. However, these do not include sufficiently detailed descriptions of the types of information to be included in reports, or the evidence that should be available to support this information.

At the time of audit, Parks Victoria was drafting guidelines for report development.

Figure 3B

How the Environment Protection Authority's approach delivers better practice performance reporting

Source: Victorian Auditor-General’s Office.

Recommendations

The Department of Environment and Primary Industries and Parks Victoria should:

5. improve the documentation of standardised and centrally managed records to include more compete descriptions, definitions, and rationale of data for all publicly-reported indicators and output measures
6. develop minimum standards for performance data collection, management, analytical processes and systems, and monitor compliance with these standards
7. develop and document detailed quality assurance standards for performance data collection and collation processes, and monitor compliance with these standards
8. clarify the specific tasks and responsibilities of staff at all levels responsible for performance report development roles, and specify the checks and approvals that are required.