Measuring and Reporting on Service Delivery

Does the way Victorian government departments measure and report on their service delivery support accountability and good decision-making? 

Video presentation

Video transcript

Key facts

What we found and recommend

We consulted with the audited agencies and considered their views when reaching our conclusions. The agencies’ full responses are in Appendix A. 

Measuring outcomes 

Clear objectives are the foundation of a meaningful performance measurement system because they define the desired outcomes that performance will be measured against. The Department of Treasury and Finance's (DTF) Resource Management Framework (the Framework) requires departments to set clear objectives and report on their progress towards achieving them.

Departments report their objectives in the Budget Paper No. 3: Service Delivery (BP3). While most departments have set clear objectives, BP3 includes examples of objectives that do not clearly express the desired outcome the department aims to achieve. For example: 

• the Department of Justice and Community Safety's (DJCS) objective, ‘Effective management of prisoners and offenders and provision of opportunities for rehabilitation and reparation’, states DJCS's responsibilities in regards to correctional services, not the intended outcome, which would likely relate to reduced recidivism 
• the Department of Premier and Cabinet's (DPC) objective, 'High-performing DPC', does not express the intended outcome for the community or other departments for the services it provides. 

In these circumstances, it is difficult to understand the goals that departments are working towards.
We also found multiple examples of objective indicators that do not meet the Framework's requirements and subsequently do not provide useful information about outcome achievement.

As a result, departments' performance reporting is missing key information about whether service delivery is achieving intended outcomes. This is a significant gap. Without information on outcome achievement, the government lacks a sound basis for its future investment and policy decisions.

In 2019, DPC introduced Outcomes Reform in Victoria (the Outcomes policy), which aims to improve the way departments report on their outcomes and support the creation of bespoke outcomes frameworks for specific service delivery areas. However, the policy does not articulate what relationship or priority departmental outcomes should have to departments’ objectives and objective indicators, and makes no reference to the Framework at all. 

As a result, there is a risk that departments may develop conflicting sets of outcomes and measures, de-prioritise their BP3 objectives and objective indicators, or create confusion among staff, government decision-makers, Parliament and the community about what the departments' objectives are and which performance information to use. 

Measuring output performance

An output should capture all the specific activities that make up a service and should contribute to the achievement of a department’s objective. The 2020–21 BP3 includes examples of outputs that that are too large in size or combine too many separate activities. This reduces transparency and accountability by making it difficult for Parliament and the community to understand the cost and performance of the individual services the output covers. 

Across all departments and service delivery areas, there are many output performance measures that provide useful insights into departments’ performance. However, no department consistently meets the requirements of the Framework for designing output performance measures. 

This is despite the Framework describing output performance measures as the 'building blocks of the accountability system' and the 'basis for the certification of departmental revenue'. The lack of clear and relevant output measures is a significant failure of the state's key performance and accountability framework. Without well designed output performance measures, departments cannot be held properly accountable to the government, Parliament and the community for their output performance. 

Figure A outlines the wide range of issues that limit the usefulness of departments’ current output performance measures.

Figure A: Limitations of 2020–21 output performance measures

*Note: As the time period of this audit predates relevant machinery of government changes, throughout this report we refer to DHHS, which is the predecessor agency of what are now the Department of Health (DH) and the Department of Families, Fairness and Housing (DFFH). 
Source: VAGO, based on the Framework and the 2020–21 BP3.

The frequency of issues in output measure design we observed across departments shows a lack of understanding of the Framework’s requirements and the service logic of the activities being measured. 

As DTF has an important role in reviewing and providing advice about departments' measures and as the owner of the Framework, it could do more to address this. However, DTF does not comply with some of its own requirements either. Further, despite accepting the recommendation in our 2014 audit, Public Sector Performance Measurement and Reporting, to improve its guidance material on performance measurement by including examples of efficiency and effectiveness measures and how to link outputs to departmental objectives, DTF has not done this this effectively. 

Recommendations about measuring objectives and output performance 

Using performance information

The information that departments publish provides some useful insights about elements of their performance. However, current publications of departments’ performance do not clearly demonstrate performance over time to show whether a department’s service delivery is improving or not. 

Not all departments publicly report performance results across multiple years in their annual reports, and BP3 only compares expected performance for the current year to results from the previous year. While DTF also publishes all departments' historical performance results as Microsoft Excel files on its website, the format means the user must manually create their own graphs to show performance trends. 

Given that identifying performance successes and issues is the purpose of performance reporting, the lack of trended data is a significant missed opportunity. 

To address this, we developed an interactive dashboard to show departments' performance information in a more meaningful and user-friendly way. It presents data from DTF’s website and departments' annual reports since 2008–09. 

Our dashboard shows that in 2019–20, departments reported meeting a combined total of 57 per cent of their output performance measure targets. They did not meet 37 per cent of their targets. We categorised the remaining 6 per cent as neutral measures. The dashboard is accessible on our website.

In addition to the lack of trended performance information, departments do not always meet requirements to give clear explanations when their output performance results vary by more than 5 per cent above or below target. They either fail to provide any reason or simply state that the target was exceeded or not met. Without proper explanations of the cause of variances, departments are not fulfilling Framework requirements and are therefore impairing accountability. 

Data accuracy

With the exception of DJCS, departments are also not properly documenting the business rules and data sources for their measures, which creates risks to data integrity. This is inconsistent with the Framework’s guidance. DPC has no data dictionary for its measures, and other departments' dictionaries do not include all of the required information. For example, some are missing vital items such as detailed measure definitions, calculation formulas and data sources. This lack of documentation creates a risk that departments may not collect and present their performance data consistently and accurately. 

For the selection of departments (DET, DHHS and DTF) and measures (across seven outputs) where we checked controls over performance reporting and recalculated the results, we found reasonable processes and confirmed accurate results. 

Unlike departments' financial statements, which we independently audit, there is no legislated requirement for departments’ performance statements to be independently audited either in BP3 or in departments’ annual reports.

In contrast, local government, water authorities and Technical and Further Education (TAFE) entities in Victoria are required to have their annual performance statements independently audited. Western Australia requires an independent audit of its departments' performance statements and this will also commence in New Zealand from January 2022.

The present scenario in Victoria means that Parliament and the community only have independent assurance of the accuracy and fair presentation of public sector agencies’ financial statements. Yet financial statements of public sector agencies only report on how much is spent, not how well resources have been used in the provision of goods and services. 

From this perspective, it is arguable that service delivery performance reporting on an outcome and output basis is at least equally, if not more, important than input-based financial reports. It is unclear then why non-financial service performance information obtains less assurance than financial information.

Recommendations to support useful performance reporting

Back to top

Departments measure and report on their service performance to show what they have delivered with public money. This information helps the government to allocate funding, and Parliament and the community to understand if departments are delivering efficient and effective services. 

DTF sets performance reporting requirements for departments. Each year, departments provide details of their objectives and associated performance measures, targets and results in the state's Budget papers. Departments also publicly report on their performance in their annual reports. 

1.1 Measuring performance

Governments have a broad range of service delivery obligations set in legislation as well as specific objectives expressed through government policies. Governments make investment decisions to support the achievement of their objectives and allocate funding to departments to deliver these objectives through the annual budget process. 

Departmental objectives relate to the most fundamental aspects of community life. They focus on delivering health, education and justice services, constructing and maintaining transport infrastructure, and efforts to protect the environment. As such, it is critical that departments use a performance measurement system that allows the government, Parliament and the community to understand the impact that taxpayer funded government services have on achieving these objectives. 

Government departments need to measure and report on their performance to:

• be accountable for, and transparent about, how they use public money
• monitor and benchmark their performance over time and identify opportunities to improve their services
• support government decision-making
• enable the government to assess if it is achieving its policy objectives.

To effectively measure performance, it is important that departments understand the 'service logic' of the policy initiatives and services they deliver. By using a service logic model, departments can identify the distinct parts of a 'service' and show how its funding and activities relate to its desired outcome. By identifying the parts that make up a service, departments can then design relevant performance measures that can show if the desired outcomes are being met.

This method is demonstrated by the Productivity Commission in its Report on Government Services (RoGS).

Productivity Commission's RoGS

Each year, the Productivity Commission produces RoGS to provide comparable, public information on the equity, efficiency and effectiveness of government services in Australia. 

As shown in Figure 1A, the Productivity Commission uses a service logic model to produce RoGS. This allows it to report on how government departments transform their inputs into outputs to achieve their desired outcomes. The figure also shows how performance measures can align with each part in the model.

Resource Management Framework

The Framework, which DTF updated in May 2020, is the overarching policy for the state Budget process and performance reporting. It also sets out a service logic that is similar to the one used by the Productivity Commission. Figure 1B shows that to meet government priorities, departments need to determine how their inputs and activities are converted into outputs that contribute to their objectives.

It is important that departments design performance measures that clearly relate to the part in the service logic they wish to measure. 

1.2 Measuring outcomes

Performance reporting that measures outcomes allows departments to better understand and demonstrate their impact in the community. Measuring outcomes can identify when a particular government policy is working and should be continued or expanded, or when it is not and requires change. 

Measuring the outcomes of government service delivery can be challenging because the types of outcomes that governments often seek, such as better education, are influenced by many different factors. This highlights the value of using a service logic to understand how a policy or program contributes to achieving an outcome and how best to measure it. 

In Victoria, government departments are required to report on their progress in achieving their outcomes through 'objective indicators'. These are expressed in the annual state Budget papers and departments report on their achievement against these objective indicators in their annual reports.

As set in government policy, departmental objective indicators:

• reflect the effects or impacts that the government, through departments, seeks to have on the community and other key stakeholders
• are usually set with a medium to long term (four years or more) timeframe
• describe the department’s contributions to government objectives.

In February 2019, DPC introduced the Outcomes policy to strengthen outcome reporting. The Outcomes policy acknowledges that a focus on measuring outputs does not provide information about the impact of a government activity. The policy aims to embed a more consistent approach to measuring:

• outcomes across the government
• the impact of cross-department initiatives and projects.

In alignment with this work, departments have developed a range of outcomes for specific service areas that overlap to varying degrees with their reporting on objective indicators in the Budget papers and their annual reports. These include, for example, outcomes specific to:

• family violence
• mental health
• public health and wellbeing
• community safety
• multicultural affairs
• gender equality.

Departments often undertake their own bespoke reporting against these frameworks.

1.3 Measuring outputs 

Each year, departments receive funding appropriations to deliver specific services, or 'outputs'. This is the ‘price’ the government pays for public goods and services.

As shown in Figure 1C, BP3 outlines the goods and services that the government plans to deliver across all departments. Parliament then endorses this plan by passing the annual Appropriation Bill (the Bill). The Bill gives the government the legal authority to use public money. Once the Bill is passed in Parliament, the government allocates funding to departments based on the outputs set in each department’s performance statement.

Each department is required to submit an invoice claim twice a year to certify its revenue. DTF assesses the amount claimed in the invoice against the department's output performance measure results.

As defined in government policy, an output:

• is a final product, good or service produced or delivered by, or on behalf of a department or public agency to external customers/recipients
• includes products and services delivered to the community or to other departments.

Prior to the mid-1990s, the Victorian Government funded agencies based on inputs. However, this method cannot provide assurance that departments are using their funds to optimise their outputs. 

The value in reporting against output measures and targets (which generally identify the desired volume of an output), is that it should allow the government, Parliament, and the community to identify the cost-efficiency of departmental service delivery. The results can then inform the government of the need to make funding changes or other interventions to improve efficiency where necessary.

1.4 Legal and policy framework for performance reporting

Departments' reporting obligations are governed by the:

• Financial Management Act 1994 (FMA)
• Standing Directions 2018 (the Standing Directions) issued by the Assistant Treasurer under section 8 of the FMA 
• Framework, which is issued under section 4.3 of the Standing Directions

Financial Management Act 1994

The FMA allows departments to use public money in Victoria. It outlines the accountability processes that departments and other government agencies must follow and details how they should report their expenditure. 

The Standing Directions establish standards for financial management accountability, governance, performance, sustainability, reporting and practice for government agencies.

Under the Standing Directions, DTF issued the Framework to support departments to meet the FMA's requirements.

The Framework

Departments must comply with the Framework and account for how they use public resources and achieve value for money in service delivery. Portfolio agencies that deliver services on behalf of departments must also use it. It guides departments on how to:

• set their performance objectives
• develop measures and targets to assess and report on their performance.

Requirements for departments’ performance statements

The Framework outlines how departments need to develop their yearly performance statements. It states that good-quality performance statements:

• help the government make informed decisions about allocating resources
• allow departments to develop and assess standards of service delivery in line with the government’s expectations
• allow Parliament and the community to understand the government’s performance and expenditure 
• drive continuous improvement by analysing historical performance and negotiating agreed targets from year to year.

According to the Framework, departments should:

• document the assumptions and methodology they use to collect, analyse and report on their performance results. This includes specifying how they calculate their data, the source and frequency of data collection, and any other business rules and assumptions 
• maintain performance records to a standard that allows an independent auditor to verify their integrity
• represent an appropriate proportion of the departments’ and state’s Budget. An output should not be too large or combine different services or activities because this reduces transparency and accountability.

Figure 1D sets out the Framework's requirements and guidance for performance statements. 

FIGURE 1D: The Framework's requirements and guidance for performance statements

Source: VAGO, based on the Framework.

1.5 Reporting on performance

Departments use objective indicators and output performance measures to monitor and report on their progress against their overall objectives. They do this through their internal reporting process as well as publicly reporting their results in BP3 and their annual reports. 

BP3 sets out the goods and services (outputs) that departments expect to deliver with government funding. This is organised by departmental objectives and their associated outputs.

In BP3, departments present performance statements that report their objectives, objective indicators and output performance measures and targets. This includes the expected performance for the current financial year and actual performance for the previous year.

All departments must also produce an annual report that details their financial and service performance for the previous financial year. DTF's Model Report for Victorian Government Departments (the Model Report) outlines the information departments must include. It states that departments must report four years of results against their departmental objective indicators. 

Performance statements

Performance statements in BP3 complement the financial information in Budget papers.

Figure 1E outlines the information contained in departments’ performance statements. 

Figure 1F is an example of a performance statement, in this case from DELWP, for one of its departmental objectives.

1.6 Roles and responsibilities

Department of Treasury and Finance

DTF provides advice to departments about their objectives and output performance measures but does not endorse or approve them. The relevant minister approves the sections of a department's performance statement that relate to their portfolio. 

DTF supports the Assistant Treasurer by:

• providing advice on the quality and relevance of the suite of objectives, objective indicators, outputs and output performance measures in the departments' performance statements
• reviewing the departments' output performance and advising the government on risks that may impact service delivery.

DTF also briefs the government in February on agencies' achievements against their targets in BP3. 

Our 2014 audit Public Sector Performance Measurement and Reporting identified the need for DTF to better support departments to develop meaningful performance statements and clear efficiency measures. At that time, we recommended that DTF: 

• improves its guidance material on performance measurement to include more practical examples to help departments measure efficiency and effectiveness 
• more rigorously and consistently assesses and communicates performance back to portfolio departments and government.

Government departments

Departments support their portfolio ministers in achieving the government’s objectives and priorities. As the accountable officer, a department’s secretary is responsible for:

• approving their department’s plans
• delivering outputs to the agreed performance standards
• supporting portfolio ministers to develop their department’s performance statement, medium-term plan and annual report.

Parliament

Parliament holds the government accountable for its overall performance and authorises the Bill following the annual Budget.

To strengthen accountability and transparency for performance management, Parliament's Public Accounts and Estimates Committee (PAEC), at the invitation of the Assistant Treasurer, reviews output performance measures as part of the annual Budget process.

1.7 Previous audits on service performance reporting

Numerous VAGO audits in the last two decades have found significant weaknesses in the way that departments measure and report performance. Figure 1G summarises the findings from these audits.

FIGURE 1G: Key findings from previous VAGO audits on service performance reporting

Source: VAGO.

VAGO’s December 2012 Reflections on audits 2006–12: Lessons from the past, challenges for the future summarised repeated and significant weaknesses, including:

• departments not using appropriate measures of performance
• departments failing to measure outcomes
• insufficient guidance, advice and oversight by central agencies to support departments to implement the performance measurement system.

Back to top

Conclusion

Departments have not consistently developed or reported on objective indicators that show their achievement against their stated objectives. This means departments are not meeting the Framework's mandatory requirements. More importantly, it weakens departments' accountability and transparency by preventing the government, Parliament and the community from accessing vital information about their performance. Without information on departments' outcome achievement, the government lacks a sound basis for future investment and policy decisions.

Common issues that weaken outcome measurement across departments include:

• incorrectly using output rather than outcome objective indicators  
• setting vague objective indicators that are hard to interpret and calculate results against
• not having baseline data to assess performance against.

While DPC's recent Outcomes policy aims to improve how departments approach measuring their outcomes, it misses a significant opportunity by not linking to the Framework, which is the state's primary accountability mechanism.

2.1 How departments set objectives

Objectives must express a clear, measurable achievement 

The starting point for a performance measurement system is to be clear about the desired objective of the activity you are measuring. Most departmental objectives for 2020–21 meet the Framework’s requirement that departments clearly set out the outcomes they intend to achieve with their funding. 

Examples of clear objectives that focus on outcomes include:

• ‘Raise standards of learning and development achieved by Victorians using education, training, and early childhood development services’ (DET)
• ‘Net zero emission, climate-ready economy and community’ (DELWP)
• ‘Victorians are healthy and well’ (DHHS)
• ‘Ensuring community safety through policing, law enforcement and prevention activities’ (DJCS)
• ‘Optimise Victoria’s fiscal resources’ (DTF).

In these examples, the objectives meet the expectations set out in the Framework. The intended achievement is clear, which means it is measurable. The beneficiaries are also clear—in these examples, the public.

However, we found some examples where the objective does not meet required or recommended aspects of the Framework. In some of these instances, the stated departmental objective does not identify the intended beneficiaries, although it is generally possible to infer it based on the aligned departmental indicators. The more problematic issue is where an objective expresses no intended result or outcome. This is a missed opportunity because an objective should signal to public servants the tangible purpose of their work and tell the community what benefits a department is striving to deliver.

Figure 2A gives more detailed examples.

FIGURE 2A: Examples of departmental objectives that do not clearly express the intended result (outcome) of their output delivery

Back to top

Conclusion

Across all departments and service delivery areas, there are many output performance measures that provide little genuine insight into departmental performance. This is despite the Framework describing output performance measures as the 'building blocks of the accountability system' and the 'basis for the certification of departmental revenue'. This is a significant failure by departments in the application of the state's key performance and accountability framework. Contributing issues include:

• outputs that combine too many separate activities 
• output measure selections that impair transparency
• output measures that do not measure output delivery
• output measures that are vague, outside the department's control, and/or only reflect meeting a minimum standard
• output measures that prevent comparison of performance over time or against other jurisdictions.

3.1 Setting outputs

Outputs are services that departments provide either to the community or other departments. An output should capture all the specific activities that make up a service and should contribute to the achievement of a departmental objective. 

Outputs that are too large or combine too many different activities

The 2020–21 BP3 includes examples of outputs that combine too many separate activities. This reduces departments’ transparency and accountability by making it difficult to understand the cost and performance of the individual services that an output covers. 

The Framework provides the following review criteria to help departments determine their output groupings:

• Are the services closely related or homogenous in nature?
• Are the services targeting a specific problem for the same customer?
• Is the purpose of the services the same?
• Is the output less than 10 per cent of the department’s total output cost and less than 0.5 per cent of the state’s total Budget?

The Framework states that if the answer is 'no' to any of these questions, then the output is too large. 

Despite this guidance, there are many examples that breach it. For example, DJCS’s output shown in Figure 3A, which has $237 million of funding for 2020–21.

FIGURE 3A: Example of an output that combines too many different activities

Source: VAGO, based on the 2020–21 BP3.

In other instances, output groups are very large in terms of the funding amount. Despite the Framework's requirements, if the activities within an output are truly homogenous, then it may be reasonable to group them together as one output. In this instance, the large amount of funding merely reflects the high cost and/or volume of the activities. However, it becomes problematic when too many disparate services are grouped together. In that instance, it makes it is hard to identify the performance of the various services within the output group. 

This issue was also raised by PAEC in its Report on the 2016–17 Financial and Performance Outcomes. PAEC recommended that departments improve the usefulness of their performance reporting by splitting some of their larger outputs by speciality, size or location. 

Examples of current output groups that are larger than what the Framework recommends include:

• DHHS's 'Acute Health Services' output, which has a budgeted cost of $17.065 billion (55 per cent of DHHS’s total funding and 21.4 per cent of the state Budget
• DJCS's 'Policing and Community Safety' output, which has a budgeted cost of $3.793 billion (42.4 per cent of DJCS's total funding and 4.8 per cent of the state Budget) 
• DET's 'School Education—Primary' output, which has a budgeted cost of $6.431 billion cost (37.8 per cent of DET's total funding and 8.1 per cent of the state Budget).

There is an opportunity for departments to split these output groups into smaller, more meaningful outputs. For example, 'Acute Health Services' incorporates elective and emergency services, acute and subacute (rehabilitation) services, and outpatient and inpatient services. This indicates that there is an opportunity to create more defined and homogenous output groups. Similarly, 'School Education—Primary' incorporates operational school funding and capital funding, which offers the potential for separate, smaller output groups aligned to specific purposes.

3.2 Determining a balanced suite of output performance measures

Departments need a suite of output performance measures to show accountability for their funding and demonstrate how their outputs have contributed to a departmental objective. 

The Framework sets mandatory requirements for output performance measures. It specifies that departments need to have a meaningful mix of quality, quantity, timeliness and cost performance measures for each output that assesses:

• service efficiency and effectiveness
• all major activities of the output.

However, we found numerous examples of suites of output performance measures that do not meet these requirements. 

How output measures contribute to a departmental objective 

Not all departments' performance statements present a clear link between departmental objectives, objective indicators, outputs and output performance measures. This makes it difficult for readers to understand how well a department is delivering its outputs, and whether its output delivery is making a meaningful contribution towards achieving an objective. 

To demonstrate this, Figures 3B and 3C compare objectives from DJCS's and DHHS's performance statements. While DJCS's statement presents a clear relationship between all its parts, DHHS does not have clear links between its objective indicators, outputs and output performance measures. 

Comparing these performance statement extracts highlights the importance of clear links between objectives, objective indicators and output performance measures:

A mix of quality, quantity, timeliness and cost measures

If outputs do not have a good balance of measures, departments cannot provide a comprehensive and transparent view of their performance and make informed decisions about trade‑offs in their service delivery. While this does not necessarily mean an equal number of measures across the four dimensions—quality, quantity, timeliness and cost—the Framework does require departments to have a meaningful mix. This is so users accessing the information can determine if the department may be:

• reducing quality standards to meet quantity, timeliness or cost targets
• reducing the quantity of outputs to meet quality or timeliness targets
• delaying project delivery to meet quality and quantity targets.

Figure 3D shows that despite the expectation set in the Framework that all outputs have a mix of output measures across all four dimensions, only 64 per cent of departments’ outputs meet this mandatory requirement. 

Figure 3E shows that while there is some variation in the mix of 2020–21 output performance measures between departments, 'quantity' is the most frequently used. The exception is DET, which uses more 'quality' measures and no measures of timeliness. 

Figure 3F gives an example of an output without a balanced mix of output performance measures.

DHHS's output 'Small Rural Services' includes a range of health and aged care services delivered in small rural towns and is divided into four sub-outputs: 'acute health', 'aged care', 'primary health' and 'home and community care services'. Only two of these sub-outputs have quality measures and none of them have a timeliness measure. Without these measures, DHHS cannot know whether it is providing timely, quality health services in rural communities. It is also not possible to see if DHHS is making performance trade-offs.  

FIGURE 3F: Balance of sub-output performance measures for DHHS's output group 'Small Rural Services'

Source: VAGO, based on the 2020–21 BP3.

Appendix D provides a further example to illustrate gaps in current departmental performance statements by comparing the measures that DHHS uses to assess the performance of its mental health services with those used by RoGS. 

Efficiency output measures

Despite requiring departments to set output efficiency measures, DTF includes no guidance in the Framework on how to construct efficiency output measures. In particular, it does not require departments to define the unit cost of their services. This makes it difficult to benchmark service efficiency across departments and other similar jurisdictions, and to understand if individual outputs provide value for money. 

Across all departmental output performance measures, there are only two (both for DTF) that truly measure efficiency:

• ‘Total accommodation cost ($ per square metre per year)’
• ‘Workspace ratio (square metre per FTE) [full-time equivalent]’.

DET also has four measures that measure service efficiency. However, it has incorrectly categorised these as departmental objective indicators rather than output performance measures. 

This absence of true efficiency measures across government departments reflects a lack of focus on an important aspect of government service delivery performance. 

The most common output measures in the 2020–21 BP3 are those measuring 'quantity'. It is possible to convert quantity measures into efficiency measures by combining them with cost to show the unit cost for a service. Figures 3G and 3H provides examples of this.

As shown in Figure 3G, instead of simply listing the number of emergency road transports, the Western Australian Department of Health uses the measure ‘Cost per trip for road-based ambulance services’ to measure the cost-efficiency of the service. 

FIGURE 3G: Extract from the Western Australian Department of Health’s 2019–20 Annual Report

Source: Western Australian Department of Health’s 2019–20 Annual Report.

Figure 3H shows examples of how departments could convert their existing quantity measures into efficiency measures by calculating the unit cost of their services. 

FIGURE 3H: Examples of how to convert quantity measures into efficiency measures

Source: VAGO, based on the 2020–21 BP3.

Departments can similarly convert existing timeliness measures into efficiency measures to provide more meaningful performance information. For example, DJPR's ‘Resources’ output has the output performance measure 'Regulatory audits completed within agreed timelines'. This output performance measure could be improved by measuring the 'average time to complete a regulatory audit'. This would allow DJPR to assess its timeliness in delivering this output and if its service delivery has improved over time.

Effectiveness output measures

Under the Framework, effectiveness is measured mostly through objective indicators because they show the outcome of an activity, and therefore whether it is effective or not. Output measures can contribute to understanding the reasons behind effectiveness.

Departments frequently measure the 'quantity' of their service delivery to do this. However, departmental quantity measures are usually only a simple count of services delivered. A more useful approach, for example, would be to measure the number of services as a proportion of the target population. This would reveal more information about the effectiveness of the reach or uptake of an intervention. This is demonstrated in Figure 3I.

FIGURE 3I: More useful effectiveness output performance measures

Source: VAGO, based on the 2020–21 BP3.

Capturing all 'major' activities in output measures

Departments do not always apply the principle of focusing on 'major' activities, and have inconsistent approaches to deciding how many output performance measures to set for each output. This issue is seen in examples where significant, costly services with large community impact have the same number of output performance measures as much lower cost services with far smaller impact. While it is important for departments to collect performance information about all of their services, if information does not reflect a major service, then it is better suited to department level reporting because it dilutes BP3's focus on significant matters. 

Figure 3J shows that DPC, which has a relatively small budget and provides little direct service outputs to the community, has a similar number of output measures to DET, which provides all government early childhood, school, and tertiary and higher education services. 

To further illustrate the very different approaches to determining the number of output measures, DPC has eight output measures for its 'Chief Parliamentary Counsel services' output, which is worth $6.6 million, and seven measures for its 'Support to veterans in Victoria' output, which is worth $9.0 million. In contrast, DET has four measures for its ‘Support for Students with Disabilities’ output, which is worth $1 242.6 million.

3.3 Constructing output performance measures

Departments need to construct output performance measures that measure the desired objective of their service delivery and relate to factors that are clearly within their control. Good output measures should provide useful information to help stakeholders understand how a department's services might be contributing to objective indicator results. However, we found numerous examples of output performance measures that do not provide meaningful information about output performance. This is because departments have output performance measures that: 

• do not measure their outputs
• do not clearly define what is being measured
• do not relate to factors within their control
• only relate to meeting legislative requirements or a basic minimum performance standard
• prevent them from comparing their performance over time.

Output performance measures that do not measure outputs

Given that departments are funded on the basis of their outputs, it is important that their performance measures clearly relate to these outputs. However, all departments' performance statements include output performance measures that measure an outcome, input or process, rather than an output. These measures do not meet the Framework's requirement to measure output performance, which is the key accountability mechanism of the state's funding model. 

Figure 3K shows five examples of output performance measures and outlines if they meet the Framework's requirement to measure outputs. For reference, Section 1.1 defines the terms input, process, output and outcome.

FIGURE 3K: Examples of 2020–21 output performance measures and whether they are input, process, output or outcome measures

Source: VAGO, based on the 2020–21 BP3. 

It is likely that departments include input and process measures in their performance statements because they provide departmental staff with useful management information. However, departments should capture and report this outside of BP3. 

Departments wrongly including outcome measures in their performance statements as 'output' measures suggests the need for them to more carefully consider the service logic of the activity being provided and ensure outcome measures are properly expressed as objective indicators, as discussed in Chapter 2. 

When departments wrongly include input, process and outcomes measures, this can exclude relevant output measures, which results in reporting gaps. This impairs the function of the state's funding model, which purchases outputs and therefore requires departments to report on their output delivery in return for that funding. 

For example, DET has included a number of outcome measures within its output measures, for example, measures of student literacy and numeracy. This becomes problematic if the activities DET provides (the outputs) to support these outcomes are not included in the performance framework. 

There are a range of funded DET activities outlined in the 2020–21 BP3 that would contribute to the achievement of literacy and numeracy levels, but these are not reflected in DET's output measures. Therefore, DET may not have performance information on the volume, timeliness, cost or quality of the outputs it was funded to deliver to support student achievement. This makes it difficult for decision makers to scrutinise why the outcome results might have occurred or ensure DET has delivered its funded outputs as intended. 

Another example that demonstrates this issue is DTF's output measures for Invest Victoria. It only has one true output measure, which counts the number of visits to the Invest Victoria website. Aside from this, one input measure is included ('total cost') and the rest are all outcome measures that outline the number of jobs created, businesses attracted to Victoria and funds generated. The results of these measures may also be strongly influenced by factors outside of DTF's control. This means there is no reporting on the actual services delivered by Invest Victoria in return for government funding, as shown in Figure 3L.

Vague output measures 

For performance measures to effectively communicate information about departments' performance, they must clearly state what they measure. The Framework’s guidance states that better-practice output performance measures are clear, concise, and use non-technical language so they can be easily understood by Parliament and the community. 

In many cases, departments’ output performance measures are clear enough for parliamentarians and the public to understand. However, we identified examples that may confuse readers with limited knowledge of a particular service area or how departments operate.

Many of these examples may be understood by departmental staff in the context of internal reporting. However, they are likely to be difficult for the public and parliamentarians to understand because they do not have access to internal departmental business rules that further explain the measure. This limits the transparency of public performance reporting.
 

Output measures that the department cannot control 

The Framework states that good measures should be ‘directly attributable to programs and/or activities delivered by the organisation under the output’. Where services are driven by external demand, such as hospital, transport or court services, the level of demand is not within the department’s control. For this reason, output measures that simply 'count' the demand are not useful to assess departmental performance. 

There are a large number of measures in the 2020–21 BP3, particularly for DHHS, that reflect levels of external demand rather than departmental actions. In all cases, such measures can be converted to measures that do show departmental performance by expressing performance as a productivity rate, or by creating a cost-efficiency measure. For example:

Valueless output measures and targets

Targets make performance information easier to understand because they provide context about what departments are trying to achieve.

The Framework states that targets 'stipulate the Government-agreed standard of service delivery for that year'. As such, it is important that a target appropriately reflects the desired standard for that output so the user of the performance information can understand whether departmental performance does or does not meet expectations.

However, we found examples where targets for output performance measures do not achieve this due to:

• the measure and target only requiring compliance with a minimum standard
• it being impossible to know whether achieving above or below the target is good or bad. 

Output measures and targets that only show compliance with a minimum standard

The Framework states that output performance measures that measure compliance with legislated standards should be used sparingly because they usually reflect a basic minimum standard rather than the desired quality of the service.

The Framework also states that departments should not set targets of 0 or 100 per cent because they cannot demonstrate if their performance has improved from year to year. 

However, in the 2020–21 BP3 there are 99 output performance measures across the eight departments that:

• have targets of 100 per cent
• only reflect minimum levels of performance. 

This accounts for around 7.9 per cent of all output performance measures. While all departments have some targets of 100 per cent, they are particularly common in DPC and DoT, with 23 and 22 respectively.

This use exceeds 'sparingly'. Figure 3M includes some examples of output performance measures that reflect meeting minimum standards and have targets of 100 per cent.

FIGURE 3M: Examples of output performance measures that reflect meeting minimum standards and have targets of 100 per cent

Source: VAGO, based on the 2020–21 BP3. 

Use of neutral measures

Neutral measures are ones where meeting or not meeting the target does not provide meaningful information about a department's performance. These targets commonly appear in DHHS and DJCS’s output performance measures. 

For example, DHHS’s output performance measure ‘Reports to Child Protection Services about the wellbeing and safety of children’ is not clear about what the department is aiming to achieve. The target for 2020–21 is 136 677 reports. A result below the target may mean that preventative services to support child safety are working as intended. On the other hand, a result above the target may mean that there are higher levels of reporting on the wellbeing and safety of children, which could also be a positive result. A similar measure with the same issue exists for counting family violence crimes. 

Measures that prevent comparison of performance over time 

The Framework requires that output measures 'enable meaningful comparison and benchmarking over time'. This requirement allows departments and government to track performance and assess the impact of changing investment decisions.

To be comparable over time, an output measure must account for variations in factors, such as population size and the number of service users. Measures that have percentages and rates help account for these factors, but raw numbers do not. For example, DTF's output performance measure 'Compliance and enforcement activities—energy' and DET's output performance measure 'Number of Digital Assessment Library items developed' are both measured in raw numbers and do not account for variations in population, service users and funding amounts. This prevents users of the information from meaningfully comparing results over time to identify performance changes. 

We assessed a selection of output performance measures to see if they support comparison of results over time. This selection covered the following output groups: 

• ‘Primary and Secondary Education’ (DET)
• ‘Mental Health Services’ (DHHS)
• ‘Budget and Financial Advice, Revenue Management and Administrative Services to Government, Economic and Policy Advice and Economic Regulatory Services’ (DTF).

As shown in Figure 3N, 42 per cent of the reviewed output performance measures do not enable comparison of performance over time. 

FIGURE 3N: Number of output performance measures that enable comparison over time

Source: VAGO, based on information from DTF.

Figure 3O gives more detailed examples to illustrate this issue. 

FIGURE 3O: Examples of output performance measures that enable and do not enable comparison over time

Source: VAGO, based on the 2020–21 BP3.

Where output measures prevent comparison over time, they also prevent comparison against other jurisdictions, which the Framework states is a preferable feature. Output measures that are expressed as percentages or rates, which therefore control for variables such as population levels, provide departments the opportunity to benchmark performance against other states and territories, which is useful for identifying performance gaps and issues. 

Discontinuing output performance measures

Another factor that may prevent departments from assessing output measure performance over time is when measures are discontinued or significantly changed. For this reason, the Framework states that it is important to minimise the number of changed measures from one year to the next. However, the Framework also acknowledges that this needs to be balanced against the need for new output performance measures as government policies and programs evolve. 

Each state Budget sees a number of measures discontinued and a number of new measures added. Figure 3P shows that of the 1 258 output performance measures in the 2020–21 BP3, 468 (37 per cent) have existed for 10 or more years. 

Since the 2011–12 state Budget, PAEC, at the invitation of the Assistant Treasurer, has had the opportunity to comment on the measures that have been proposed for discontinuation. 

In the 2019–20 BP3, 102 measures were proposed for discontinuation. PAEC’s review of these measures found that: 

• 39 per cent of them have been replaced by improved measures
• around 25 per cent relate to projects or programs that were completed or discontinued
• the department did not provide a clear reason for discontinuing the measure in 14 per cent of cases. 

PAEC recommended that DTF, in consultation with all departments, ensures that future BP3s contain clear explanations for all proposed discontinued measures to enable meaningful review by PAEC.

In PAEC's review of the 2020–21 BP3, it identified only two measures where departments did not provide a clear reason for discontinuing the measure.

Back to top

Conclusion

It is difficult for the government, Parliament and the community to use the results departments publish in BP3 and their annual reports to understand performance. This is due to: 

• frequent gaps in data sources and calculation method documentation
• a lack of performance reporting against objective indicators
• a failure to present trended performance results over time
• limited explanations of variances from targets.

Together, these issues reflect the lack of priority that departments give to transparently and accountably demonstrate their performance results. This is inconsistent with the purpose of the Framework as 'a governance and operational framework for public sector accountability for the investment of public sector resources'.

4.1 Reporting accurate results

Performance reporting in BP3 and departments' annual reports is key in demonstrating accountability for public sector service delivery. In both cases, it is vital that departments report accurate results against objective indicators and output measures.

As shown in Figure 4A, several of our past audits have identified issues with the accuracy of externally reported performance data. A common issue is weak or absent data controls, which can lead to inaccurate and/or incomplete reporting.

FIGURE 4A: Issues with the accuracy of performance data found in past audits

Source: VAGO.

To support accurate and consistent data capture and result calculation, the Framework requires departments to document their methodology for recording, calculating and reporting their performance results and make this available for DTF to review on request.

While the Framework only requires this for output performance measures, we also assessed if departments have data definitions and documented business rules for their objective indicators. This is because departments need to have clear internal rules and processes to ensure their performance statements contain meaningful, accurate information. 

However, as shown in Figure 4B, we found numerous gaps in the information required to clearly document how objective indicator and output measure results are calculated. For example:

• DPC does not have a data dictionary, or any other documentation, that outlines how it calculates its departmental objective indicator and output performance results. As such, it is difficult to ensure DPC calculates its results accurately and consistently each year. 
• DET only has high-level, general descriptions of its measures with no supporting technical information.

FIGURE 4B: The completeness of departments’ calculation documentation to support their 2019–20 objective indicator and output performance measure results

✓ Met. ✗ Not met.
Note: Output performance measures include quantity, quality, timeliness and cost. Measure description details what activity is being measured, defines key terms and explains what is being reported. Data collection outlines what data is being collected, how the data is collected, the frequency of data collection and data security arrangements. Business rules defines what the measure counts and outlines any assumptions relevant to how the data is captured. Inclusions and exclusions identify any key quantitative or qualitative data, categories, groups or activities that are specifically included or excluded. Method defines how the result is calculated. Data validation outlines the process for validating/assuring the quality of the raw data and/or calculated result, for example, whether the result is verified internally by a business unit, endorsed by the deputy secretary, or by an internal or external audit. Target setting details how the target is set.
Source: VAGO, based on information provided by departments.

Despite departments with data dictionaries having relevant sections populated, we found examples where the information was not clear enough or did not provide sufficient detail on how a measure is calculated. For example:

If data dictionaries do not include all of the key information, departments are highly reliant on the knowledge and experience of key staff to ensure their performance data is prepared consistently and accurately year on year. If these key staff leave the department, there is a risk that this knowledge will be lost and that future data reporting could be incorrectly captured or interpreted.

In addition, we found that DTF does not request information on departments’ business rules and does not review departments’ data dictionaries. While the Framework does not require DTF to conduct reviews, by not reviewing or ‘spot checking’ departments’ data DTF is missing the opportunity to assure itself that departments’ processes are supporting accurate performance statements. 

Controls over performance reporting

Departments need systems and procedures to ensure the accuracy and completeness of their performance information. These can include:

• clearly defined and documented business rules
• training staff to follow data collection processes
• quality assurance checks on how data has been collected and how results have been calculated
• reviews by someone external to the business area that collected the data, such as an internal audit team. 

We requested evidence from DET, DHHS and DTF about how they collect, store, calculate and report on a selection of performance measures. We used this data to recalculate some of their reported results. We found that despite there being gaps in their business rules for fully documenting the selected measures, the three departments do utilise controls to support data accuracy and we were able to accurately recalculate their published results.

Controls in place

Figure 4C sets out the systems for collecting and storing data and the internal controls to ensure data accuracy used by the three departments for the selected measures. 

FIGURE 4C: Performance information systems and internal controls at DET, DHHS and DTF

Source: VAGO, based on information provided by departments.

The three departments we examined have systems to ensure that their reported data results are reviewed and signed off by senior management prior to publication. All three departments also have central units that ‘sense check’ results by comparing them to previous years and considering any major events or incidents that may have impacted the results. 

DHHS also has data input validation processes built into its mental health information systems to ensure mandatory data fields are completed. 

DJCS employs a better-practice approach. Its central unit tests the accuracy and completeness of data submitted by its business units on a risk basis. DJCS’s central unit does this by recalculating the performance result using the business rules and methodology set out in the data dictionary.

Across all departments, it is common practice for the business unit responsible for performance against a measure to set the measure and associated targets. They are also usually responsible for:

• collecting data to assess their progress against the measure
• determining how to calculate results
• preparing public reporting on the results.

The creates a risk that if departments do not have a separate business unit checking results, then they are not managing the conflict of interest that exists by having the same areas set, collect and report on their own measures. 

Accuracy of output measure results 

To test the accuracy of information reported in departments' 2019–20 annual reports, we recalculated the results for the following performance measures, as shown in Figure 4D.

FIGURE 4D: Output measure results that we recalculated

Source: VAGO, based on the 2020–21 BP3.

We did not identify any calculation errors. However, some of DTF's business rules did not provide clear enough guidance on how it calculates its results. For example, the output performance measure 'Delivery of major milestones within agreed timelines' does not provide any details of the rating system for determining if major milestones were delivered within agreed timelines. DTF uses a traffic light rating system, but does not specify the criteria for determining what sits within each category.

With DET's 'Average days lost due to absence at Year 5, 6, 7–10, 11, 12', measured schools and health services are permitted to retrospectively submit data. As a result, there is a risk that the reported result may change over time. However, we only found minor discrepancies when we redid the calculation.

We were not able to recalculate the results for DET’s measures that rely on NAPLAN data, as this information is collected, calculated and reported by an external agency. 

4.2 Reporting on objective achievement 

Departments are required to publicly report on their performance in two places:

• The BP3 outlines the products and services that the government funds. As the state Budget is usually released before the end of the financial year, each department reports actual results for around 9 months and estimates performance for the remaining months.
• Each department’s annual report provides information on actual performance for the full financial year, including whether the department has achieved its objectives. 

However, the performance information that departments publish does not clearly demonstrate their progress towards achieving their stated objectives. As outlined already, in many cases this is because departments lack true measures of their objectives. In addition to this issue, no departments have established baseline data for their objective indicators to measure their performance against. 

Reporting on progress over time

It is a mandatory requirement in the Framework for departments to report their performance against their departmental objective indicators in line with DTF's Model Report. The Model Report requires departments to report multiple years of results to show performance over time, which enables the reader to make basic comparisons between past and present performance.

In 2019–20, only five of the eight departments complied with this requirement. We identified a range of gaps in the ways that DHHS, DPC and DTF use their annual reports to report on their progress over time. 

In DTF's 2019–20 annual report, it reported performance over four years for seven objective indicators. For the remaining six objective indicators, DTF only provided narrative descriptions of performance.

In 2019–20, DHHS and DPC reported four years of results, but for ‘lower level’ indicators rather than their objective indicators. Some departments use lower level indicators as a tool for tracking progress against an overarching objective indicator. However, this approach does not replace the Framework's requirement that departments report against their objective indicators. 

Figure 4E shows the objective indicators DHHS set in the 2019–20 BP3 for the departmental objective ‘Victorians have the capabilities to participate’.

FIGURE 4E: Extract from DHHS’s performance statement in the 2019–20 BP3

Source: 2019–20 BP3.

However, as Figure 4F shows, the 'indicator results' DHHS reported in its annual report are entirely different to the objective indicators in BP3. They do not relate to the same service areas, which include vulnerable groups, such as children in child protection, carers and people with disability. While the lower level indicators do provide useful information about aspects of DHHS's performance against the objective, DHHS has not complied with the Framework because it has not provided a transparent record of the department’s achievement against its departmental objective. 

Reporting actions rather than results 

In its 2019–20 reporting, DTF described actions it had completed rather than the results it had achieved for five of its 13 objective indicators. For example, DTF provided commentary on the work it carried out during the year instead of measuring if the objective indicator was achieved. This is shown in Figure 4G. For another objective indicator, ‘High quality whole of government common services provided to Government agencies, as assessed by feedback from key clients’, DTF only provided results for one year. 

FIGURE 4G: Extract from DTF’s 2019–20 Annual Report

Source: DTF’s 2019–20 Annual Report.

4.3 Reporting on output performance

Departments do not publicly report on their output performance in a way that allows the reader to compare results between departments or understand performance over time. This limits Parliament and the community’s ability to hold departments accountable for their performance. 

Departments' performance statements in BP3 are available online. However, BP3 does not provide parliamentarians or the public with trended data over multiple years, which is the most practical way to understand departments’ performance over time.

Parliamentarians and the community can access all departments’ current and prior year performance results through Microsoft Excel spreadsheets that DTF publishes on its website. However, it is difficult for readers to interpret this data without having detailed knowledge of departments' work, and users must create graphs to visualise the raw data themselves. 

Given the limitations of departments' public reporting, we developed a dashboard using data from DTF’s website and the departments' 2019–20 annual reports. We have also included data published in the 2021–22 state Budget papers to update our dashboard to include 2020–21 performance results. This dashboard, available at our website (www.audit.vic.gov.au), can be used to analyse departments' output performance measure results from 2008–09. 

Figure 4H shows that for 2019–20, departments reported meeting a combined total of 57 per cent of their output performance measure targets, and not meeting 37 per cent. The remaining 6 per cent are neutral measures, where it is not possible to determine if a target has been met or not. 

Explaining variance in performance

Departments do not always comply with the Framework’s requirement to explain significant performance variations against the targets in their performance statements. Departments' explanations are critical to the usefulness of output performance measures as a way to monitor and assess their performance. They also support a culture of transparency by requiring departments to justify their spending during the yearly revenue certification claim process. 

However, we found examples where departments with significant performance variances have not provided clear explanations. Some have simply stated that there is a variance, or that a variance is positive because it exceeded the target. These insufficient explanations make it difficult for Parliament and the public to understand whether variations in performance should or should not be of concern and whether the result is due to factors within or outside of a department’s control. 

In its yearly reports on the Budget estimates, PAEC has repeatedly identified weaknesses in departments’ explanations of performance variations, including:

• unclear and incomplete explanations
• failure to identify the underlying cause of variances 
• failure to provide more information than just a statement that there was a variance
• too many speculative explanations that are not based on clear evidence. 

We used our dashboard to identify significant variations in departments' performance. Figure 4I shows that almost half of all output performance measures varied from their target by more than 5 per cent in 2019–20 (592 output performance measures out of a total 1 252).

At 10 instances, DHHS had the most significant number of variances with no explanation given in BP3. DTF had three variances with missing explanations, and DoT and DJCS each had one. While the remaining departments' output performance measures included explanations for variances, these vary in quality as shown in Figure 4J.

FIGURE 4J: Examples of how departments explain variances 

Source: VAGO, based on the 2020–21 BP3.

4.4 Auditing departments' performance results

Unlike departments' financial statements, which we independently audit, there is no legislated requirement for state government departments’ performance statements to be independently audited. In contrast, local government, water authorities and TAFE entities in Victoria are required to have their performance statements independently audited. We undertake this work as a part of our annual financial audit work program. It involves testing if the Local Government Performance Reporting Framework indicators included in councils' annual reports accurately report performance. Where necessary, we consider processes that councils use to ensure they report performance information accurately. 

The present scenario in Victoria means that while Parliament and the public have independent assurance of the accuracy of government agencies' financial statements, this is not available for performance statements, which demonstrate the delivery of public services to the community. 

To address this issue and increase public confidence about reported performance information, some jurisdictions require public entities to have their service delivery performance reporting independently audited. Figure 4K provides examples of this. 

FIGURE 4K: Examples of jurisdictions that audit non-financial performance statements

Source: VAGO, based on information from the Queensland Audit Office’s Monitoring and reporting performance, and the New Zealand Accounting Standards Board's Public Benefit Entity Financial Reporting Standard 48 Service Performance Reporting.

Back to top

We have consulted DELWP, DET, DFFH, DH, DJCS, DJPR, DoT, DPC and DTF, and we considered their views when reaching our audit conclusions. As required by the Audit Act 1994, we gave a draft copy of this report, or relevant extracts, to those agencies and asked for their submissions and comments. 

Responsibility for the accuracy, fairness and balance of those comments rests solely with the agency head.

Response provided by the Secretary, DELWP

 

Response provided by the Secretary, DET

Response provided by the Secretary, DFFH

Response provided by the Secretary, DH

Response provided by the Secretary, DJCS

Response provided by the Associate Secretary, DJPR

Response provided by the Secretary, DoT

Response provided by the Secretary, DPC

Response provided by the Secretary, DTF

 

 

Back to top

 

Back to top

Note: In February 2021, DHHS was separated into two new departments: DH and DFFH. Given the period of focus for this audit, this report refers to DHHS. Any audit findings in this report that relate to DHHS will apply to the two new departments. 

Our methods

Methods for this audit included:

• desktop research identifying better practice in performance measurement and reporting

• assessing departments' compliance with legislation and guidance including the FMA, the Standing Directions, the Framework and the Model Report
• identifying, collecting and reviewing relevant documents
• interviewing relevant staff 
• examining departments’ performance statements in BP3s and annual reports. 

We conducted our audit in accordance with the Audit Act 1994 and ASAE 3500 Performance Engagements. We complied with the independence and other relevant ethical requirements related to assurance engagements.

Back to top

As discussed in Section 3.2, most departments' performance statements do not clearly measure their service efficiency and effectiveness. This makes it difficult for them to identify opportunities to improve their operations and demonstrate value for money. We used the Productivity Commission's RoGS to show how departments could restructure their performance information to better monitor their performance over time. 

RoGS uses a service logic model, which we outline in Section 1.1, to compare the efficiency, effectiveness and equity of government services across jurisdictions. RoGS clearly defines the inputs (funding and resources) that departments use to deliver outputs (services) and achieve an outcome.

Figure D1 shows the RoGS performance reporting framework for mental health services. It distinguishes outputs from outcomes and defines performance measures for equity, effectiveness and efficiency. 

Figure D2 compares this framework to DHHS's BP3 output performance measures for its mental health output group. It shows that DHHS does not provide all of the necessary information to assess the equity, effectiveness and efficiency of its services.

The grey boxes in Figure D2 identify the gaps in DHHS's performance statement, which include:

• a lack of measures to monitor the effectiveness of services for children and young people and the inclusion of consumers and carers in decision-making
• a lack of equity measures to show whether services are accessible for a range of community groups.

While DHHS does list the total output cost for its mental health services, which was $1.7 billion in 2019–20, it does not provide unit costing for different types of mental health services, such as hospital and community-based services. These gaps make it difficult for the department to show if it is improving mental health services over time and in comparison, to other jurisdictions.

Back to top