Follow Up of Selected 2014–15 Performance Audits

What these audits examined and how

The objective of these audits was to determine whether accepted audit recommendations have been effectively addressed.

We considered whether:

• performance issues relating to the recommendations have been addressed
• timely action has been taken to address the recommendations
• plans are in place to address incomplete recommendations
• actions are monitored and reviewed for impact.

As part of the audits, we verified self-attestations, reviewed documents, interviewed staff, and tested compliance with policy and process improvements.

We carried out these audits in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

The cost of these follow up audits was $350 000.

Report structure

Each part of this report covers one audit and includes conclusions specific to that particular audit. The remainder of the report is structured as follows:

• Part 1—Additional School Costs for Families
• Part 2—Effectiveness of Support for Local Government
• Part 3—Operational Effectiveness of the myki Ticketing System.

Responses

We have consulted with the Department of Education and Training, the Department of Environment, Land, Water and Planning, the Municipal Association of Victoria, Public Transport Victoria and the Department of Treasury and Finance and we considered their views when reaching our audit conclusions. As required by section 16(3) of the Audit Act 1994, we gave a draft copy of this report to those agencies and asked for their submissions and comments. We also provided a copy to the Department of Premier and Cabinet.

The following is a summary of those responses. The full responses are included in Appendix A.

Follow up of Additional School Costs for Families—the Department of Education and Training agrees with the contents of the report.

Follow up of Effectiveness of Support for Local Government—the Department of Environment, Land, Water and Planning notes the audit confirms Local Government Victoria has responded to all the recommendations from our 2015 audit. The Municipal Association of Victoria (MAV) is comfortable that the report is accurate in its references to MAV, and reflects the actions MAV has undertaken to implement the recommendations from our 2015 audit.

Follow up of Operational Effectiveness of the myki Ticketing System—Public Transport Victoria accepts our recommendations and has developed an action plan to address them.

1.1 Context

1.1.1 Free instruction

The Education and Training Reform Act 2006 (the Act) provides for free instruction for Victorian children under the age of 20 in the standard curriculum. Free instruction is a longstanding principle that was first established in Victoria in the Education Act 1872.

The Act does not define 'free instruction'. In its parent payment policy, the Department of Education and Training (DET) defines free instruction as 'the teaching staff, administration and the provision of facilities in connection with the instruction of the standard curriculum program, including reasonable adjustments for students with disabilities'.

1.1.2 Student Resource Package

Free instruction is funded by the Commonwealth and state governments, and delivered to schools through DET's Student Resource Package (SRP). The SRP funding model was introduced in 2005, and shifted the focus from funding outputs to funding educational outcomes.

SRP funding is provided to schools under one of three categories:

• student-based funding—provided as a total per student and includes equity loading, if applicable
• school-based funding—for school infrastructure
• targeted initiatives—such as targeted welfare and student support programs.

1.1.3 Parent payments

The Act provides for free instruction but also allows for school councils to request voluntary financial contributions from parents and to charge fees to parents for goods, services or other things provided by the school to students, as outlined in DET's parent payment policy. This means that parents who send their children to a government school in Victoria will likely be asked to contribute a combination of essential, optional and voluntary payments to support free instruction. The three types of additional costs parents may be asked to pay are outlined in Figure 1A.

Figure 1A

Parent payment categories

Essential learning items These are the items, activities and/or services that a school deems essential to student learning in the standard curriculum. Parents may choose to supply their own items or purchase them through the school. Examples of essential items include items that the student takes permanent or temporary possession of (text books, stationery), activities that all students are expected to attend (excursions) and school uniforms.

Optional items These are the items or services that are optional and offered in addition to the standard curriculum. These can be accessed on a 'user pays' basis and include extracurricular programs such as music tuition or optional camps and excursions.

Voluntary financial contributions Parents can be invited to make a voluntary contribution to the school for a specific purpose, such as equipment in addition to the standard curriculum or a building or library fund.

Source: VAGO based on DET's parent payment policy.

1.1.4 Roles and responsibilities

DET leads the delivery of education and development services to children, young people and adults directly through government schools and indirectly through the regulation and funding of early childhood services, non-government schools and training programs.

DET provides funding for public schools through the SRP and is responsible for deciding how it will be distributed. DET is also responsible for developing the parent payment policy and overseeing each school's implementation of the policy.

1.1.5 Our previous audit

Our 2015 audit Additional School Costs for Families examined whether DET and government schools were achieving value for money in the management and application of parent payments.

We found:

• there was no shared understanding of the different types of parent payments and free instruction, which resulted in parents being charged at some schools for items that should be provided without charge
• DET did not know how much it costs to provide free instruction for each student, so DET could not give assurance that school funding is adequate
• DET's oversight of schools was limited—it did not have oversight of schools' compliance with the parent payment policy and DET's performance measures at schools did not adequately measure efficiency and economy.

We made seven recommendations to DET, all of which they accepted and agreed to act on by December 2016. Following a departmental restructure, DET expected to complete recommendation four by June 2017 (see Figure 1D).

1.2 Conclusion

DET has taken action to respond to our recommendations, although some actions are still ongoing. DET needs to do further work to address the underlying issues identified in our 2015 audit.

The government has commissioned a number of reviews into school funding and costs. Although DET has responded to our first recommendation by increasing the sample size of schools used to analyse data, the SRP has not changed. DET has recently established a school funding reform team to drive and coordinate school funding reform policy.

Defining what an efficient and economic school should look like remains a challenge. DET's oversight of efficiency and economy at schools has not changed since our 2015 audit and it remains limited. To address this, DET is working to review efficiency and economy performance measures for schools.

Since our 2015 audit, schools better understand DET's new parent payment policy and they are applying the parent payment categories more consistently. However, risks remain due to DET's limited monitoring and oversight of schools.

We tested a sample of 10 schools' policies and found that not all of their policies complied with DET's requirements. This highlights the need for DET to be proactive in its oversight and monitoring of parent payments and in its support to schools.

DET is conducting an internal audit of schools to test schools' compliance with the policy.

1.3 School funding and costs

At the time of our 2015 audit, DET calculated school funding based on data collected in a rolling benchmark review. The last review occurred in 2012 and was based on a limited sample size—103 out of a then total of 1 537 schools. We found that DET did not understand the actual cost of providing free instruction and therefore could not provide assurance that funding for schools was adequate.

In 2015, we consulted with principals and advocacy groups who said that the SRP funding was inadequate and did not reflect the real cost of delivering free instruction. Principals said that attempts to advise DET of this had had no impact.

Figure 1B shows our 2015 audit recommendations to address this and the actions DET has taken in response to the recommendations.

Figure 1B

Recommendations and action taken

Recommendation		Status
1	That DET improves the basis for estimating the funding required to meet efficient school costs, including examining the factors that influence costs and using statistically valid sampling methods.	In progress
	DET advised that government priorities resulted in two studies of school funding and cost rather than commissioning another benchmark rolling review. To address our recommendation, DET increased the sample size in the study. DET has provided additional equity funding to schools but made no change to the SRP in response to the reviews. DET is currently establishing a school funding reform team, which includes the responsibility to drive and coordinate school funding reform policy.
3	That DET regularly and comprehensively consults school principals and school council members to better understand school funding requirements.	Ongoing(a)
	DET liaises with schools through the Business Managers Group, the Expert Principal Advisory Group, the SRP Consultative Committee and the four school regions. Topics for discussion have included the SRP, CASES21(b), financial audits and controls testing, equity funding, finance training for schools, school funding reform and the parent payment policy review.

(a) Since DET continues to consult with schools and stakeholders, we have assessed the status as 'ongoing' due to the continuous nature of consultation.

(b) CASES21 is the computer system provided to government schools to support student administration, financial management and reporting.

Source: VAGO.

1.3.1 Student Resource Package

Performance improvements

Since our 2015 audit, the government commissioned the Bracks Review into Government School Funding, released in April 2016. A joint project advisory group—comprising representatives from DET, the Department of Premier and Cabinet and the Department of Treasury and Finance—is undertaking work on the cost of school funding (the cost of schooling review). School funding is also currently subject to state and Commonwealth negotiations.

Residual risks

DET advises that it did not commission another benchmark rolling review due to these government priorities. However, in order to address our recommendation, DET increased the sample of schools used to undertake the cost of schooling review. This review used the National Assessment Program—Literacy and Numeracy (NAPLAN) as a measure of effectiveness for schools. There were four measures of effectiveness, with a reference group of schools created for each. The size of the four reference groups ranged from 259 to 471 schools. This is an improvement on the sample size of 103 schools used in the 2012 benchmark rolling review.

A reference group created by using measures of effectiveness is not a representative sample of all types of schools. Therefore, DET needs to be confident that NAPLAN is a reliable indicator of schools' effectiveness and the criteria for forming the reference group are sound.

The cost of schooling review examined the net recurrent income per student, which is calculated differently to the SRP and is not directly comparable. This means that, without further work, this review has limited ability to impact the SRP. Furthermore, the review did not consider what it actually costs schools to provide free instruction. Schooling costs were considered as part of the joint project advisory group analysis of expenditure on salaries for principals and teachers, which make up, on average, 85 per cent of school operating expenditure.

These recent funding and cost reviews have not yet led to a change in how the SRP is calculated, although since 2016 schools have received additional equity funding.

DET has advised that the recent reviews are intended to inform its ongoing work in exploring needs-based funding options and models. DET has established a school funding review team to drive and coordinate school funding reform policy.

1.3.2 Consulting schools to better understand costs

DET has continued to discuss school funding at three stakeholder committees to better understand school funding requirements. These are the Business Managers Group, the Expert Principal Advisory Group and the SRP Consultative Committee.

Performance improvements

The committees' meeting minutes show regular discussion about issues affecting school finances, such as the SRP, CASES21, financial audits and controls testing, equity funding, finance training for schools, school funding reform and the review of the parent payment policy. Schools and stakeholders participating in these committees can raise concerns and provide feedback to DET in the meetings.

The four regions also consulted their schools and sought feedback on school costs and the SRP immediately after our audit in 2015.

Residual risks

While ongoing consultation occurs in the committees discussed above, there has not been a sustained focus at a regional level. The schools we consulted expressed that they had limited opportunities to provide feedback to the DET regions. Meeting minutes, as well as our discussions with DET and stakeholders, indicate that consultation through DET's regions and discussion on school costs has occurred when DET has needed to discuss specific issues, rather than in an ongoing way. It is not clear whether DET's regional consultation has led to DET having a better understanding of school funding requirements.

1.4 Parent payment policy and practice

Our 2015 audit concluded that DET's parent payment policy was vague in parts and not sufficiently prescriptive. We also found that DET's definition of 'free instruction' was unclear and similar to the definition of 'essential' learning items.

In 2015, we surveyed 366 schools and assessed their parent payment policies. We found that:

• none of the schools had a policy that completely complied with DET's policy
• 116 of the schools did not have a parent payment policy
• parent payment requests did not align with the schools' or DET's policies
• the three parent payment categories—essential, optional, voluntary—were being applied inconsistently by schools.

When schools receive payment from parents, they record the payments as revenue in CASES21. This financial software system is provided to all schools to record revenue and expenditure, and it uses account codes that DET lists in its chart of accounts.

At the time of the 2015 audit, DET did not structure its chart of accounts to allow schools to record revenue from parent payments against the three parent payment categories. The account codes were confusing and some of the codes involved a combination of essential and optional items.

Our 2015 audit concluded that DET did not collect adequate information on how much schools ask parents to pay. The schools also did not know how much parents paid because CASES21 did not enable the recording of payments against the three parent payment categories.

Figure 1C shows our 2015 audit recommendations to address this and the actions DET has taken in response to the recommendations.

Figure 1C

Recommendations and actions taken

Recommendation		Status
6	That DET updates its parent payment policy and guidance material to provide clear guidance on acceptable parent payment practices.	Complete
	DET reviewed and updated its parent payment policy. The new policy refined the definition of free instruction and essential educational items. It includes tightened requirements for hardship and promotes more transparent communication about schools' parent payment policies. DET also created a policy template for schools that includes all key requirements under DET's policy.
7	That DET regularly reviews school parent payment policies and practices, and intervenes where those practices are identified as breaching legislation or policy requirements.	Ongoing
	An internal audit on locally raised funds is underway. As part of this, DET will assess whether 30 schools are complying with the parent payment policy.
2	That DET enhances the capabilities of CASES21 and requires schools to collect and report the data needed for it to better understand school revenue and costs.	Complete
	DET has altered its chart of accounts. Revenue from parent payments is now recorded according to the three parent payment categories—essential, optional and voluntary.

Source: VAGO.

1.4.1 Updated parent payment policy

Performance improvements

In response to our audit, DET reviewed its parent payment policy, and released a new policy in July 2016. DET's new policy addresses our concerns about the similarity of the definitions of essential education items and free instruction by refining the definition of 'free instruction' and removing the term 'instructional supports, materials and resources'.

As well as providing an updated definition of free instruction, DET's new parent payment policy has several new elements, including:

• a set of core principles
• guidelines for responding to hardship
• improved communication so that school parent payment practices are well communicated, clear and transparent.

As part of the new policy, DET introduced a compulsory parent payment policy template for schools to use, which includes all the key requirements of DET's policy. Schools were required to adopt and comply with the policy when requesting payment from parents for the 2017 school year.

We spoke to schools and stakeholder groups about parent payments and policies. Most schools reported that the difference between free instruction and essential educational items was easier to understand. Schools particularly valued DET's 'Understanding the parent payment categories' information sheet.

Our discussion and review of a sample of 10 schools' parent payment policies further demonstrated that the terms 'free instruction' and 'essential educational items' are applied more consistently.

Residual risks

Confusion remains about some elements of parent payments:

• Essential versus optional camps—schools have discretion to decide whether a camp or excursion is essential or optional. Some schools indicated that they make camps and excursions optional rather than essential, as not all parents have the ability to pay for them. Some also reported that camps create additional costs for schools in the form of casual relief teachers (CRT) to replace the teachers on camp. The cost of CRT cannot be passed on to parents if the camp is essential to the school curriculum, although it can be passed on if it is optional.
• Camps, Sports and Excursions Fund (CSEF)—schools have stated that it is unclear which essential items the CSEF funding can be used for, as it can be used for some camps and sports items, but not all.
• Instruction provided outside of the school—for example, external swimming lessons are 'instruction' and are required to be provided free of charge. Schools may have a different interpretation about whether an out-of-school activity is instruction, rather than an excursion or demonstration.

As stated in DET's policy, DET offers extra support to schools through the relevant DET region and a specific advisory email address. DET provided examples of how it has supported schools in reviewing policy and payment requests to help them comply with the policy when schools or parents have asked for help. However, when we spoke to schools, many principals and business managers felt unsure about how to seek help and who to contact.

1.4.2 Monitoring of schools' parent payment policy

DET has not tested compliance with the parent payment policy since its 2011 internal audit, despite requiring schools to comply with the policy. The 2011 internal audit rated compliance with the parent payment policy at a sample of 37 schools. It found that 21.5 per cent were fully compliant with the policy, 51.5 per cent were partially compliant and the remainder were noncompliant.

Performance improvements

DET is conducting an internal audit of locally raised funds in schools, which will be undertaken annually. As part of this audit, DET is testing 30 school policies and practices, with a key focus on assessing compliance with the parent payment policy and increasing DET's oversight of how schools are applying parent payments.

Residual risks

We sourced parent payment policies from a sample of 10 randomly selected schools to test compliance with the parent payment policy. All schools selected had a parent payment policy, which is an improvement from our 2015 audit.

Schools’ policies did not all comply with DET’s policy. Our review of schools' parent payment policies and requests to parents found some common trends:

• half of the schools had adopted DET's new policy template
• some schools adopted portions of the policy template or condensed the policy into a shorter summary
• some policies did not include definitions for essential, optional or voluntary payment categories
• all schools indicated that the policy had been approved by the school council, although not all indicated this on the policy
• in some instances, schools had not customised the sections of the policy requiring customisation
• some costs were bundled rather than itemised, such as stationery costs, and some schools had itemised lists available upon request
• not all schools have published their parent payment policy online—although not a strict requirement, DET encourages schools to publish their policies online to promote open and transparent communication with parents.

We also found that, rather than having a separate hardship policy, schools were including hardship statements in the parent payment policy.

Our testing of school compliance with DET's policy highlights that some schools do not fully understand the requirements for adopting the new policy, or the policy has not been clearly explained to schools. This demonstrates the need for DET to be proactive in its oversight of each school's parent payment policy to better support schools to comply with the requirements.

1.4.3 Chart of accounts

Performance improvements

To help schools better record the parent payment revenue received, DET has updated its chart of accounts to align with the updated parent payment policy.

Schools now record revenue according to the parent payment categories of essential, optional and voluntary payments. Schools and DET can directly trace parent payment requests to the parent payment revenue collected.

DET provided us with the total revenue figures for 2016 per school against the new and deactivated codes. The data confirms that the new revenue account codes are in effect and schools are using them.

Although DET deactivated the old account codes, it is aware that it is possible for schools to charge to an old code. Data from CASES21 demonstrates that 566 schools erroneously recorded revenue against one of the deactivated fee codes in 2016 (class materials and subject contribution codes). DET provided evidence of examples of its assistance to schools to correct the issue. The revenue listed in the wrong codes equates to 0.5 per cent of revenue for parent payments (1 per cent of revenue for essential and optional items).

1.5 Efficiency and economy of schools

DET provides schools with funding for free instruction, but it is the responsibility of schools to use their funding efficiently and effectively. DET measures and monitors schools' performance using the 'school improvement measures'.

Our 2015 audit concluded that DET was using a school performance framework at the time that did not indicate which schools were:

• using their resources economically
• working efficiently
• working effectively with the school community to achieve outcomes.

In 2015, when we asked schools about their efficiency practices, we found they varied widely between schools. Although we concluded that it was appropriate for schools to use different and tailored practices, we also noted there were some effective cost‑saving practices that schools could use to improve their efficiency.

Figure 1D shows our 2015 audit recommendations to address this and the actions DET has taken in response to the recommendations.

Figure 1D

Recommendations and actions taken

Recommendation		Status
4	That DET incorporates comprehensive efficiency and economy measures into its school performance framework to establish oversight, compliance and accountability.	In progress
	DET's response to this recommendation forms part of a broader departmental project to review and improve DET's schools performance measures. Draft measures are developed.
5	That DET provides guidance and training to school councillors, principals and business managers on efficiency and economy better practice.	Ongoing(a)
	DET updated the training modules for principals, business managers and councillors. The new training content includes guidance in applying value for money at the school level. This content has been included in the training modules since 2016.

(a) DET has completed its update of the training materials. This recommendation is assessed as 'ongoing' due to the continuous nature of delivering training.

Source: VAGO.

1.5.1 Performance monitoring

Performance improvements

DET commissioned a review of the current school improvement measures to:

• identify gaps in measurement practices
• improve current measures
• identify additional or replacement measures.

The review has made recommendations to add new measures to the school improvement measures of productivity. The new measures focus on achieving value for money alongside productivity, such as by measuring school expenditure and student outcomes.

DET will need to consider and approve any new or amended measures before they can be implemented, which DET expects to occur in 2018.

1.5.2 Value for money in schools

Performance improvements

To improve efficiency and economy in schools, DET updated its training for school councillors, principals and business managers to include training materials on better practice in efficiency and economy. The training packages explore the concept of value for money and emphasise the need to strategically apply it in schools. School principals are given practical value-for-money examples to implement at the school level, such as combining with other schools to increase their purchasing power.

School councils, including principals, are offered the 'Improving School Governance' training package. The finance module covers efficiency and economy. Attendance records show that in 2016, 623 school councillors from 124 schools participated in the 'Improving School Governance' finance training. The training is also available online for those who cannot attend in person.

Business managers and principals are offered the 'Dollars and Sense' and 'Talking Finances' training. The efficiency and economy training content is included in the financial management module. In 2016, training records show that there were 206 participants in the financial management module, from 162 schools.

Principals and business managers, who are the target audience, attended the training as well as assistant principals, administration managers, teachers, and staff in other support roles. Eighty per cent of participants rated their satisfaction with the financial management module as high or very high. Since 2017, DET has measured whether participants are confident in applying the knowledge they have learnt at their schools. As of June 2017, 88 per cent of the respondents rated their confidence as high or very high.

During the audit we spoke to a number of schools. Most had staff who attended one or both of the training packages. New principals and business managers we spoke to found the training to be particularly valuable.

While not directly attributable to the training, some of the value-for-money practices implemented at schools included:

• bulk purchasing resources and stationery
• reducing energy consumption by using timers
• using volunteer parent labour
• installing water bores
• joining with nearby schools to increase purchasing power for items such as stationery and camps.

2.1 Context

Victoria's 79 local councils provide a wide range of services to their communities and are responsible for managing significant assets and infrastructure. Individual councils have varying needs for support, depending on factors such as location, demographics, and council capacity.

Both the Municipal Association of Victoria (MAV) and Local Government Victoria (LGV) provide support to councils to help them carry out their duties efficiently and improve service delivery for local residents.

MAV is the peak membership body for local councils and is incorporated by the Municipal Association Act 1907 (the MA Act). MAV advocates for the interests of its member councils, and delivers a range of programs and services designed to build the capacity of local government, facilitate networks, discuss policy, and support councillors' professional development. MAV also provides procurement and insurance services to member councils.

LGV is part of the Department of Environment, Land, Water and Planning (DELWP) and oversees the administration of the Local Government Act 1989 (the LG Act) and other related acts, including the MA Act and the City of Melbourne Act 2001. LGV supports the Minister for Local Government (the minister). It also provides guidance to local councils on complying with the LG Act, and manages a number of initiatives designed to improve capacity, performance, and governance in the sector.

2.1.1 Our previous audit

In February 2015, we tabled our audit report Effectiveness of Support for Local Government. In this audit we examined the support provided to local councils by both MAV and LGV. We broadly defined 'support' as any activity or program designed to help councils carry out their obligations. Support may include providing advice, guidance, and services such as training.

Overall, neither MAV nor LGV could clearly demonstrate that their support activities were contributing to improved council performance or more efficient service delivery. Both lacked robust performance monitoring and evaluation mechanisms to measure the outcomes of their support initiatives.

A lack of appropriate governance arrangements and outdated legislation compromised MAV's support to local councils. We recommended a comprehensive update of all internal MAV policies and controls relating to conflicts of interest, fraud and corruption, and gifts, benefits and hospitality. We also identified deficiencies in MAV's record management practices and performance review processes for staff and its board, and weaknesses in probity in procurement and project management processes.

We found that there was further scope for MAV and LGV to coordinate their support activities and clarify areas of collaboration. While we acknowledged that MAV and LGV have distinct roles and responsibilities—and that they will often have differing views on issues affecting the sector—both share a common interest in improving the capability and reputation of local government.

We made 15 recommendations—eight directed to MAV, four to LGV, two jointly to both, and one to DELWP.

2.2 Conclusion

Both MAV and LGV have responded to our recommendations and have taken appropriate action to address the underlying issues identified in our 2015 audit.

Following our audit, MAV comprehensively reviewed and updated all relevant internal governance policies and controls—including in the areas of conflicts of interest, procurement and records management. MAV also implemented an improved performance monitoring and reporting framework at both an organisational and an individual program level. While it will take time to see the full impact of these policy changes, MAV's increased focus on probity and performance reporting should increase the transparency of its operations and accountability to its member councils.

LGV is currently conducting a review of the MA Act focused on clarifying MAV's functions and modernising its governance and accountability framework. In May 2017, LGV released a consultation paper that outlined a range of possible reforms to the MA Act, including clarifying MAV's functions and legal status as a public sector entity, specifying the governance and reporting responsibilities of the board, and increasing transparency in procurement. These proposals aim to ensure that MAV—like other public entities—is subject to appropriate oversight, while remaining primarily accountable to its member councils.

LGV has demonstrated an improved approach to evaluating its support programs, and more routinely collects feedback from councils on the relevance and usefulness of its guidance materials. It has also developed a process for monitoring councils engaged in group procurement under section 186 of the LG Act. This process will provide LGV with greater assurance that group tenders are conducted competitively and achieve value for money for local government.

MAV and LGV are now working together more strategically, and have established a joint annual strategic planning process and associated action plan.

2.3 Reviewing the Municipal Association Act 1907

In our 2015 audit, we found that there had been no comprehensive review of the MA Act since its inception in 1907. As a membership association incorporated by an Act of the Victorian Parliament, MAV operates in a unique legislative environment. Its governance model is unusual in that it is not subject to the range of oversight legislation that applies to many other public sector entities and bodies, such as the Public Administration Act 2004. MAV is also not subject to the Commonwealth Insurance Act 1973 or oversight by the Australian Prudential Regulation Authority, despite managing a mutual liability scheme on behalf of councils.

Although the minister is responsible for the administration of the MA Act, MAV's activities have not been subject to regular monitoring to ensure that it is complying with the MA Act. In our 2015 audit, we recommended that DELWP review MAV's legislative framework as a priority, to clarify its role and responsibilities, and to ensure its activities are subject to modern governance mechanisms. We also recommended that LGV routinely advise the minister on MAV's performance and compliance with the MA Act.

Figure 2A shows our 2015 audit recommendations to address this issue and the actions DELWP and LGV have taken in response to the recommendations.

Figure 2A

Recommendations and actions taken

Recommendation		Status
1	That DELWP as a priority, reviews and determines MAV's functions, roles, responsibilities, powers and obligations through an analysis of its existing legal framework and: ensures this is reflected in the MA Act ensures contemporary standards of governance and accountability are met, including the role, function and make-up of MAV's board consults with relevant departments to consider whether the Public Administration Act 2004, Financial Management Act 1994 and Public Records Act 1973 should apply to MAV either directly or through its enabling legislation assures itself, following any review, that all activities undertaken by MAV are clearly within its power.	In progress
	In 2015 the minister launched a review of the LG Act. The terms of reference for the review committed LGV to reviewing all legislation the minister has administrative responsibility for, including the MA Act. In May 2017 LGV released a consultation paper that included proposed reform options for the MA Act. Reforms being considered include clarifying MAV's functions and legal status, specifying the governance and reporting responsibilities of the board, and mandating the adoption of a procurement policy. The consultation paper also included a proposal to introduce a prudential supervision framework for MAV's insurance services, similar to the standards required by the Australian Prudential Regulation Authority and other Victorian public sector insurance agencies.
14	That LGV routinely monitors the performance of MAV including its compliance with the MA Act, and advises the minister accordingly.	In progress
	LGV has continued to brief the minister on MAV's annual report. LGV advised that no further monitoring of MAV's activities has occurred, pending the review of the MA Act.

Source: VAGO.

Performance improvements

It is too early to determine the impact that the current review of the MA Act will have on MAV. The review aims to create contemporary legislation that delivers a governance and accountability framework for MAV to ensure it is accountable to its member councils, the minister, and the wider local government sector.

2.4 MAV's internal controls and governance

Our 2015 audit identified inadequacies in MAV's internal policies and processes to manage staff conflicts of interest, corruption and fraud, and gifts, benefits and hospitality. We also found weaknesses in MAV's procedures for procurement and project management, and inconsistent record management practices. We found that MAV's internal audit activity had been limited in addressing gaps in its controls and governance framework.

Figure 2B shows our 2015 audit recommendations to address this issue and the actions MAV has taken in response to the recommendations.

Figure 2B

Recommendations and actions taken

Recommendation		Status
2	That MAV reviews its policies and controls relating to conflict of interest, corruption and fraud, and gifts, benefits and hospitality to align with better practice, and trains its staff accordingly and proactively monitors the application of these policies and controls.	Completed
	MAV has comprehensively reviewed and updated its policies and procedures for conflict of interest, fraud and corruption, and gifts, benefits and hospitality. These policies are available on MAV's website, along with instructions for reporting suspected fraudulent or corrupt conduct. MAV's new policies align with better practice standards developed by the Victorian Public Sector Commission (VPSC).
3	That MAV develops and implements a performance management framework for its board and staff that is aligned with better practice.	Completed
	MAV has developed and implemented a new staff performance plan and review policy that details a clearer, formalised process for managing staff performance. The new policy sets out the responsibilities and expectations of managers and staff, and provides for individual staff to develop work plans that link their activities to specific goals within MAV's strategic plan. MAV has also implemented a board performance policy that closely mirrors the model the VPSC recommends for public sector boards.
4	That MAV develops and implements a project management framework aligned with better practice covering all project phases from initiation to completion.	Completed
	MAV has introduced a project management policy that covers the entire life span of a project from initiation to evaluation. The policy includes a number of templates that MAV staff can use to help define project objectives, assess risks, and monitor project outcomes.
5	That MAV reviews and updates its records management policy to align with better practice.	Completed
	MAV has comprehensively reviewed its recordkeeping practices in line with the provisions of the Public Records Act 1973.
6	That MAV reviews its internal audit program and ensures it routinely covers all key procedures and controls associated with all aspects of procurement, conflict of interest and fraud and corruption.	Completed
	The internal audit program explicitly addresses the areas of risk we identified in our 2015 audit. MAV's response to a recent internal audit on managing conflicts of interest, and fraud and corruption controls demonstrates a commitment to implementing internal audit recommendations. MAV's new internal audit plan will provide the board with greater assurance of the effectiveness of its governance framework and control environment.
7	That MAV as a priority, reviews and updates its procurement policies and procedures, so that: they comply with better practice high probity standards and appropriate controls around conflicts of interest are applied to all phases of procurements it actively monitors compliance with updated policies and procedures.	Completed
	The procurement function at MAV (MAV Procurement) has comprehensively revised its procedures and developed a number of new policies, guidelines, and checklists to assist staff. MAV Procurement has committed to following the Victorian Local Government Best Practice Procurement Guidelines 2013. MAV's managers are monitoring compliance with the new policies, and MAV Procurement will be the focus of an internal audit in 2017. MAV has introduced a new policy and online purchase order system for internal purchasing to ensure consistency and transparency in all purchasing decisions.

Source: VAGO.

Performance improvements

Since our audit, MAV has committed to maintaining the same standards of governance, probity and staff conduct as local councils and other public sector bodies.

To achieve this MAV has updated or introduced new policies aligned with better practice, covering staff and board performance, project management, records management, internal purchasing and procurement. MAV has also comprehensively reviewed policies and controls to manage conflicts of interest, corruption and fraud, and gifts, benefits and hospitality, providing a consistent framework to guide staff conduct.

MAV has trained staff in the new processes and has developed an internal audit schedule focused on monitoring compliance. The first two internal audits of the new program—on managing conflicts of interest, and fraud and corruption controls—have been completed. MAV has adopted the minor process improvements recommended by these audits.

2.5 Evaluating support for local government

2.5.1 MAV

In our 2015 audit, we were critical of MAV's approach to performance monitoring—both at an organisational and program level. We noted that MAV's strategic plans and annual reports listed MAV's activities, rather than demonstrating how these activities have achieved defined outcomes.

Figure 2C shows our 2015 audit recommendations to address this issue and the actions MAV has taken in response to the recommendations.

Figure 2C

Recommendations and actions taken

Recommendation		Status
8	That MAV improves the monitoring, evaluation and reporting of its support activities, including developing relevant and appropriate performance measures, and publicly report its progress and performance.	In progress
	MAV is in the process of implementing an improved strategic planning and performance monitoring framework for reporting that includes relevant outcome measures. MAV advised that its new approach to performance monitoring will be fully realised in the forthcoming 2017–19 strategic work plan and the 2017–18 annual report. MAV will use the outcomes reported as a benchmark to assess future performance.
9	That MAV improves councillor development, training and events evaluations to clearly measure and demonstrate their impact on participants and on council performance.	Completed
	MAV has developed specific performance measures to guide the delivery of councillor development, training and events, with a view to demonstrating the impact on participants over time. For some activities MAV is now able to demonstrate measurable improvement—such as an increase in the number of candidates standing for election who request briefings from MAV, increased participation of rural and regional councillors in MAV training and events, and an increase in the cultural diversity of participants in training and events.

Source: VAGO.

Performance improvements

MAV has refined its performance monitoring framework over successive strategic work plans and annual reports to ensure they enable a meaningful assessment of MAV's impact within the sector.

MAV has advised that its new approach to performance monitoring will be fully realised in the forthcoming 2017−18 annual report and new 2017–19 strategic work plan, and MAV will use the outcomes reported as a benchmark to assess future performance. MAV's new approach to performance monitoring will provide the MAV board and member councils with a clearer view of MAV's achievements and an assessment of how its activities are contributing to specific outcomes.

2.5.2 LGV

LGV administers a number of funding initiatives and provides a range of support and guidance materials to help local councils fulfil their role. In our 2015 audit, we noted that LGV did not evaluate its programs and support initiatives against defined objectives. We also found that LGV's business plan did not clearly identify how the agency's planned activities contributed to its overarching goals and did not provide clear performance measures.

Figure 2D shows our 2015 audit recommendations to address this issue and the actions LGV has taken in response to the recommendations.

Figure 2D

Recommendations and actions taken

Recommendation		Status
10	That LGV improves its monitoring, evaluation and reporting on support activities to demonstrate the achievement of intended objectives and outcomes, including in relation to its business plan, and ensures it has relevant and appropriate performance measures, and publicly reports its progress and performance.	Completed
	LGV has improved its monitoring, evaluation and reporting for support activities at both an organisation-wide level, and for specific programs. LGV has conducted several full evaluations of programs in line with DELWP's guidance, and has redesigned the administration and project acquittal requirements of some programs in response to the findings of the evaluations.
11	That LGV develops processes to seek feedback from councils on the use of its guidance material and website information to identify opportunities for continuous improvement.	Completed
	LGV has improved its methods for seeking feedback from councils. As well as using detailed surveys, LGV has conducted in-depth focus group interviews with council staff who use its guidance material, and has updated it to better align this material with council needs. LGV's use of steering committees and reference groups enables it to consult with relevant stakeholders through the life of a particular project or initiative, which is useful in building feedback into the process of program delivery rather than only seeking feedback once a project is completed.

Source: VAGO.

Performance improvements

LGV can now more effectively demonstrate the effectiveness of its support for local government, and has implemented feedback processes to ensure that its guidance materials are relevant to councils' needs. LGV has improved its monitoring, evaluation and reporting on support activities and programs. In response to the findings of evaluations, LGV has redesigned the administration of some programs to ensure they have clearly defined goals that align with broader sector objectives.

LGV has also improved the quality of project acquittals and reporting for initiatives that provide funding to individual councils, which has improved transparency in the use of LGV funds. LGV's most recent program evaluation, which focused on the Collaborative Councils Sustainability Fund Program, is notable for being conducted internally by LGV, rather than by external consultants, thereby building the capacity of LGV's staff.

LGV's current business plan has a stronger focus on performance outcomes. Despite this, many of the stated outcomes are relatively broad and difficult to measure, such as LGV having good credibility in the local government sector. LGV does not report publicly against its business plan, although it indicated that there are plans to report on how LGV's activities contribute to achieving the aims of the Ministerial Statement on Local Government. This should increase the transparency of LGV's activities and improve accountability to the local government sector.

2.6 Monitoring local government procurement

Section 186 of the LG Act allows the minister to approve a council entering into a contract for the purchase of goods or services, or for carrying out works, without conducting a competitive tendering process. Councils must apply to the minister, specifying the reasons for seeking an approval and outlining the potential benefits. For example, some local councils have sought approval to join a shared library management service, enabling them to save on the cost of conducting separate tendering processes and access savings through a shared contract.

The minister has also granted all councils a 'blanket' approval under section 186, which enables them to participate in MAV Procurement's group tendering processes and contracts. In our 2015 audit, we noted that LGV did not monitor whether councils achieved the savings intended by the section 186 approvals, or whether the MAV Procurement contracts used by councils were established following an appropriate competitive tendering process.

Figure 2E shows our 2015 audit recommendation to address this issue and the actions LGV has taken in response to the recommendation.

Figure 2E

Recommendation and actions taken

Recommendation		Status
15	That LGV actively monitors entities that have been granted approvals under section 186 of the Local Government Act 1989, to ensure they comply with any requirements specified in the approval, and advise the minister accordingly.	In progress
	LGV now requires council chief executive officers (CEO) to provide a formal attestation that they have realised the intended benefits they sought when applying for an approval under section 186 of the LG Act. LGV has also developed a new monitoring protocol that from July 1 2017 will require MAV Procurement—and any other purchasing agents who receive an approval under section 186 of the LG Act—to provide an annual declaration that they are complying with the requirements of the relevant approval instrument. The review of the MA Act also includes proposed reforms to MAV's procurement activities, including mandating the adoption of a procurement policy and requiring MAV to provide information to councils on how group tenders are conducted.

Source: VAGO.

Performance improvement

LGV has improved how it monitors procurement in local government, and now requires council CEOs to provide a formal attestation that they have accrued the intended benefits they sought when applying for an approval under section 186 of the LG Act. Although LGV does not undertake any further testing or investigation of the CEOs' attestations, the process has introduced a level of accountability that was previously absent.

LGV has also developed a new monitoring protocol that will require MAV Procurement—and any other purchasing agents that receive an approval under section 186 of the LG Act—to provide an annual declaration that they are complying with the requirements of the relevant approval instrument. This process will begin on 1 July 2017 and should provide LGV with a greater level of assurance that group procurement activities are conducted competitively, and that councils using purchasing agents are achieving the savings they anticipated.

The current review of the MA Act may result in further changes to how group procurement is monitored.

2.7 Coordination and collaboration

In our 2015 audit, we found that there was scope for LGV and MAV to work more closely together to identify council needs and consider how the activities of each can contribute to achieving the strategic goals of the sector. There were no formal mechanisms or processes to support strategic planning or coordinate activities to avoid duplication.

In our report, we identified the Victorian State–Local Government Agreement as a focal point for future collaboration, and suggested LGV and MAV develop a work plan to implement the goals of the agreement. While the Victorian State–Local Government Agreement has not been progressed by government, MAV and LGV developed a joint strategic action plan to define when and how they should work together.

Figure 2F shows our 2015 audit recommendations to address this issue and the actions LGV and MAV have taken in response to the recommendations.

Figure 2F

Recommendations and actions taken

Recommendations		Status
12	That LGV and MAV review and document how and when they should work together to ensure the efficient, effective and economic delivery of support to councils, including clarifying roles and responsibilities for support activities, and communicate this to councils.	Completed
13	That LGV and MAV undertake regular joint strategic planning to: share knowledge and intelligence on council needs agree on council support priorities and areas of collaboration agree on a program of work to be reflected in the agreed annual work plan between state and local government, and local government peak bodies.	Completed
	LGV and MAV have established an annual strategic planning process to formalise when and how to undertake collaborative activities. They have developed a strategic action plan that outlines joint activities to be undertaken and defines key differences in their roles and responsibilities. LGV and MAV meet regularly to share information about their work, and both participate in a number of forums and working groups alongside other local government stakeholders to discuss the support needs of councils. The current review of the MA Act provides an opportunity to consider further the relationship between MAV and LGV and clarify their distinct roles and functions.

Source: VAGO.

Performance improvements

Although LGV and MAV have distinct roles and responsibilities—and sometimes have considerably different views on issues affecting the sector—both are now working together more strategically to support councils.

LGV and MAV have established an annual joint planning process and a strategic action plan that formalises when and how they will undertake collaborative activities. They also meet regularly to share information about their work and discuss council needs.

Depending on the findings of the review of the MA Act, and any future amendments, MAV and LGV should continue to conduct a joint annual strategic planning process to identify areas of collaboration and define goals for the delivery of joint activities.

3.1 Context

The myki ticketing system is a stored-value smartcard system that allows passengers to pay for travel on metropolitan trains, trams and buses, V/Line commuter trains and some regional town buses. It was first introduced in Victoria in 2008, replacing the magnetic-stripped Metcard tickets for metropolitan public transport by late 2012. The first myki contract expired in 2016 and the government entered into a new $700 million seven-year contract in July 2016 for the operation of the myki ticketing system.

3.1.1 Agency roles

Public Transport Victoria

Public Transport Victoria's (PTV) primary objective under the Transport Integration Act 2010 (the TI Act) is to plan, coordinate, provide, operate and maintain a safe, punctual, reliable and clean public transport system, consistent with the vision statement and transport system objectives contained in the TI Act. This includes providing and operating a public transport system, managing the public transport ticketing system contract, and collecting and distributing the revenue from fares.

Department of Treasury and Finance

The Department of Treasury and Finance (DTF) provides economic, financial and resource management advice to help the Victorian Government deliver its policies. This includes helping the government oversee infrastructure projects and frameworks such as the High Value High Risk framework and the Gateway Review process, which myki is subject to.

3.1.2 Our previous audit

In June 2015, we tabled our audit report Operational Effectiveness of the myki Ticketing System. We examined myki's operational effectiveness and whether myki was achieving the outcomes and benefits expected from its introduction. At the time of our audit, PTV was planning to retender the contract for myki's continued operation, which was due to expire in 2016.

There were significant implementation issues with the system, which precipitated six major amendments to the original 2005 contract. The time taken to design and deliver myki more than quadrupled from two years to more than nine years. This led to significant unanticipated additional costs—a $550 million (55 per cent) increase on the project's original budget of almost $1 billion.

The significant delays in implementing myki and the related cost increases were largely due to deficiencies in the original contract and governance arrangements, which compromised the achievement of myki's original business case objectives and related benefits.

Although our 2015 audit found that PTV had subsequently improved oversight of the myki service provider and related contractual arrangements, significant risks to the state remained:

• PTV did not possess a complete and reliable picture of myki's operational performance due to weaknesses with the contract's revised performance regime and its implementation.
• PTV had not clearly defined the benefits and outcomes they were seeking from the myki retender, which reduced the transparency of and accountability for myki's future performance.
• None of the agencies responsible for myki had assessed if it had achieved any of its expected benefits—despite previous commitments to the Public Accounts and Estimates Committee in 2012 that this would occur.
• Compressed time frames from the myki retender, caused by previous delays, and the imminent expiration of the contract in place at the time risked exposing the state to significant additional costs.

In our 2015 audit, we concluded that PTV needed to urgently address these issues to avoid perpetuating previous mistakes, and to ensure that Victoria could maximise value from the future operation of myki.

We made five recommendations—four directed to PTV and one to DTF—all of which were accepted.

3.2 Conclusion

PTV and DTF have responded to our recommendations and have taken some action to address the underlying issues identified in our 2015 audit, although further work is required.

PTV has developed a stronger performance regime for the new myki contract, which includes a broader range of performance measures and more robust performance indicators. It is still too early to assess whether the new performance regime has improved the delivery of ticketing services and service provider accountability.

PTV is adopting a more rigorous approach to monitoring the service provider's performance. It is also intending to exercise its right under the contract to audit and verify performance data early in the service term. These actions should help PTV gain a more complete and reliable picture of myki's operational performance, and verify the accuracy of reported performance results and the related performance payments charged by the service provider. The effectiveness and integrity of the performance regime will depend on how PTV continues to monitor performance, enforce the performance regime and manage the contract.

PTV has not yet completed a post-implementation review (PIR) against myki's original objectives and benefits. The delay in conducting the PIR is a significant missed opportunity for PTV to incorporate relevant lessons into the new myki contract. The PIR would have provided valuable insights into how to refine and update myki's original value proposition and assumed benefits, particularly in light of the changes to myki since the project was first approved in 2004. PTV could still use lessons from the PIR to inform any future procurement of public transport ticketing services.

PTV is in the process of developing a benefits management plan for the Ticketing Services Retender (TSR). The development and implementation of a benefits management plan should improve PTV's oversight and management of TSR benefits realisation, and provide greater transparency and accountability for the $700 million investment into the TSR. However, PTV should have completed this before it awarded the new contract in July 2016 so that PTV, bidders for the retender and the state had a clearer understanding of the benefits expected, and how to measure them.

DTF provided comprehensive advice to the Victorian Treasurer about the impact of Cabinet conventions on the Gateway Review process, the High Value High Risk framework and the related benefits reviews. It also drafted a policy on business case disclosure for Cabinet consideration, which included a proposal for responsible agencies to retain detailed project information beyond changes in government to enable more effective project oversight and accountability. As the draft submission did not progress to Cabinet, Cabinet conventions are still posing a barrier and preventing agencies from accessing the information needed to effectively implement, monitor and evaluate investment projects.

3.3 Measuring myki's performance

Our 2015 audit found the performance regime in the initial 2005 myki contract was complex, onerous and—with 81 key performance indicators (KPI)—difficult to apply in practice. This impeded effective contract management. Further, the performance indicators only applied once myki was fully functional.

PTV revised the performance regime in March 2013 under the sixth (and last) amendment to the then myki contract—the New Ticketing Solution Project Agreement, Sixth Amendment and Restatement Deed (AD6). The aim was to improve clarity and focus by reducing the number of measures from 81 to 27.

The AD6 performance regime also included incentives for the service provider to promptly address faults with the myki devices and related software upgrades that previously disrupted system operations. It also increased the service provider's focus on assuring the accuracy of financial information recorded by the system.

Our 2015 audit found that fundamental issues remained:

• There was no supporting evidence to show that PTV had set the AD6 performance benchmarks (including financial incentives and abatement thresholds) at appropriate levels and that the benchmarks were adequate for driving performance improvement.
• Performance indicators did not address key aspects of performance—examples include focusing on equipment availability but not transaction speed, and not distinguishing performance problems with equipment at busy train stations during peak periods from equipment at infrequently used stations during off peak periods.
• The performance regime did not have the flexibility to address emerging service priorities and performance issues.
• Performance measures did not provide a complete view on how well the ticketing system was performing as a whole and were therefore unable to inform an assessment of myki's impact on improving performance and management of the public transport system.

Figure 3A shows our 2015 audit recommendations to address these issues and the actions PTV has taken in response to the recommendation.

Figure 3A

Recommendation and actions taken

Recommendation		Status
1.1	That PTV strengthens its performance monitoring arrangements for myki prior to awarding the new contract by: assessing the adequacy of existing performance measures and standards for driving improvements in performance	Completed
	Developing a new performance framework PTV has developed a performance framework for the new contract comprising 34 KPIs with an additional four optional KPIs. The KPIs are divided into five 'layers', according to the purpose they serve in the performance regime. The five layers are: Layer 1—Service disruption Layer 2—Customer experience Layer 3—Service responsiveness Layer 4—Flexible performance requirements Layer 5—Business performance indicators. See Appendix B for more details on the performance management framework under the new ticketing contract. Assessing the adequacy of existing performance measures PTV assessed AD6 KPIs for their appropriateness for the new contract by examining: their alignment with the nature and scope of service requirements under the new contract their alignment with performance objectives of the new contract the appropriateness of benchmarks and thresholds compared with existing performance levels the effectiveness of financial incentives to drive desired service provider behaviour and outcomes the applicability of AD6 KPI data sources for the new regime. These actions resulted in PTV keeping all of the AD6 KPIs for the new contract but with revised methods for measuring and accounting for performance.
1.2	That PTV strengthens its performance monitoring arrangements for myki prior to awarding the new contract by: developing new measures addressing how well the equipment operates as distinct from the length of time for which it is available	Completed
	PTV developed seven new KPIs that monitor how well myki equipment operates from a customer's perspective. These monitor: touch on/touch off transaction speed of devices card vending machine transaction speed billing accuracy for cardholders condition and presentation of cardholder-facing devices start-up times of bus and tram driver consoles, which monitor and control on-board myki devices.
1.3	That PTV strengthens its performance monitoring arrangements for myki prior to awarding the new contract by: reviewing on at least an annual basis and, where necessary, adjusting performance incentives to support further improvements in performance or achievement of emerging service priorities	Completed
	PTV has introduced flexible KPIs in the new performance regime that can be set annually to respond to emerging issues and service priorities. Flexible KPIs are a bonus component of the performance regime and are linked to financial incentives. PTV and the service provider can agree to nominate up to four flexible KPIs per year.
1.4	That PTV strengthens its performance monitoring arrangements for myki prior to awarding the new contract by: developing a broader framework to assess myki's efficiency and effectiveness and its impact on improving performance and management of the public transport system.	Completed
	PTV's 2016–2020 Corporate Plan details the intended outcomes and associated KPIs for public transport services. Two of the six strategic domains of the corporate plan apply to myki: deliver customer services network operations and asset performance. PTV measures 'deliver customer services' KPIs using customer satisfaction surveys for each transport mode. The surveys include measuring the importance of myki to customers. 'Network operations and asset performance' includes a KPI to measure myki device availability, which was not featured in PTV's corporate plan at the time of our 2015 audit.

Source: VAGO.

Performance improvements

PTV has developed a stronger performance regime for the new myki contract, which should provide a more accurate, complete and clearer picture of how well the myki ticketing services system is performing as a whole. Overall, the new regime contains more robust performance indicators with improved methods for measuring and accounting for performance. A notable example is applying weightings to the KPI for equipment unavailability to take into account busy train stations and peak periods, which is more critical compared to equipment unavailability at an infrequently used station during off peak periods.

The new performance regime also addresses broader aspects of performance than the previous contract, including performance from a customer's perspective such as device transaction speed, and provides flexibility to address emerging service priorities and issues—features that the previous regime lacked.

PTV's corporate plan includes monitoring the importance of myki to customers and myki device availability. This assists PTV with monitoring myki's impact on the delivery of public transport services.

Residual risks

It is still too early to assess whether the new performance regime has improved the delivery of ticketing services and service provider accountability. The effectiveness and the integrity of the performance regime, and PTV's ability to determine incentive payments and penalty abatements accurately, will depend on how PTV monitors the service provider's performance, enforces the performance regime and manages the myki contract.

3.4 Monitoring myki's performance

The previous myki contract required the service provider to prepare monthly KPI reports, which PTV used to assess the service provider's performance and determine incentive payments and penalty abatements.

Our 2015 report found that PTV did not have all the information needed to sufficiently assess and validate the service provider's reported performance results. PTV was therefore unable to assure that it issued all incentive payments and penalty abatements correctly. Our 2015 report also found that, although the contract allowed PTV to conduct an audit that would give it access to the performance data and systems maintained by the service provider, it did not do so. PTV's inability to verify the accuracy of reported results and failure to use its audit powers affected the integrity and effectiveness of the performance regime.

Figure 3B shows our 2015 audit recommendation to address these issues and the actions PTV has taken in response to the recommendation.

Figure 3B

Recommendation and actions taken

Recommendation		Status
2	That PTV uses its right under the contract to audit and verify the performance data provided by the contractor.	Ongoing
	Exercising audit rights under the previous contract PTV engaged an external party in June 2015 to perform an 'agreed‑upon procedures engagement' to determine whether the service provider was complying with the agreed procedures for calculating and reporting KPIs. The external party conducted the engagement in accordance with the Auditing and Assurance Standards Board (AUASB) Standard on Related Services ASRS 4400 Agreed‑Upon Procedures Engagement to Report Factual Findings. The agreed‑upon procedures engagement did not find any issues. Exercising audit rights under the new contract PTV engaged an external party to conduct a performance requirements audit in May 2017. The purpose of the audit is to confirm that the service provider is complying with the agreed procedures for calculating and reporting KPIs. The scope of the audit includes examining five of the 34 KPIs. The in-scope KPIs relate to service disruption, service responsiveness and business performance. Verifying performance data under the new contract PTV has revised its approach to reviewing the service provider's monthly KPI reports. PTV is using staff with subject matter knowledge to review reported KPIs relevant to their area of expertise. It is also providing detailed feedback and seeking monthly clarification on reported KPIs from the service provider.

Source: VAGO.

Performance improvements

PTV has improved its approach to monitoring performance under the new myki contract.

PTV is adopting a more rigorous approach to reviewing the service provider's monthly KPI reports and verifying performance data under the new myki contract. By using staff with subject matter expertise to review reported KPIs, PTV has been able to identify possible information gaps and anomalies in calculations, such as the omission of data, or inaccurate classification of items as meeting the performance requirement, which are linked to incentive payments and penalty abatements.

These anomalies may have resulted in the service provider overcharging for performance payments. PTV has requested further information from the service provider to clarify these matters. Under the contract, both PTV and the service provider have up to 12 months to dispute the amount of any performance payment included in a service payment invoice. PTV has advised that it is seeking to resolve performance payment matters within the 12-month time frame.

It is positive that PTV is planning to exercise its audit right early in the seven-year service term under the new contract. Conducting a performance requirements audit will help confirm whether the service provider is complying with the agreed procedures for calculating and reporting KPIs, and will enable PTV and the service provider to address any identified issues early in the service agreement.

Residual risks

Verifying each monthly KPI report is time consuming and depends on the service provider's cooperation in responding to PTV's requests for further information. In response to PTV's comments and request for further information about the January 2017 KPI report, the service provider stated in a March 2017 letter that it had satisfied the performance schedule under the contract in full and did not need to provide any further information to PTV. The service provider also suggested that PTV use its right to conduct a performance requirements audit to confirm the accuracy of the KPI reports.

If they remain unresolved, the accumulation of issues will make it difficult for PTV to have a clear picture of the true performance for each month. This will affect PTV's ability to accurately determine incentive payments and penalty abatements, and verify the accuracy of the performance payments charged by the service provider.

Conducting a performance requirements audit to confirm whether the service provider is calculating and reporting KPIs in line with the agreed procedures should help PTV address this issue. Findings from the performance requirements audit should also enable PTV to identify what steps it can take to ensure reported KPIs and the resulting performance payments are accurate.

3.5 Project benefits and outcomes

Background

The original 2004 myki business case set out the customer, transport operators and government objectives and justified the state's decision to invest almost $1 billion of taxpayer funds to develop and implement a ticketing system that would deliver significant benefits to Victoria. These benefits were expected to deliver between $6.3 and $10.8 million per year, of which $4 to $5 million was described as relatively tangible. The major expected benefits included:

• increased attractiveness of the public transport system and therefore increased patronage
• reduced fare evasion and concession fraud
• increased ticket validations due to improved reliability
• operating cost savings due to reduced bus boarding times.

Our 2015 audit found that PTV had not conducted a post-implementation review (PIR) of myki against the original objectives and expected benefits, and it was not clear to what extent myki had achieved its objectives. This was a significant accountability and transparency issue given that myki is a high value high risk project and the additional costs of $550 million, significant project delays and substantial reduction in project scope compared with the original myki project that was initially approved in 2004.

Our 2015 audit also found that Cabinet conventions were posing a barrier to this work. Business cases are usually Cabinet-in-Confidence documents and must be surrendered to the Department of Premier and Cabinet (DPC) when there is a change of government. As such, Cabinet conventions were preventing agencies from accessing the detailed project information contained in business cases to conduct benefits reviews.

We found that this was impeding the effective governance of myki, and potentially all other major investment projects by the state, which have life cycles that usually extend beyond the term of Parliament and any government. It was also compromising the effectiveness of the state's Gateway Review process and High Value High Risk framework, both of which are overseen by DTF. Our 2015 report concluded that if the barrier to accessing business cases was not resolved, there would be a risk that business cases would be developed simply to get initial approval, which could then be disregarded following a change of government.

Figure 3C shows our 2015 audit recommendations to address these issues and actions PTV and DTF have taken in response to the recommendations.

Figure 3C

Recommendations and actions taken

Recommendation		Status
3	That PTV seek access to the original myki business case in consultation with DPC and: conduct a post-implementation review of the myki project against its original objectives and benefits incorporate relevant lessons into the new myki contract as soon as possible and in any future subsequent procurement of public transport ticketing services.	In progress
	Accessing the original business case PTV sought access to the original myki business case from DPC in September 2015. DPC responded by offering PTV access to an April 2004 version of the myki business case and another 'relevant' document it had located in January 2016. DPC did not confirm whether the business case it was making available to PTV was the final approved version and did not inform PTV what the other relevant document was. PTV did not access the business case DPC was making available as DPC was unable to attest that it was the original (final) business case and PTV had an April 2004 copy in its possession already. PTV advised that it did not access the other 'relevant' document as our recommendation was for PTV to obtain the 'original business case'.
	Post-implementation review PTV appointed an external provider in February 2017 to undertake the PIR. The PIR will be performed against the myki business case PTV inherited from the former Transport Ticketing Authority, although it is unclear whether it is the original (final) business case. The current status of the PIR is not clear.
4	That DTF coordinate with DPC to advise the government on the impacts of current Cabinet conventions on the Gateway Review and the High Value High Risk framework and related benefits reviews.	Completed
	DTF prepared a brief to the Victorian Treasurer in September 2015 advising about the impact of current Cabinet conventions on the Gateway Review process, the High Value High Risk framework and the related benefits reviews. Following this briefing, DTF drafted a business case disclosure policy in February 2016 for Cabinet to consider, as requested by the Victorian Treasurer. In drafting the Cabinet submission proposal, DTF consulted with all departments through the Infrastructure Policy Reference Group and DPC.

Source: VAGO.

3.5.1 myki post-implementation review

Performance improvements

Performance improvements are yet to be realised as PTV is still addressing our recommendation to conduct a PIR.

PTV appointed an external provider to undertake the PIR in February 2017—more than 12 months after being able to access the business case held by DPC and more than seven months after the new myki contract was signed in July 2016.

Residual risks

PTV has not yet completed the PIR against the project's original objectives and benefits. This means that the new myki contract, which came into effect 1 January 2017, has not been informed by the lessons from a PIR, even though it could have been.

PTV asserts that a PIR of myki against the original business case would be of limited value because the new contract, unlike the original project, does not involve the development of a new ticketing system. However, the original myki contract established the objectives and benefits expected from both the initial build and ongoing operation of the system. Although the new contract focuses exclusively on myki's continued operation, a PIR could have provided valuable insights on how to refine and update myki's original value proposition and assumed benefits, particularly in light of the changes to myki since the project was first approved in 2004.

The significant challenges experienced during myki's initial build and rollout heighten the need for a PIR to improve accountability and transparency of the achievements from this significant expenditure. A PIR could also help PTV understand how benefits could be measured in the new myki contract. It would also provide PTV with important insights on the actions it should take to optimise achievement of the benefits in the new myki contract and any future procurement of public transport ticketing services.

We have been unable to verify the status of the PIR and when it is expected to be completed.

3.5.2 Impact of Cabinet conventions

Performance improvements

DTF provided a comprehensive briefing to the Treasurer about the impact of current Cabinet conventions on the Gateway Review process, the High Value High Risk framework and the related benefits reviews. The briefing included five options for increasing the public disclosure of business cases and one option for internal retention of infrastructure business case information. It also included a risks and benefits analysis of each option and examined each option's capacity to support DTF's project assurance frameworks and post-implementation evaluations beyond a government's term in office.

DTF recommended the following options in the briefing:

• Release future business case information for major asset investment projects with commercially sensitive information removed.
• Require departments to prepare investment business plans (IBP) for all approved infrastructure projects, and submit IBPs for high value high risk projects to DTF. IBPs would not be considered by Cabinet or classified as Cabinet‑in‑confidence documents, and would therefore be exempt from Cabinet conventions.

In recommending that both options be adopted, DTF provided advice to government on how the underlying issue identified in our 2015 audit could be addressed.

The Treasurer requested DTF to prepare a draft business case disclosure policy for Cabinet to consider, but did not approve DTF's recommendation to require departments to prepare IBPs, noting that this activity was to be performed by the soon‑to-be established body Projects Victoria.

The draft business case disclosure policy that DTF developed proposed the release of business cases for projects above $100 million total estimated investment with commercially sensitive information removed. The submission also sought approval for responsible agencies to retain detailed project information beyond changes in government to enable more effective project oversight and accountability.

DTF envisaged that this proposal would still enable agencies to complete comprehensive post-implementation evaluations and benefits reviews in the event of a change of government, without requiring agencies to prepare a separate IBP. This addresses the original intent of our recommendation.

Residual risks

Although DTF developed a draft business case disclosure policy, we were unable to verify if this was provided to the Treasurer for submission to Cabinet. DTF advised that it provided a submission to the Treasurer, but that it has not been progressed to Cabinet.

As a result, Cabinet conventions still pose a barrier, preventing agencies from accessing detailed project information needed to effectively implement, monitor and evaluate projects with business cases approved since our 2015 report. This impedes effective project oversight and accountability. Agencies remain unable to assess the extent to which the objectives of major investment projects have been achieved. This is a significant accountability and transparency issue.

3.5.3 Ticketing Services Retender—expected benefits and intended outcomes

At the time of our 2015 audit, PTV had begun planning the myki retender—known as the TSR—to ensure that the system would continue to operate after the existing contract expired in 2016. However, PTV had not clearly defined the expected outcomes and benefits from the retender.

The procurement strategy review, required by the DTF Gateway Review process, had also previously identified this issue in August 2014. The review noted that the TSR procurement strategy was lacking in key areas and highlighted that the identified critical success factors and benefits needed to be more specific and measurable.

In response to the review, PTV refined the TSR procurement strategy in October 2014. However, in our 2015 audit we found that the amended benefits in the TSR procurement strategy were still vague, and the associated measures were indicators of outputs rather than of outcomes and benefits.

We also found that the proposed measures would not effectively assess the outcomes from the retender, including implementation of the resulting contract. This would reduce the transparency and accountability of myki's performance under the new contract and the impact of the TSR. This was particularly important given the scale and complexity of myki, and the significant investment in its continued operation. Our 2015 report concluded that PTV needed to address this issue urgently.

Figure 3D shows our 2015 audit recommendation to address this issue and the actions PTV has taken in response to the recommendation.

Figure 3D

Recommendation and action taken

Recommendation		Status
5	That PTV clarifies the retender benefits and intended outcomes and develops measurable associated indicators.	In progress
	PTV is in the process of finalising a benefits management plan.

Source: VAGO.

Performance improvements

Performance improvements are yet to be realised, as PTV is still addressing our recommendation to clarify the retender benefits and intended outcomes, and develop measurable associated indicators.

PTV began work to develop a benefits management plan for the TSR after it awarded the new myki contract in July 2016. This is despite the tender decision review of the TSR in May 2016—which is required by the Gateway Review process—also noting the need for project success indicators. The review noted that PTV did not have a benefits realisation management plan for the TSR and recommended that PTV 'undertake a benefits realisation exercise for the overall project'. The readiness for market review as part of the Gateway Review process had also previously recommended this in August 2015.

PTV should have completed this work before it awarded the new contract in July 2016 so that PTV, bidders for the retender and the state had a clearer understanding of the expected benefits and how to measure them.

Residual risks

The development and implementation of a benefits management plan should improve PTV's oversight and management of the TSR benefits realisation, and provide greater transparency and accountability for the $700 million investment into the TSR.

It is important that PTV monitors and reports on the project success indicators throughout the service term and not just at the conclusion of the TSR project. This will help realise benefits from the investment by helping PTV identify when expected benefits are not on track so that it can take action to rectify identified issues. This is particularly important, given that the service term of the current contract is seven years.

A continuous focus on benefits throughout the service term will support PTV to make better decisions and provide lessons that will inform the shaping of future investments into public transport ticketing services.

3.6 Recommendations

We recommend that Public Transport Victoria:

1. complete a post-implementation review of myki against its original objectives and expected benefits, and incorporate relevant lessons into the benefits management plan for the current myki contract (see Section 3.5.1)

2. complete a benefits management plan for the current myki contract and monitor the progress of benefits realisation throughout the service term (see Section 3.5.3).

Additional School Costs for Families

RESPONSE provided by the Secretary, Department of Education and Training

Effectiveness of Support for Local Government

RESPONSE provided by the Secretary, Department of Environment, Land, Water and Planning

RESPONSE provided by the President, Municipal Association of Victoria