Managing the Performance of Rail Franchisees

Conclusion

Train and tram services have become more reliable and punctual under the current franchise agreements, and customer satisfaction has improved. However, weaknesses in the design and implementation of the performance regimes prevent PTV from maximising value from the current agreements.

It is positive to see that PTV has gained a good understanding of the strengths and weaknesses of the current agreements and is well prepared to negotiate better outcomes in future agreements. The challenge for PTV is to successfully negotiate and properly implement the new agreements and improve its management of contracts.

Our key area of concern has been PTV's lack of focus on managing its train and tram system assets, which it has only recently begun to address. As a result, PTV has limited assurance that these critical assets are being optimally maintained and used by operators to maximise the efficiency and effectiveness of train and tram services.

Findings

Managing contracts and performance

Measuring performance

PTV measures the performance of its train and tram operators, or franchisees, in several ways. It measures and publicly reports on the reliability and punctuality of train and tram services, and surveys customer satisfaction with these services.

One of the challenges for PTV has been that the current arrangements do not enable it to periodically review and reset reliability and punctuality performance thresholds. These thresholds set minimum standards for performance and outline consequences if performance falls below the threshold.

The franchise agreements also include operational and customer experience performance regimes that are linked to financial incentives and penalties.

Until recently, PTV has relied heavily on manual systems and on data reported by train operators to assess the performance of trains. In 2015, it introduced an automatic train position monitoring system to improve the way it keeps track of franchisees' performance.

PTV's automated system to monitor tram performance, introduced in 1987, uses old technology with significant limitations that reduce its effectiveness. PTV's own checking of the reliability of both systems is limited. It largely relies on the franchisees to assure the ongoing integrity of the monitoring systems and the data they produce. This means PTV has only limited assurance about the reliability of the underlying performance data that it reports publicly and uses to determine bonus and penalty payments for franchisees.

The customer experience performance regime under the current franchise agreements was poorly designed and costly to administer, making it ineffective. PTV decided that it was not delivering value for money, and largely abandoned it in 2012. Since then, PTV has relied on limited subjective data to assess the way franchisees fulfil their customer experience obligations and to determine the payment of associated bonuses and penalties.

Reporting performance

PTV's public reports identify reliability and punctuality performance thresholds for franchisees. However, they do not clearly explain that these thresholds actually represent underperformance levels instead of performance targets. PTV needs to ensure that its public reporting is easy to understand and clearly reflects franchisee performance.

Managing performance

While there have been improvement in train and tram reliability and punctuality, we found gaps in how PTV manages its franchise agreements:

• The agreements are not reviewed periodically, to track benefits, evaluate how objectives are being achieved and identify risks, issues and opportunities to improve the way the agreements are managed.
• There are no recorded knowledge management processes and inadequate recording of important information about franchisees' performance.

The current agreements give franchisees an exclusive right to negotiate a seven‑year extension of their agreement with the state, as long as they meet certain performance benchmarks. In December 2015, PTV assessed that both franchisees met the required benchmarks. However, PTV could not provide evidence to demonstrate the rationale for some of the benchmarks and how they were assessed.

PTV recognises the need for a more comprehensive contract management framework supported by effective systems and adequate resources. It has started to address these issues.

Managing assets

We found significant weaknesses in how PTV oversees the maintenance and renewal of assets leased to its franchisees under the current agreements. In particular, PTV does not have adequate medium- to long-term asset strategies for its train and tram network assets.

PTV also does not know enough about the condition of the assets used across both the tram and train networks. The lack of meaningful information about asset condition reduces PTV's capacity to appropriately plan, budget and prioritise maintenance and renewal work.

It is positive that PTV and the Department of Economic Development, Jobs, Transport & Resources have begun to address these weaknesses.

Extending the current franchise agreements

PTV has prepared for future franchise agreements by thoroughly examining the strengths and weaknesses of its current agreements. To help it negotiate with the current franchisees, PTV has developed an in-depth understanding of operating costs and has identified opportunities to improve the franchisee's financial reporting. This puts PTV in a good position to negotiate important improvements with franchisees.

Recommendations

We recommend that Public Transport Victoria:

1. finalise and introduce a contract management framework with appropriate processes and systems to better:

• manage knowledge and improve record keeping
• define roles and responsibilities and decision-making processes
• manage contract risks
• track benefits and evaluate performance throughout the life of the contract to identify problems, opportunities and weaknesses

(see Section 2.2).

2. improve how it monitors and manages the performance of train and tram franchisees by:

• periodically auditing franchisees' performance data and related systems (see Section 2.3.1)
• planning and implementing monitoring systems for the train and tram networks that provide comprehensive real-time journey information (see Section 2.3.1)
• preparing performance benchmarks for future franchise agreements that drive improvement and are periodically reviewed and reset where necessary (see Section 2.3.2)
• clearly explaining performance thresholds and how they relate to incentives and penalties in public reporting on train and tram performance (see Section 2.3.2)
• developing an effective way to monitor customer experience performance, using lessons from the current agreements (see Section 2.4)

3. before the next franchise agreement starts, work collaboratively with Victorian Rail Track (VicTrack) and the Department of Economic Development, Jobs, Transport & Resources to prepare and introduce a comprehensive framework for managing assets that is consistent with the Department of Treasury & Finance's Asset Management Accountability Framework and includes:

• clearly defined roles and responsibilities for all agencies and franchisees, including responsibility for operational control and management system assets
• systems and practices to capture and analyse comprehensive data about the condition of train and tram assets, including a baseline condition survey when the new agreements start and periodic reviews from then on
• clearly defined and ranked standards for asset management
• strategies for managing each groups of assets

(see Section 3.2)

4. evaluate the performance-based right to a contract extension under the current franchise agreement, to understand the benefits and weaknesses of this approach for future agreements (see Section 4.2.1)
5. improve its systems and capability to collect and analyse comprehensive information on franchisee costs throughout the life of franchise agreements (see Section 4.3.1).

In June 2016, the government announced a new agency—Transport for Victoria (TFV)—that will plan, coordinate and operate Victoria's transport system and associated agencies. We acknowledge that responsibilities for coordinating and implementing the above recommendations may be affected by this change.

Responses to recommendations

We have consulted with the Department of Premier & Cabinet, the Department of Economic Development, Jobs & Transport, Public Transport Victoria, and Victorian Rail Track and we considered their views when reaching our audit conclusions. As required by section 16(3) of the Audit Act 1994, we gave a draft copy of this report to those agencies and asked for their submissions and comments.

The following is a summary of those responses. The full responses are included in Appendix A.

Public Transport Victoria accepted all of the recommendations in the report and provided a detailed action plan on how it plans to address these recommendations. The Department of Economic Development, Jobs & Transport responded with a commitment to support Public Transport Victoria in implementing the recommendations.

1.1 Train and tram franchise agreements

The current train and tram franchise agreements—known as MR3—were established in 2009 through a competitive tender process. This was the third time that franchise agreements had been put in place for Melbourne's train and train services.

Figure 1A summarises the history of the franchise agreements for Victoria's train and tram services.

Figure 1A

Phases of Victorian metropolitan rail franchising

Source: VAGO, based on information from PTV.

Under MR3, the former Department of Transport appointed:

• Metro Trains Melbourne (MTM) as the train franchisee
• Keolis Downer Rail, trading as Yarra Trams, as the tram franchisee.

Both agreements were for an initial term of eight years, and expire in November 2017. Regional train services (V/Line) are not part of these agreements.

When PTV was established in April 2012, it took responsibility for these franchise agreements. Figure 1B shows the various parties involved in the agreements and Figure 1C explains the contractual arrangements.

Figure 1B

MR3 contractual framework

Source: VAGO, based on information from PTV.

Figure 1C

Summary of MR3 contractual arrangements

Source: VAGO, based on information from PTV.

1.2 Train and tram performance under the current franchise agreements

To manage the performance of the train and tram franchisees, PTV uses:

• publicly reported punctuality and reliability performance
• an operational performance regime (OPR), with associated incentive and penalty payments
• a customer experience performance regime (CEPR).

The MR3 agreements specify minimum service standards for punctuality and reliability—known as thresholds—which are not performance targets.

Under the OPR, PTV determines incentive and penalty payments by measuring the impact of an individual service delay or cancellation on passengers. This is a different measure to the publicly reported figures, which reflect whether trains and trams run on time. Appendix C outlines the OPR in more detail.

1.2.1 Train performance

In general, MTM has stabilised and improved train performance under the current agreement. Figure 1D shows that performance has significantly improved—it has exceeded the punctuality threshold—87 per cent—since late 2011 and has met or exceeded the reliability threshold since 2009 (the entire period of the agreement).

Figure 1D

Trains quarterly punctuality and reliability, December 2009 to June 2016

Note: The December 2009 quarter covers only one month of operations.

Source: VAGO, based on information from PTV published in Track Record.

PTV attributes improvements in train punctuality to more state investment in network upgrades, new rolling stock, franchisee performance initiatives and stable patronage growth.

The satisfaction of train customers is measured by a quarterly phone survey. As shown in Figure 1E, customer satisfaction has improved under the current agreement and has met the performance targets that were determined when the agreement was negotiated.

Figure 1E

Train customer satisfaction monitor results (phone survey), March 2010 to June 2016

Note: Under the current agreement, the train franchisee was required to achieve and maintain customer satisfaction index of at least 70 by 1 December 2014 and 80 per cent by 1 December 2018.

Source: VAGO, based on information from PTV.

The train franchisee is eligible for incentive payments and may incur penalties under the OPR and CEPR. Under the OPR, MTM received $28.6 million in total (comprising incentive payments less penalties) between December 2009 and June 2016. Under the CEPR, MTM received $1.3 million in incentive payments and had not incurred any penalties at the time of the audit.

1.2.2 Tram performance

Under the current agreements, tram performance has been steady and above the service thresholds—98 per cent for reliability and 77 per cent for punctuality—as shown in Figure 1F.

Figure 1F

Trams quarterly punctuality and reliability, December 2009 to June 2016

Note: The December 2009 quarter covers only one month of operations.

Source: VAGO, based on information from PTV published in Track Record.

PTV attributes improvements in tram punctuality to state initiatives, such as new timetables, and franchisee initiatives to improve communication and efficiency. Tram reliability and punctuality is affected by hot weather and the special events held from January to March. The introduction of the free tram zone in Melbourne's central business district in January 2015 resulted in a decrease in performance due to increased patronage. Tram performance is also increasingly affected by road traffic congestion.

The satisfaction of tram customers is measured by a quarterly phone survey. As shown in Figure 1G, customer satisfaction has improved under the current agreement and has met the performance targets that were determined when the agreement was negotiated.

Figure 1G

Tram customer satisfaction monitor results (phone survey), March 2010 to June 2016

Source: VAGO, based on information from Public Transport Victoria.

The tram franchisee is eligible for incentive payments and may incur penalties under the OPR and CEPR. Under the OPR, Yarra Trams was penalised $4.7 million in total for the period December 2009 to 30 June 2016. Under the CEPR, Yarra Trams received $2.2 million in incentive payments and has not incurred any penalties to date.

1.3 Patronage

Increasing numbers of passengers also affects the operational performance of trains and trams. When more passengers are on board a tram or train, it must stop for longer—known as dwell time—to allow passengers to board or exit.

From 2006 to 2008, the number of train passengers grew by 12 per cent annually, which had a negative impact on punctuality. Since 2009, the number of passengers has stabilised, which has helped to improve operational performance under MR3.

PTV forecasts that passenger numbers will grow by 3.3 per cent annually for trains and 3.8 per cent annually for trams, from 2015 to 2021. This increased patronage, and the major construction projects planned during this period, will present operational challenges for the franchisees.

1.4 Payments to the franchisees

Since 2009, the state has paid over $7.6 billion to the franchisees to run Victoria's train and tram networks—$5.4 billion to MTM and $2.2 billion to Yarra Trams, up to 30 June 2016. These payments cover a range of services and activities, summarised in Figure 1H and do not include the payments for rolling stock lease amounts made by the franchisees on behalf of the state.

Figure 1H

Summary of franchise payment types

Source: PTV.

The service fee for running the networks (base contract and maintenance, in Figure 1H) and a share of the revenue from metropolitan ticket sales make up the major part of payments to the franchisees. The franchisees carry the risk that their running costs might exceed the combined service fee and share of ticket revenue.

Until 2014, the state guaranteed ticket revenue due to uncertainties with the implementation of the myki ticketing system. From 2014 onwards, franchisees have been exposed to the risk of variable ticket revenue. This risk is mitigated through periodic resetting of franchisee forecasts of ticket revenue to align with actual revenue, and having a risk-sharing regime that sets an upper and lower limit on the amount of risk the franchisee would bear. Figures 1I and 1J include the actual ticket revenue and risk mitigation payments after January 2014. However, PTV does not include actual ticket revenue in its public reporting in Track Record.

Figure 1I

Payments to MTM, December 2009 to June 2016

Note: 2009–10 only includes the period December 2009 to June 2010.

Note: Excludes rolling stock lease payments.

Source: VAGO, based on information from PTV.

Figure 1J

Payments to Yarra Trams, December 2009 to June 2016

Note: 2009–10 only includes the period December 2009 to June 2010.

Note: Excludes rolling stock lease payments.

Source: VAGO, based on information from PTV.

Separate to these payments, franchisees can earn commercial revenue from advertising and renting out surplus land or buildings. Franchisees also receive payments for their involvement in projects beyond their project agreements with PTV—for example, the level crossing removal program. The scope of this audit did not cover these project agreements.

1.5 Managing train and tram assets

Victoria's public transport assets were valued at $29.8 billion at 30 June 2016, according to VicTrack's annual report for 2015–16. This estimate includes tracks, stations, bridges, buildings, structures, plant and equipment, signalling and communications, land and the Melbourne Underground Rail Loop.

Asset management involves managing all of the activities carried out throughout the life of the asset, including maintenance and renewal, to make sure that it delivers its full value and contributes to the service objective.

Managing Victoria's train and tram assets is complex because of the range of agencies involved—the assets are owned by VicTrack, leased to PTV, and then subleased to the franchisees. Figure 1K describes the roles and responsibilities of these agencies for maintaining and renewing Victoria's train and tram assets under MR3.

Figure 1K

Summary of maintenance and renewal responsibilities in MR3

Source: VAGO.

The train and tram network is a significant asset for the state, so it warrants rigorous oversight and management by PTV to make sure that the assets enable the delivery of effective and efficient public transport.

1.6 Extending train and tram franchise agreements

MR3 includes a provision for the current franchisees to negotiate exclusively with the state for a seven-year extension if they meet certain performance benchmarks.

PTV has determined that both MTM and Yarra Trams have met the performance benchmarks specified in the agreements, enabling them to negotiate a new franchise agreement, which will be known as MR4. If these negotiations fail, the state could extend the agreement at a fixed price for up to three years, while it undertakes a tender process for new franchisees.

In preparation for these MR4 negotiations, PTV has undertaken a project to examine the strengths and weaknesses of the current agreements and identify opportunities to improve value for money in future franchise agreements.

1.7 Agency responsibilities

Public Transport Victoria

PTV was established in April 2012. Its primary objective is to plan, coordinate, provide, and maintain a safe, punctual, reliable and clean public transport system. PTV oversees all aspects of the franchisees' day-to-day operation of the train and tram networks, including performance requirements.

Department of Economic Development, Jobs, Transport & Resources

The former Department of Transport, which initiated the MR3 franchise agreements, was disbanded in 2013 and its functions were distributed between the now Transport Division of the now Department of Economic Development, Jobs, Transport & Resources (DEDJTR) and PTV.

DEDJTR is responsible for the state's transport and infrastructure policy and planning. It is overseeing the negotiations of new franchise agreements, and the secretary of DEDJTR is chairing the primary governance body for this project.

In June 2016, the Victorian Government announced the creation of Transport for Victoria (TFV). This new agency will plan, coordinate and operate Victoria's transport systems and associated agencies.

TFV will sit within DEDJTR and have overarching responsibility for the transport portfolio. Under this arrangement, PTV will remain as the agency responsible for managing and administering the train and tram franchise agreements.

VicTrack

Created in 1997, VicTrack is a state-owned business operating under the Transport Integration Act 2010. It owns Victoria's railway land, infrastructure and assets. Through a subsidiary (the Rolling Stock Holdings group of companies), it also owns much of the state's rolling stock. VicTrack also has a role in providing telecommunications services to franchisees.

1.8 Previous VAGO audits

Since the 1990s, we have regularly scrutinised Melbourne's public transport operations. We have conducted audits examining various aspects of the train and tram system, including franchising arrangements, financial performance, asset management and operational performance. A summary of these audits and their findings is provided in Appendix B.

1.9 What this audit examined and how

We examined PTV's effectiveness in managing the performance of Melbourne's metropolitan train and tram franchisees, by assessing whether:

• PTV's management of the franchise agreements is delivering value for money
• PTV's strategic planning for the extension of the franchise agreements is rigorous and designed to produce value for money.

Achieving value for money in tram and train services involves consideration of a wide range of factors and is not only a financial question. We focused on contract management, oversight of franchisee operational performance, and asset management and whether they contributed to value for money.

The agencies included in this audit were:

• PTV
• DEDJTR
• VicTrack.

We conducted the audit in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

The total cost of the audit was $555 000.

1.10 Report structure

The report is structured as follows:

• Part 2 examines contract and performance management
• Part 3 examines how PTV oversees the management of the state's rail assets
• Part 4 examines how well PTV is preparing for negotiations with the franchisees for the next franchise agreements.

2.1 Conclusion

Weaknesses in the design of the performance regimes and in PTV's contract management mean that it is not extracting maximum value for money from its franchise agreements.

PTV is aware of the gaps in its contract management approach and has begun work to address these deficiencies.

PTV has also comprehensively reviewed the OPR and CEPR in preparation for future franchisee agreements. It has identified several improvements that should address current weaknesses if they can be successfully negotiated and implemented.

2.2 Contract management

An effective contract management system:

• clarifies roles and responsibilities for contract management and decision-making
• is managed by appropriately skilled and well-trained staff who understand the contract
• consistently applies the provisions of the contract, including payments and performance regimes
• incorporates ongoing reviews of the contract, adapts to changing circumstances and manages contract variations
• guides relationship management, dispute resolution and issue management
• outlines contingency planning and risk management.

The contract management plan for the train and tram franchise agreements was established in November 2009. It defines the overall approach, including contract objectives, roles and responsibilities, reporting and communication protocols, and governance requirements. The plan is supported by 36 contract administration guides that define how PTV will monitor, manage and enforce the obligations of the agreements.

We found that the contract management plan and guides were useful resources for day-to-day contract management activities, but there were two areas that needed to be strengthened:

• First, there was no provision in the plan for a rigorous, systematic review of the agreements during their lifetimes. Consequently, PTV has not undertaken such reviews and it has not systematically captured issues and risks as they arose, identified opportunities to improve contract management, or routinely tracked the benefits of the agreements and evaluated whether they are achieving their objectives.
• Second, PTV's record keeping and document control has not been adequate. This creates a risk that knowledge needed for effective contract management may be lost when key staff leave PTV.

In October 2015, PTV completed an internal audit of its contract management of the tram franchise agreement. PTV believes that the gaps identified could equally be applied to the management of the train franchise agreement. The gaps include:

• unclear roles and responsibilities for contract management
• no designated contract manager
• no systematic approach to identifying, mitigating and monitoring risks
• ad hoc quality assurance processes
• out-of-date contract management plans and associated documentation
• no strategy to capture the lessons learnt.

We found similar weaknesses in PTV's contract management practices, as did our May 2015 audit Tendering of Metropolitan Bus Contracts.

It is encouraging that PTV has acknowledged these weaknesses and plans to address them by developing an organisation-wide contract management framework, to be implemented by November 2017.

2.3 Operational performance regime

Weaknesses in the current OPR compromise PTV's ability to rigorously assess and drive franchisee performance.

A well-designed performance regime starts with clearly defined measures and targets that are relevant to the franchise agreement's stated aims. It is important that these measures and targets can reliably assess actual performance. The performance regime also needs to be cost effective to administer and should create positive incentives for operators to continually improve their performance.

When a performance regime has been effectively implemented, it produces complete and accurate information about the franchisee's performance. PTV can then use this information to identify and manage underperformance and to inform decisions about investment in, and the design of, future agreements.

Because the performance regimes for Victoria's train and tram system are embedded in the franchise arrangements, their ability to effectively measure the franchisee's performance depends on how well PTV manages the franchise agreements. Appendix C provides further detail on the contractual framework for the OPR.

2.3.1 Reliability of performance data

Trains

Until March 2015, PTV relied on manually recorded and self-reported reliability and punctuality data from MTM. This information was used to determine bonus payments and penalties under the OPR and to inform publicly reported performance results.

Manual recording of this information has obvious and inherent risks—such as franchisee error and bias—which affect the reliability of the data. PTV managed this risk by conducting regular field surveys to assess the accuracy of MTM's reporting. From 2011–12 to 2014–15, PTV's surveys identified instances of under-reporting of late trains. This led to PTV issuing approximately $2 million in penalties to MTM.

Station skipping

Although PTV's field surveys successfully detected under-reporting of trains arriving late at their destination, they did not detect instances of trains skipping stations and did not monitor train movements throughout the course of an entire trip. Train operators may decide to skip stations to recover time from a previous disruption to avoid penalties imposed when trains arrive late at their destination. In March 2013, PTV asked franchisees to begin self-reporting discretionary station skipping to help address this problem.

MTM's self-reporting of trains skipping stations excludes instances when a train was forced to skip a station to make up time for reasons beyond its control—for example, a delay caused by a passenger requiring medical attention. The publicly reported figures on station skipping—labelled as 'unplanned expresses' in the reports—show a significant reduction in the practice from June 2014 to June 2016.

PTV relies on honest and accurate self-reporting of station skipping by MTM. PTV also relies on franchisees to report the causes of service delays accurately and in good faith, as these reports form the basis for bonus and penalty payments under the OPR.

Automated Performance Reporting System

In March 2015 PTV replaced the manual data recording system for trains with an automatic train monitoring system, the Performance Reporting System (PRS). The PRS reduces the risk of errors that could arise from manual recording, significantly increases the number of monitoring points on the network, and allows for more sophisticated data analysis. However, the PRS still does not provide comprehensive train journey information, because it does not measure train punctuality at all stations. PTV reports that it is not feasible for the PRS to be implemented at all stations, due to technical limitations and costs.

PTV carried out field surveys in June 2015 to test the accuracy of the PRS. These surveys did not identify any significant discrepancies between manually collected data and the data generated by the PRS. However, PTV has not done any further audits or reviews of the PRS and, therefore, has limited assurance about the ongoing integrity of the information generated by this system.

Trams

Data about tram performance is recorded in the Automatic Vehicle Monitoring (AVM) system. This state-owned system records tram journey information and is operated by Yarra Trams.

Each tram route has three to five monitoring points to identify early departures, late arrivals, short runs or cancellations. When a tram passes a monitoring point, information is captured in a central database and sent to PTV so that it can analyse tram services' punctuality and reliability.

PTV acknowledges that the monitoring sites along each route do not provide enough data to accurately assess service performance for the entire length of a journey.

PTV also acknowledges that there are longstanding problems and risks that affect the reliability of the AVM system. It uses technology from 1987, and the risks and limitations of the technology may affect safety, operational performance and passenger information. Our June 2014 report Using ICT to Improve Traffic Management and our August 2014 report Coordinating Public Transport found that the system was obsolete and unable to determine the precise position of each tram.

PTV has completed a project to extend the life of the AVM until 2020 and has recently begun another project to identify potential replacement options. PTV acknowledges that the life extension of the AVM is an interim measure, and that there are problems and risks with the current system. Despite these risks, PTV relies solely on Yarra Trams to assure the ongoing integrity of the system and the data it produces. This means that PTV is publicly reporting performance results and determining bonus and penalty payments with only limited assurance about the reliability of the performance data that underpins these results.

2.3.2 Public reporting of operational performance

PTV publishes daily results of train and tram reliability and punctuality on its website. It collates these results and other operational information into monthly and quarterly reports titled Track Record, which provide the public with an indication of train and tram performance. Appendix C provides further detail on the percentage‑based measures used to produce the data in Track Record and how they differ from the operational performance regime.

There are several problems with how the agreements were designed to measure performance and how PTV reports on performance:

• Lack of performance targets—there are no targets for the reliability and punctuality of trains and trams, but this is not made clear in the way results are reported. Track Record reports the minimum performance standards, known as thresholds, mandated in the franchise agreements. If the franchisee's performance falls below this threshold, the agreements state that a 'call-in' is triggered. A 'call-in' event requires the franchisee to explain their poor performance and how it will be addressed. This means that the thresholds represent the minimum acceptable performance for franchisees rather than performance targets. They are based on the average performance of the previous franchisee for the period May 2008 to June 2009 minus two percentage points for punctuality and one percentage point for reliability.
• Fixed thresholds—the franchise agreements do not allow for periodic resetting of the reliability and punctuality thresholds, to reflect service or network improvements or actual performance.
• Unavoidable service delays—the publicly reported reliability and punctuality figures take into account events that are outside the control of the franchisees but these events are excluded from the OPR.
• Bonus and penalty payments not aligned to performance results—the publicly reported reliability and punctuality figures are not directly connected to the bonus and penalty payments that franchisees receive under the OPR, but this is not made clear in Track Record. For example, there are instances where the franchisee performs above the threshold, but is still penalised under the OPR because it has failed to meet performance targets. This means that the figures in Track Record do not give the public a sound understanding of franchisee performance.

PTV's review of public reporting in preparation for negotiating the metropolitan train and tram franchise agreements from 2017 onwards (MR4) has identified the need for several key improvements. It is important that public reporting of targets, thresholds and payments is simple and transparent.

2.4 Customer experience performance regime

Under the franchise agreements, the CEPR measures customer experience with train and tram services. This regime, linked to bonus and penalty payments, was first introduced in MR3 franchise agreements.

The CEPR includes ten components, detailed in Figure 2A, monitored by PTV (and previously by the former Department of Transport) through quarterly network inspections and customer satisfaction phone surveys.

Figure 2A

Summary of CEPR components

Source: VAGO, based on information provided by PTV.

Until July 2012, PTV used the quarterly network inspections and phone survey results to assess franchisees' annual performance against the thresholds in the CEPR. Franchisees received a bonus payment when performance for each component exceeded an upper threshold, and incurred a financial penalty when performance was below the lower threshold. The annual payments were capped at $500 000 for trams and $1 million for trains.

In July 2012, PTV abandoned the CEPR components that required network inspections. PTV determined that the CEPR did not provide value for money because:

• the cost of physical inspections was high
• the threshold range was too wide, making it hard to perform below the threshold, but also difficult to exceed it in many areas, which discouraged improvement
• the contracts lacked a provision to reset the benchmarks as more performance data was gathered
• there were some unintended outcomes—for example, during the three years the CEPR was in place:
• Yarra Trams received more than $1 million for exceeding set thresholds even though its performance declined in some of the CEPR components
• MTM received over $0.7 million in incentive payments, even though it did not consistently exceed the thresholds
• the incentives and penalties were relatively small given the investment required to improve some areas, such as the condition of rolling stock
• the former Department of Transport did not proactively share the CEPR performance data with the franchisees to help them improve their performance.

From July 2012, PTV continued incentive payments and penalties based solely on the results of quarterly customer satisfaction surveys. It reviewed past survey results to develop upper and lower thresholds for the incentive payments and penalties, and discounted factors that were beyond the franchisees' control but which could negatively impact on results, such as rises in ticket prices.

Although PTV's decision to abandon network inspections was reasonable, surveys only give a very subjective assessment of how well the franchisees have been fulfilling their customer service obligations. Since July 2012, the train and tram franchisees have received incentive payments of $0.6 million and $1.1 million respectively, and neither has been penalised.

This lack of objective performance data will continue at least until the current agreement ends in November 2017.

2.4.1 Customer service standards

As well as meeting the requirements of the CEPR, franchisees are also expected to meet agreed customer service standards under the franchise agreements. We found that PTV has not held the franchisees to account for their compliance with these standards.

The MR3 agreements require that the franchisees 'make reasonable endeavours' to comply with the standards, which include cleanliness, graffiti removal, provision of information, staffing and fault rectification times. The franchisees are required to submit annual system upkeep plans, which outline how they will comply, and PTV has the right to audit franchisees' compliance with the plans. There are no financial penalties or incentives attached to the upkeep plans, but failing to comply triggers a warning and a requirement for franchisees to explain their performance and how problems will be addressed.

In preparing for MR4, PTV has identified that full compliance with the current standards has not always been possible because of their ambiguity. For example:

• The train franchisee is required to remove internal and external graffiti from rolling stock within 24 hours of notification or, where not reported, within 72 hours of occurrence. However, because specific record-keeping practices and inspection regimes are not mandated in the franchise agreement, it is difficult to define when the graffiti 'occurred' or how long it was visible to customers.
• The train franchisee is required to wash the exterior of rolling stock once every 14days, but PTV and the franchisee have recently agreed that this target was never achievable.

PTV has not proactively reviewed system upkeep plans, nor has it completed any audits or inspections of franchisee compliance with the customer service standards since July 2012. Instead, PTV has relied on customer feedback provided through surveys, call centre and website enquiries, and 'mystery shopper' programs operated by the franchisees. As a result, PTV has not objectively assessed franchisees' compliance with their customer service obligations.

2.4.2 Revised customer experience regime

In 2014, after consulting customers and franchisees, and completing other research, PTV prepared a revised set of customer experience standards that sought to capture the end-to-end customer experience. The standards include achievable, desirable and aspirational standards based on current performance.

PTV has recognised that capital investment is required to achieve some of these standards, and that effective and efficient measurement of performance against every standard is not achievable. PTV plans to use these standards, together with the lessons learnt from the CEPR and the current franchise agreement provisions, to develop a revised CEPR to be included in the MR4 franchise agreements. The success of this new regime will depend on the forthcoming negotiations and PTV's implementation of the new agreements.

3.1 Conclusion

PTV does not have enough assurance that train and tram assets are performing at their optimum level and delivering value for Victoria.

There are longstanding gaps and deficiencies in the way PTV and former responsible agencies have managed the state's train and tram network assets. Deficiencies in PTV's knowledge of the condition of train and tram assets hamper its efforts to make sure that the franchisees are effectively managing the assets. This compromises PTV's planning of maintenance and renewal works and asset funding.

PTV and the Department of Economic Development, Jobs, Transport & Resources (DEDJTR) are now addressing the absence of long-term asset management strategies. In preparing for the next franchise agreements, known as MR4, PTV has acknowledged the inadequate attention it has given to asset management over the course of the current franchise agreements and has started to address these longstanding deficiencies.

3.2 Strategic asset management

Effective maintenance and renewal practices are integral to strategic, whole-of-life asset management to ensure the assets deliver their full service potential.

We found that PTV has not been managing the state's train and tram assets strategically. PTV does not have a comprehensive understanding of the condition of its assets. Therefore PTV has limited capacity to make evidence-based decisions on how it funds and prioritises maintenance and renewal work and limited ability to develop sound long-term asset management strategies.

3.2.1 Asset management strategies

Since the mid-1990s, the Department of Treasury & Finance (DTF) has set clear expectations for Victorian public sector agencies to have a whole-of-life approach to asset management that focuses on the services the assets need to deliver.

Asset management strategies are critical features of this approach—they set out how assets will help to deliver service objectives and include planning for asset maintenance, upgrades, acquisitions and disposals.

A sound asset management strategy requires a robust understanding of the base condition of the existing assets and their capacity, capability and usage. The strategy should take into account available resources, funding constraints, and competing service and asset priorities.

In our 2007 report Managing Victoria's Rail Infrastructure Assets, we recommended that the former Department of Infrastructure develop a long-term asset management strategy for metropolitan rail infrastructure, to help identify and prioritise the maintenance and renewal of these assets. The department did not address this recommendation. Consequently, there was no long-term asset management strategy in place for the train and tram networks at the beginning of the current franchise agreements—known as MR3—in 2009.

PTV acknowledged this gap and began work to address it in November 2015. It published a strategic asset management policy and guide, which outlines the need for medium- and long-term asset management strategies to ensure that train and tram assets deliver the required services. PTV has also committed resources to planning its asset management.

Although these are positive steps, PTV has significant work to do before it can show that it has robust asset management strategies in place and can comply with DTF's 2016 Asset Management Accountability Framework. This framework requires PTV and other public sector agencies to attest to their compliance with a range of mandatory requirements, including the development of asset management strategies, by 30 June 2018.

DEDJTR has also started developing a broad transport asset management program, including a draft transport asset management policy. This will guide all transport agencies as they develop strategies for specific modes of transport, to ensure they align with DEDJTR's draft Network Development Strategy, which is currently in development.

3.2.2 Asset condition information

To fully understand the condition of assets, agencies need to continually collect and analyse information throughout the asset's life cycle. This information can come from:

• inspections and audits
• key performance indicators for operational performance, safety and environment
• customer feedback
• projects undertaken on assets and infrastructure
• maintenance and renewal activities
• monitoring of failures and faults
• remote monitoring of assets
• asset life cycle costing.

PTV collects a large amount of information on Victoria's train and tram assets from the franchisees in shared databases, as required by the franchise agreements. PTV also meets regularly with the franchisees to specifically discuss maintenance and renewal of assets.

Figure 3A summarises the main maintenance and renewal reporting that franchisees provide to PTV and any issues we identified with each report.

Figure 3A

Maintenance and renewal reporting requirements

Source: VAGO, based on information from PTV.

When preparing for the MR4 franchise agreements, PTV identified weaknesses in the information it collects. These include:

• information being reported in different formats
• asset information being held in several databases that can be accessed by the franchisees, PTV and VicTrack, but which are not fully integrated
• franchisees not fully complying with the requirement to update PTV's asset management systems within one month of completing maintenance and renewal works
• some KPIs missing targets or thresholds to measure against—some were difficult to measure and the KPIs were skewed towards measuring asset reliability, in terms of how often they fail, rather than measuring asset condition
• the agreements not adequately addressing information needs on other longer‑term network-wide asset requirements, such as bridges and stations
• difficulties in combining asset information from multiple sources and formats to provide a single picture of trends, issues and risks.

These issues created gaps in PTV's knowledge of the overall health and condition of the assets.

PTV has recently started to address these gaps by planning an asset condition assessment, which is subject to funding approval. PTV plans to document the baseline condition of assets through a further comprehensive survey, which will need to be completed periodically throughout MR4.

VicTrack asset information

As owner of the state's rail assets and infrastructure, VicTrack has an obligation to know the condition and value of these assets in order to comply with applicable accounting standards. Under DTF's new Asset Management Accountability Framework, VicTrack's obligations for asset management reporting will increase.

Although PTV has provided VicTrack with the information required to comply with their accounting and financial reporting obligations, PTV and VicTrack have not worked collaboratively to share more detailed information about assets, and VicTrack was not involved in the asset management meetings between PTV and the franchisees.

For this process to be successful, VicTrack needs to be involved in the strategic asset management planning. The Public Transport Asset Framework Steering Committee was recently established, and VicTrack's inclusion is a positive step. This committee aims to oversee and approve the development and implementation of an asset management framework for Victoria's public transport assets.

3.2.3 Network condition

The limited information available about Melbourne's train and tram networks indicates that the condition of these assets has deteriorated since 2009.

Tram

PTV collects information twice a year on the condition of the tram network by running specially equipped test trams across the network, which collect data on track condition as they go. Although this information is limited, it shows that the condition of Melbourne's tram network deteriorated between 2012 and 2016.

PTV has attributed the degradation in the condition of tram tracks to a range of factors, primarily the recent increase in the use of heavier trams with lower floors. As the use of this type of tram is likely to increase in the future, PTV will need to address this challenge when managing tram network assets.

Train

In May 2015, the Office of the National Rail Safety Regulator (ONRSR) reported that Victoria's metropolitan and regional rail infrastructure and rolling stock (for the train network only) was generally fit for purpose, but much of it had deteriorated and was in poor condition. The ONRSR also reported that the track infrastructure in Melbourne was significantly below the standard of track in equivalent interstate networks and, although this did not present any immediate safety risks, PTV's medium- to long-term management of asset-related risks was inadequate.

In January 2016, PTV responded to this report by committing to systematically apply a more comprehensive approach to managing train assets. To achieve this, PTV plans to implement a range of initiatives, including:

• undertaking asset renewal projects
• developing medium- and long-term asset management strategies
• aligning its asset management systems with international standards
• using the MR4 refranchising process to improve the asset management framework included in the agreements.

Adequacy of maintenance and renewal funding under MR3

Between the previous franchise agreement and the current MR3 agreement, annual funding for maintenance of tram network assets and renewal works has been relatively constant.

Under the MR3 agreement, annual funding for work on train network assets increased by around $42 million. PTV has reviewed the adequacy of funding levels for asset maintenance and renewal works on the train and tram networks and found that it cannot accurately determine whether current funding is sufficient, partly because it lacks comprehensive information on the condition of the assets.

PTV's benchmarking against other jurisdictions suggests that its funding for asset maintenance and renewal as a percentage of asset renewal costs is relatively low. However, equivalent networks were difficult to identify—particularly for trams—and the results were therefore of limited use in determining required funding.

Given the importance of train and tram assets, PTV's inability to demonstrate a comprehensive understanding of required funding levels is significant.

3.3 PTV's oversight of asset maintenance and renewal

PTV pays franchisees for maintenance and renewal work that they complete. PTV does some limited checking of this work but does not analyse the information provided or how effectively franchisees are tracking and managing issues with assets.

As a result, PTV has limited assurance that franchisees' maintenance and renewal work is keeping train and tram assets in optimal working order throughout their life.

3.3.1 Payments

During the current agreements, up to June 2016, the state had paid $1.96 billion to the franchisees to cover the cost of maintaining and renewing train and tram assets—$1.6 billion to MTM and $362 million to Yarra Trams.

The franchisees submit monthly invoices to PTV for the maintenance and renewal work they have undertaken, along with activity reports and other evidence to support their invoices. The franchise agreements prescribe the format and content of these activity reports.

The franchisees have not consistently provided the required information about asset maintenance and renewal on time—in some cases, they were over a year late in providing this information to PTV. PTV withheld payments in these instances, but the delays in providing information have contributed to PTV's lack of knowledge about the actual condition of rail network assets.

3.3.2 Quality assurance over maintenance and renewal work

PTV gains assurance on the quality of franchisees' maintenance and renewal works by:

• conducting risk-based spot audits of a sample of completed works, to check that the quality and quantity of the completed work matches the information the franchisee has provided
• relying on franchisees' compliance with their own quality assurance processes and the technical plans, and their compliance with safety regulations applicable to network assets.

We found that PTV's checks of the quantity of maintenance and renewal work were adequate, but that it rarely checked the quality of these works because some assets could not be accessed and checked without interfering with the operation of the train and tram network, and others could not be accessed at all. We also found that PTV has not analysed the information and results from these activities to identify trends and issues.

3.3.3 Managing maintenance and renewal issues

PTV has not effectively managed maintenance and renewal issues throughout the current agreements.

Over the course of the MR3 agreements, franchisees have submitted reports about maintenance and renewal work to PTV, which they discussed together at monthly meetings. However, until recently these reports and meetings primarily covered operational issues and progress with maintenance and renewal works.

PTV did not adequately track and resolve maintenance and renewal issues at the meetings, and PTV has not sufficiently analysed maintenance and renewal trends or issues to inform its strategic asset maintenance and renewal planning.

Figure 3B is a case study about the management of operational control and management systems (OCMS) under the MR3 franchise agreements. It highlights the challenges of resolving issues when roles and responsibilities have not been clearly defined.

Figure 3B

Case study: Managing OCMS—black diamond switches

Source: VAGO, based on information from PTV and VicTrack.

While preparing for negotiating the MR4 franchise agreement, PTV has identified opportunities to improve the way OCMS are managed in the next agreement by:

• developing a comprehensive asset strategy
• seeking a significant funding increase
• reviewing the contractual arrangements
• increasing the expertise within PTV
• identifying clearly defined roles and responsibilities for each party—PTV, VicTrack and the franchisees.

4.1 Conclusion

PTV could not provide evidence to support the rationale and assessment process for some of the performance benchmarks that led to the franchisees earning the right to exclusively negotiate an extension of the franchise agreements. Given the commercial value of such a right, PTV needs to carefully scrutinise the way franchisees have been assessed against the performance benchmarks so that lessons learnt are applied when negotiating future arrangements.

It is encouraging to see that PTV's planning for the negotiations with franchisees has been thorough. PTV has developed an in-depth understanding of the strengths and weaknesses of the current agreements and has identified important, achievable opportunities for improvement.

4.2 The right to negotiate a franchise extension

Under the train and tram franchise agreements, PTV (and, at the time, the former Department of Transport) set benchmarks annually and advised each franchisee of whether they had been met.

However, PTV has not been able to provide sufficient evidence to demonstrate how it or the former Department of Transport determined some of these benchmarks, nor could it demonstrate how franchisees' performance was measured against them.

4.2.1 Contractual arrangements for franchise extension

To earn the right to negotiate a contract extension, franchisees must meet both fixed and flexible benchmarks, and comply with contract and safety requirements. Figure 4A summarises these benchmarks and contractual arrangements.

Figure 4A

Contractual arrangements for franchise extension

Source: VAGO.

If PTV successfully negotiates further seven-year contracts with the existing franchisees, this will bring to an end the current 15-year arrangements. There is no further performance-based right to extend them. PTV must then open the tendering process to other prospective operators.

The MR4 project does not include an evaluation of whether a performance-based right to extend these types of agreements is an effective way to improve franchisees' performance. Without evaluating the effect of this right, PTV will not have a full understanding of whether this approach is useful and whether it should be included in future agreements.

4.2.2 Annual review process and results

The current agreements include a six-year assessment period to determine whether franchisees met the criteria to negotiate a contract extension. At the end of this period, PTV assessed the overall performance of both franchisees and determined that they had met the benchmarks specified in the MR3 agreements. Figures 4B and 4C show the outcome of this assessment.

Figure 4B

MTM's performance against extension benchmarks

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: ✔ achieved, ✘ not achieved.

Source: VAGO, based on information provided by PTV.

Figure 4C

Yarra Trams' performance against extension benchmarks

(a) For seven months from 30 November 2009 to 30 June 2010.

Note: ✔ achieved, ✘ not achieved.

Source: VAGO, based on information provided by PTV.

On 31 December 2015, PTV informed franchisees of the outcome of its assessment. The state was then obliged to enter into good-faith negotiations with the franchisees, which began in October 2016.

Fixed benchmarks

In 2009, at the beginning of the current agreements, the former Department of Transport determined the two annual fixed performance benchmarks—service reliability and customer experience—based on the performance of the previous franchisees in these areas during the previous year.

The current agreements do not allow for periodic review and resetting of the benchmarks. As a result, PTV has had only a limited ability to use these benchmarks to drive continuous improvement.

Reliability benchmark

The first fixed performance benchmark was a reliability target—that 98.5 per cent of daily services listed in an agreed timetable be delivered. This target was based on the minimum service thresholds explained in Section 2.3 of this report. The thresholds included a provision for PTV to request a meeting and to seek an explanation from the franchisee if their performance fell below a 'call-in' threshold of 98 per cent.

This threshold was determined using data from the 12 months before the commencement of the MR3 franchise agreements. It was based on the franchisees' average performance over this period minus one percentage point. This meant that the performance target of 98.5 per cent was in fact half a percentage point below the average performance achieved by the previous franchisee. As a result, this benchmark was easier for franchisees to achieve and did not act as an incentive for franchisees to improve their performance.

Customer experience benchmark

The second fixed benchmark was that franchisees should not incur a financial penalty under the customer experience performance regime (CEPR). The CEPR was largely abandoned in 2012 due to a range of problems—see Section 2.4 for further discussion.

From 2012, PTV measured franchisees' performance against this benchmark using the results of the quarterly phone survey. PTV's evaluation excluded survey results related to factors that were beyond the franchisees' control, such as ticket prices.

PTV's decision to continue with incentive payments and penalties using subjective phone surveys meant that achieving this benchmark required significantly less effort by each franchisee.

Flexible benchmarks

Flexible benchmarks were intended to provide PTV with the flexibility to focus franchisees on key problem areas or particular initiatives.

Under the franchise agreements, the flexible benchmarks must:

• reflect the franchisees' role in meeting the passenger-related objectives included in Victoria's public transport policy during the franchise period
• be reasonably achievable in the relevant assessment year without requiring any additional capital investment
• be developed in consultation with the franchisee
• must specify a reasonable way for the franchisee to demonstrate compliance if objective measurement is not possible.

Figure 4D provides some examples of flexible benchmarks for trains and trams.

Figure 4D

Flexible benchmarks for train and tram franchise

Source: VAGO, based on information provided by PTV.

PTV developed the flexible benchmarks and assessed franchisees' performance as part of its day-to-day contract management activities. The process was not subject to formal consultation and assurance processes, and governance was limited to PTV's chief executive officer endorsing the benchmarks and associated outcomes.

Correspondence about the flexible benchmark process shows that it did occur within the required time frames. However, PTV could not provide evidence for the rationale behind some of the benchmarks or how performance had been assessed against them. For example, for three of the six assessment years, PTV could not provide evidence that explained why flexible benchmarks were chosen for MTM or give sufficient detail about how it had assessed MTM's performance.

We found that PTV's consultation and communication with the franchisees about setting and assessing the performance benchmarks was either conducted via email or was not recorded. Given the important role of the performance benchmarks in the franchise agreements, it is surprising that there is no documentation to show how they were established and assessed. It is also contrary to the principles of evidenced-based decision-making.

Despite these shortcomings, there is evidence that some of the flexible benchmarks helped PTV to motivate franchisees to improve their performance in particular areas. The case study in Figure 4E provides an example of a flexible benchmark that resulted in network-wide benefits.

Figure 4E

Case study: Flexible benchmarks in the tram franchise, 2014–2015

Source: VAGO, based on information provided by PTV.

4.3 Preparing for the next agreements

PTV's project to prepare for the MR4 franchise agreements has been thorough and gives a sound basis for achieving value for money from the new agreements. However, achieving value for money from these agreements depends on successful negotiation and effective implementation and management of any future franchise arrangements.

4.3.1 Value for money

PTV has a detailed understanding of the financial and non-financial factors that affect the value for money of the current MR3 agreements. It has identified important improvements for the MR4 agreements.

Understanding network operating costs

PTV's analysis of the MR3 agreement found that operating costs increased over the course of the current agreements—particularly for the train network—due to:

• increases in wages of franchisee staff and the introduction of new roles to support service performance
• increases in the number of kilometres travelled by trains and trams
• increases in the cost of infrastructure compared to the MR2 agreements
• the operation of additional premium and host stations, line extensions, and maintenance of additional facilities on the train network.

Some of these increased costs are funded by the state—such as the costs associated with additional train services—but others have been absorbed by the franchisees.

PTV expects that operating costs will continue to increase throughout the MR4 franchise agreements, largely due to emerging cost pressures, such as:

• wage increases agreed in recent industrial agreements
• the cost of maintaining and renewing ageing train and tram fleets
• increased staffing levels to improve service delivery—for example, the introduction of driver managers to monitor and support driver performance
• uncertainty about how much funding will be required to maintain and renew assets
• the need to renew operational control systems
• uncertainty about ticket revenue because of significant future transport projects, such as Melbourne Metro Rail.

PTV expects the operating environment to become more challenging during the course of the MR4 franchise agreements—the state will implement a significant transport infrastructure program that may result in disruptions to train and tram services. PTV also estimates that patronage will continue to increase by 3 to 4 per cent annually.

Efficient Operator Comparator

PTV has developed an Efficient Operator Comparator (EOC) to help it demonstrate that the MR4 agreements will deliver value for money. The EOC compares operating costs estimated by the franchisees against those of comparable efficient rail operators. The EOC is an important tool—PTV has recognised that the lack of competitive tension in the negotiation process has increased the need to establish value for money.

PTV's development of the EOC has been guided by Partnerships Victoria's guidelines for public sector comparison tools. Although the EOC uses a different model, it shares some fundamental similarities and principles with Partnerships Victoria's framework. We found that PTV used this framework appropriately, modifying it where needed to suit its particular circumstances.

To understand franchisees' operating costs, PTV worked with MTM and Yarra Trams and used a transparent and collaborative 'open book' approach. PTV identified weaknesses in the way it has collected financial information from franchisees under the MR3 agreements. PTV also identified opportunities to improve franchisees' financial reporting—for example, by requiring more comprehensive information from franchisees and making sure that it has the systems and capability to analyse and understand trends.

Allocating and managing risks appropriately throughout the life of the agreement is fundamental to achieving value for money from the agreements. PTV and the franchisees need to be clear about who bears the cost if a risk to the operation of train and tram services eventuates—for example, if ticket revenue does not increase as predicted or if a new timetable is delayed.

In our 2005 audit Franchising Melbourne's Train and Tram System, we recommended that the former Department of Infrastructure clearly outline which risks were to be retained by the state and which were to be transferred to the franchisee. We found that PTV has addressed this recommendation. It engaged external contractors to assess the rigour of the EOC, and this continuing work will result in ongoing adjustments to the EOC.

We found that the EOC documents demonstrated detailed analysis of past performance, franchisee costs, environmental context and an understanding of the causes of any increasing or decreasing cost. This work—completed by PTV's subject matter experts, project leads and DEDJTR—has given PTV greater assurance that its assumptions about operating costs are well informed.

Identifying strengths and weaknesses in the franchise agreements

PTV's MR4 project was divided into streams that created clear accountability and use of subject-matter expertise in areas of commercial significance. After consultation and research, PTV developed position papers that were endorsed at the relevant governance forums.

The position papers include detailed background, identify weaknesses in the MR3 agreements and provide options to address them. PTV has appropriately benchmarked franchisees' performance against past data and the performance of other national and international networks.

PTV has identified a range of ways that value for money can be improved in the MR4 agreements, such as:

• measuring franchisees' operational performance at a greater number of monitoring points, allowing PTV to address and penalise undesirable practices such as skipping stations
• setting higher reliability and punctuality thresholds than those set for the MR3 agreements
• focusing on overall customer experience, including better fleet and station presentation, customer information and assistance, achievement of minimum standards and a renewed incentive regime
• implementing a more strategic approach to maintaining, renewing and managing assets to improve understanding and monitoring of asset condition and the associated risks
• reducing the risks associated with the current disparate operational control and management systems
• facilitating major network changes (for example, the Melbourne Metro Tunnel) to improve the operation of the broader network
• developing more restrictive standards for advertising to align with community expectations.

Some examples of strengths, weaknesses and opportunities for improvement are summarised in Figure 4F.

Figure 4F

Priority issues for the MR4 agreements

Source: VAGO, based on information provided by PTV.

PTV's ability to improve train and tram services will depend on the success of its contract negotiations and its implementation of the new agreements. PTV's new contract management framework, described in Part 2, includes periodic reviews of the MR4 agreements throughout their life span to determine whether the agreements are meeting their objectives and providing the anticipated benefits. This will help PTV determine any changes needed to the MR4 agreements and prepare for future agreements.

4.3.2 MR4 project management

PTV began planning for the MR4 franchise negotiations in early 2015, with a budget of $25.7 million, plus contingency, over three years. To manage this project effectively, PTV requires considerable expertise, rigorous planning and clearly defined objectives, scope and governance structure.

The MR4 franchise negotiation is not a standard procurement process because it involves exclusive negotiations, so it does not fit within the policies and guidelines of the Victorian Government Procurement Board or Partnerships Victoria. It is also not subject to the Department of Treasury & Finance's (DTF) High Value High Risk process.

Nevertheless, the principles of effective procurement and project management still apply, as does PTV's Investment Lifecycle Framework process, which is consistent with DTF's Investment Lifecycle Framework.

We found that PTV's project team has developed a detailed strategic procurement plan, which clearly outlines objectives, time frames and a structure for the project's governance. Further details on the governance structure and roles and responsibilities of key groups are provided in Appendix D.

One gap we identified in the governance structure was the exclusion of VicTrack, the owner of the state's rail assets and infrastructure. DEDJTR explained that VicTrack was not included in the governance structure because the meetings covered broad policy matters that were not related to assets. However, because rail assets and infrastructure play an important role in the functioning of the network, VicTrack should have been formally included in the MR4 project governance structure. DEDJTR has partly addressed this gap by establishing the Public Transport Asset Management Steering Committee, which includes representatives from VicTrack.

PTV has adequately considered probity requirements by engaging a probity adviser, and it has also committed to hiring an independent probity auditor.

We also found PTV has developed an appropriate stakeholder engagement and communication plan and kept a detailed stakeholder engagement record. The governance bodies and workshops have facilitated effective consultation within PTV and between PTV and other departments and external stakeholders. This stakeholder engagement has helped PTV to develop a detailed understanding of the strengths and weaknesses of the MR3 agreements and identify opportunities for improvement.

RESPONSE provided by the Secretary, Department of Premier & Cabinet

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria – continued