Operational Effectiveness of the myki Ticketing System

Background

In July 2005, the state entered into a contract to develop myki, a smartcard public transport ticketing system to replace the ageing Metcard system.

Just over half of myki's almost $1 billion initial budget—around $520 million—was for establishing the system by 2007 and operating it for 10 years. The remainder was for operating Metcard in parallel with myki during the transition, and to cover the former Transport Ticketing Authority's (TTA) related costs for managing Metcard and its replacement.

Metcard was switched off in December 2012 following the completion of myki's rollout across metropolitan services. Public Transport Victoria (PTV) has since assumed responsibility for all of TTA's related functions and finalised myki's rollout across selected regional services in February 2014.

It was expected that myki would deliver around $6.3–$10.8 million per year in economic benefits to the state and its objectives included:

• enhancing the community's image of public transport
• providing the best value-for-money solution at the lowest whole-of-life cost
• being operational at, or shortly after, Metcard's expiry in 2007.

However, there have been significant implementation issues with the system. Specifically, the time taken to design and deliver it more than quadrupled from the original expectation of just two years, to in excess of nine years. This has led to significant unanticipated additional costs—resulting in a $550 million, or 55 per cent, increase on the project's original budget.

The operational performance of myki has also attracted significant complaints and criticism from users. In particular, overcharging has been, by far, the single most common complaint from users about myki since its rollout.

In response to these challenges, the former government commissioned a review of myki in late 2010 to determine its future. It decided to proceed with implementing myki, but with modifications. The resulting changes substantially reduced myki's initial scope, further extended its delivery time frames, and fundamentally changed the way the state pays for the system.

The current myki contract is due to expire in June 2016, with an option to extend it for a further six months to December 2016. PTV is currently planning the myki retender, which it expects to complete by November 2016.

Objectives of the audit

The audit examined the operational effectiveness of the myki ticketing system. It assessed whether the arrangements in place for the system's implementation and ongoing management are effective and whether the expected outcomes and benefits from the introduction of myki for users, operators and the state have been, or are on track to being, achieved.

Conclusions

The implementation of myki experienced significant delays and related cost increases, due largely to deficiencies in the original contract and governance arrangements.

This has resulted in a poor outcome for Victoria's public transport system and users, which has compromised achievement of myki's original business case objectives and related benefits.

The overly ambitious and vaguely specified initial project scope and contract for myki contributed to this outcome.

These deficiencies meant the state was not in a position to assure it was paying for a ticketing system that met clearly articulated and agreed performance standards—particularly during the initial build and rollout.

Shortcomings in both cross-agency coordination and oversight of the initial project conspired with these issues to produce significant delivery risks that were not well managed. Collectively, these challenges undermined the viability of the original contract.

PTV has since improved its oversight and management of the myki contractor by strengthening related governance and contractual arrangements.

However, significant risks to the state remain. Specifically:

• PTV does not yet possess a complete and reliable picture of myki's operational performance due to weaknesses with the contract's new performance regime and its implementation
• the benefits and outcomes sought from the myki retender have not been clearly defined, which reduces transparency and accountability for myki's future performance
• compressed time frames for the myki retender, caused by previous delays and the imminent expiration of the current contract, risk exposing the state to significant additional costs.

PTV needs to urgently address these issues to avoid perpetuating previous mistakes, and to ensure that the state can maximise value from the future operation of myki.

Findings

Governance and contractual arrangements

The original myki governance arrangements were previously examined by:

• the Victorian Ombudsman's 2011 Own motion investigation into ICT-enabled projects jointly conducted with VAGO
• the Public Accounts and Estimates Committee's 2012 Inquiry into Effective Decision Making for the Successful Delivery Of Significant Infrastructure Projects
• related internal reviews undertaken by the Department of Treasury and Finance in 2011 and 2014.

These reviews consistently found that the roles and responsibilities of key agencies initially charged with myki's development were neither well defined nor effectively implemented. This led to uncertainty and delays in decision-making, particularly during myki's initial build and rollout.

Critical weaknesses highlighted by these reviews include:

• a lack of clarity regarding whether the former Department of Transport (DOT) or TTA was the ultimate decision-maker on myki matters
• ineffective communication between DOT and TTA impeded TTA's implementation of DOT's ticketing policies and TTA's capacity to meet prescribed deadlines
• poor engagement between the myki contractor and public transport operators
• TTA did not possess the capacity and capability needed to effectively manage the project.

In combination with myki's ambitious initial scale and poorly defined functional performance requirements, these issues contributed to significant delays in implementing the new ticketing system and compromised the viability of myki's initial contract. This precipitated six major contractual amendments as the project encountered significant difficulties.

These amendments strengthened the contract's viability but at the cost of substantially reducing myki's scope, increasing its total cost to the state, and extending its delivery time frames by a further seven years. Collectively, these issues have compromised the achievement of myki's originally intended benefits.

PTV's establishment in April 2012, as the new authority responsible for managing myki and Victoria's public transport system, has led to improvements in the governance and contractual arrangements in place for myki.

This has resolved the previous disconnect in responsibilities for public transport ticketing policy development and implementation that hampered TTA's management of myki. The new arrangements have also clarified PTV's role, decision-making authority and accountability for managing the myki contractor.

Monitoring myki's operational performance

The performance regime in the initial myki contract was complex, onerous and, with 81 measures, difficult to apply in practice. Further, the performance measures only applied once myki was fully operational.

The Department of Treasury and Finance (DTF) raised concerns with the initial performance regime in successive briefings to the former Treasurer in 2012 and 2013 noting that the performance measures were 'difficult to interpret and apply' and that a new regime was needed to drive 'appropriate behaviours' from the contractor.

PTV subsequently developed a new performance regime which it implemented in April 2013. The new regime has:

• introduced incentives for the contractor to promptly address faults with myki devices and related software upgrades that previously disrupted system operations
• increased the contractor's focus on assuring the accuracy of financial information recorded by the system.

However, fundamental issues remain. Specifically:

• current measures in the contract's new performance regime do not address all key aspects of performance
• there is no framework for assessing myki's overall effectiveness, efficiency and benefits
• PTV does not adequately assure the reliability of results, reported by the contractor, that underpin incentive payments.

PTV needs to urgently address these issues as they are impacting on the effectiveness and integrity of the performance regime.

Despite PTV not being able to adequately assure the reliability of the contractor's performance data, it has paid $1.4 million in performance-linked incentives and has applied abatements totalling $325 000, between April 2013 and December 2014.

Assessing the achievement of myki's benefits

None of the agencies responsible for myki have assessed whether it has achieved any of its expected outcomes and benefits, including providing value for money.

Consequently, there is no evidence myki has satisfactorily achieved any of its original business case objectives.

In 2012, DOT, TTA and DTF advised the Public Accounts and Estimates Committee that a review assessing the extent to which myki's benefits have been realised would be undertaken following the completion of its rollout. This has not yet occurred.

While DTF subsequently prepared a 'lessons learnt review' in 2014 which identified valuable insights into the issues affecting myki's delivery, including the appropriateness of governance and contractual arrangements, the review did not assess whether any benefits had been achieved.

Instead DTF sought, and received, an exemption from the former Treasurer in 2013 from performing the benefits review required by the Gateway Review process. It justified this on the grounds that the business case had not been updated since the project was first approved, and because it was unable to access it, given it was a Cabinet document of the former government.

This advice was deficient. While DTF correctly noted the constraint to accessing the business case posed by existing Cabinet conventions, it did not advise the Treasurer that this impedes:

• the effective governance of myki, and of all major state investments requiring a full business case to be approved by Cabinet
• DTF's oversight of these projects under the High Value High Risk framework.

Urgent action is required to address this issue as this risks diminishing the value and role of a business case. It reduces it to simply being a hoop agencies jump through to get the initial approval, which they can then disregard following any change of government.

The original business case committed almost $1 billion of taxpayer funds on the premise that myki would deliver significant benefits to Victorians. It is therefore critical that PTV and DTF assess the benefits achieved to understand the nature and extent of any benefits that remain achievable going forward. This will help assure full accountability for and transparency of the achievements from this significant expenditure of taxpayer funds.

While the current time lines may mean that it will be difficult to undertake a post-implementation review against the business case in time to inform the retender, such a review would provide PTV with important insights on the actions it should take to optimise achievement of benefits in the current retender and next iteration of myki.

Notwithstanding, as noted earlier, the substantial reduction in myki's scope over time, coupled with significant increases in the original project time line and budget, mean that it is almost certain that myki has failed to satisfactorily achieve its original business case objectives and assumed benefits.

Planning for the myki retender

The myki retender is occurring under significant time pressure due, in part, to previous delays in implementing myki.

These delays have forced PTV to commit to exercising its option to extend the current contract for a further six months to December 2016 in order to accommodate the retender evaluation and transition period. This has virtually eliminated any contingency in the retender schedule.

Consequently, there is a significant risk that the tender outcome will not be finalised before the current contractual term expires. If this occurs, it may seriously worsen the state's negotiating position and expose it to significant additional costs.

The current contract provides no option for further extensions. This means there is no certainty that this can feasibly occur. Any significant extension of the transition and handover period beyond December 2016, if a new contractor is appointed, is likely to require extensive negotiations with the incumbent operator and substantial additional funds to cover unanticipated costs.

PTV is taking steps to mitigate the risk of this occurring and, in so doing, is considering the relevant lessons from the previous myki contract. However, to achieve this PTV needs to effectively manage the retender and related schedule.

PTV is yet to clearly define the expected outcomes and benefits from the retender process and the new contract. This reduces the transparency of the new contract's impact, and accountability for PTV's related performance.

PTV needs to address this issue as a matter of urgency as it has the potential, yet again, to compromise the achievement of planned outcomes from this critical system.

Recommendations

1. strengthens its performance monitoring arrangements for myki prior to awarding the new contract by:

• assessing the adequacy of existing performance measures and standards for driving improvements in performance
• developing new measures addressing how well the equipment operates as distinct from the length of time for which it is available
• reviewing on at least an annual basis and, where necessary, adjusting performance incentives to support further improvements in performance or achievement of emerging service priorities
• developing a broader framework to assess myki's efficiency and effectiveness and its impact on improving performance and management of the public transport system

2. uses its right under the contract to audit and verify the performance data provided by the contractor
3. seek access to the original myki business case in consultation with the Department of Premier and Cabinet and:

• conduct a post-implementation review of the myki project against its original objectives and benefits
• incorporate relevant lessons into the new myki contract as soon as possible and in any future subsequent procurement of public transport ticketing services.

That the Department of Treasury and Finance coordinate with the Department of Premier and Cabinet to:

4. advise the government on the impacts of current Cabinet conventions on the Gateway Review and High Value High Risk framework and related benefits reviews.

That Public Transport Victoria:

5. clarifies the retender benefits and intended outcomes and develops measurable associated indicators.

Submissions and comments received

We have professionally engaged with Public Transport Victoria, the Department of Treasury and Finance, and the Department of Economic Development, Jobs, Transport & Resources throughout the course of the audit. In accordance with section 16(3) of the Audit Act 1994 we provided a copy of this report to those agencies and requested their submissions or comments.

We have considered those views in reaching our audit conclusions and have represented them to the extent relevant and warranted. Their full section 16(3) submissions and comments are included in Appendix B.

1.1 Introduction

Public transport ticketing systems

Modern public transport ticketing systems are much more than simply fare collection tools. They provide the key interface with customers and, with smartcard technology, can assist strategic planning for the public transport system, through the collection of valuable usage data. A well designed and implemented system should be easy for commuters to use, and benefit transport operators by streamlining fare collection and providing access to important data on travel behaviour.

Smartcard systems are the new standard in transport ticketing internationally due to their convenience and efficiency. Tickets can be purchased and 'topped up' using automated processes, they allow rapid movement through stations and easy interchange between different travel modes.

However, international experience has also shown that developing reliable and effective public transport ticketing systems is both difficult and expensive.

1.2 Overview of myki

In Victoria, public transport ticketing depends on which service a passenger uses:

• myki for metropolitan trains, trams and buses, V/Line commuter trains and some regional town buses—Appendix A in this report contains a map showing the regional train and coach network and a list of regional town bus networks included in the myki ticketing system
• paper tickets for V/Line coach and long distance train services
• paper tickets for some regional town buses.

myki smartcard

In July 2005, the state entered into a contract to develop a smartcard public transport ticketing system, known as myki, to replace the former Metcard system, which expired in 2007.

Under the terms of the original contract, myki was to be fully operational by July 2007. The initial whole-of-project budget was $998.9 million and included $177.1 million for delivering myki and $345.5 million for operating it for 10 years. The remaining budget included funds for operating the Metcard system in parallel with myki during the transition, and related operating costs for the former Transport Ticketing Authority (TTA). TTA was established in 2003 to manage Metcard, and the procurement and rollout of the new ticketing system, now known as myki. TTA was abolished in 2013 following the establishment of Public Transport Victoria (PTV) which has since assumed responsibility for all of TTA's functions.

The myki system includes the following components:

• myki smartcard—used to pay for travel on public transport. The myki smartcard technology enables a money value—myki money—and/or a travel pass—myki pass—to be stored on the card. Public transport users that 'top up' with myki money need to touch on and off for each trip so that the myki system can automatically calculate and deduct the lowest fare for the travel taken.
• Devices—the system currently has around 23 500 operational devices. These include card vending and top-up machines, fare payment devices, bus and tram driver consoles, station gates and hand-held devices. These devices read and translate information stored on smartcards and transmit information to and from the back office.
• The back office and related systems—these support the operation of the myki system.

Card vending and top-up machine

It was expected that myki would deliver a range of benefits for users, transport operators and the state including:

• ease of use
• convenient purchase and payment options
• efficient operations in terms of boarding
• high-quality information for transport planning
• increased flexibility to change and drive user patterns through differential pricing and fare structures.

myki is an open-architecture system, meaning it allows the flexibility of adding, upgrading and swapping components over time without limiting the state to using a single vendor.

1.2.1 Implementation of myki system

Despite their clear benefits for commuters and transport system operators, smartcard ticketing systems are challenging to design and implement due to:

• the complexity involved in designing and delivering a solution that meets the needs of a unique transport system, such as Victoria's, with multiple modes and zones of travel
• the need to properly inform and assist commuters as they adapt to the new system.

There are few examples of the ambitious approach taken in Victoria, which sought to introduce a statewide smartcard ticketing system across all public transport modes and zones. The Department of Treasury and Finance's (DTF) 2011 Project review of the myki ticketing system noted that most smartcard ticketing systems were initially introduced in the metropolitan area only—as in Perth, London, Hong Kong and Singapore—or a metropolitan area plus adjacent region—as in South East Queensland. There are considerable challenges in bringing together a range of different fares across metropolitan, regional and rural areas, different public transport operators, and various types of customers.

Figure 1A compares myki to other smartcard systems operating in Perth, South East Queensland, and London by initial scope and key functions. It shows that myki is the only smartcard ticketing system that was initially integrated across metropolitan and regional areas, and possessed a wider range of functions than the ticketing systems in other jurisdictions.

Figure 1A

Ticketing system comparisons

Source: Victorian Auditor-General's Office based on the Project Review of the myki ticketing system, February 2011, Department of Treasury and Finance.

The myki system covers:

• 15 train lines with 208 stations in metropolitan Melbourne
• 24 tram routes with 500 trams in metropolitan Melbourne
• 346 bus routes with 1 753 buses in metropolitan Melbourne
• 5 regional train lines with 51 stations
• 127 regional bus routes.

The myki system also offers a complex range of ticketing fares, with both zoned and time‑based charging and a variety of concessions and discounts. The system executes 150 business rules each time a card is scanned, which constitutes around 1.07 million fare transaction‑type permutations, making it one of the most complex smartcard ticketing solutions in the world.

As at December 2014, more than 13.4 million myki cards had been issued with the system processing around 7.8 million 'touch on' transactions per week from 9.9 million active cards. In 2013–14 the total farebox collected by myki across all transport modes was around $800 million.

1.2.2 Implementation of myki

Time line in implementing myki

The myki system was due to be operational by July 2007. However, there have been multiple delays, scope changes and cost increases. Figure 1B illustrates these delays.

Figure 1B

Delays in implementing myki

Source: Victorian Auditor-General's Office.

Initially myki was implemented on regional buses in Geelong in December 2008. The system was extended to bus services in five other regional centres from March 2009. It began running on metropolitan trains in December 2009, and was extended to metropolitan trams and buses in July 2010. In December 2012, myki became the only form of ticket valid on Melbourne public transport and the previous ticketing system, Metcard, was switched off. In February 2014, myki replaced paper tickets on V/Line services across selected services in Victoria.

The total time of the design and delivery phase has more than quadrupled from the original expectation of two years, to in excess of nine years. This is more consistent with smartcard projects in other cities, where it usually takes five or more years.

Figure 1C illustrates the time taken to implement public transport ticketing systems in other jurisdictions.

Figure 1C

Delivery time frames for public transport ticketing systems

Source: Victorian Auditor-General's Office based on the Victorian Government submission to the Legislative Council Select Committee on Train Services, former Department of Transport.

Cost

At 30 June 2014, the revised capital and operating budget for myki until 2016 was $1 550.5 million, an increase of 55 per cent on the initial budget of $998.9 million.

Significant additional costs of approximately $200.1 million, over the budgeted amount of $106.8 million, were incurred for operating the Metcard system in parallel with myki beyond its contracted termination date in 2007.

Issues with implementing myki

There were a number of implementation issues with myki—including slow card reader response times, intermittent technical failures particularly on trams and buses, and concerns with the accuracy of data for measuring patronage.

The Public Transport Ombudsman received over 5 450 complaints about myki between July 2010 and June 2014. Most of the issues raised were associated with account charges and refunds and reimbursements.

Data maintained by PTV shows that during the same period it received in excess of 200 000 complaints and enquiries relating to myki.

Our 2012 audit, Fare Evasion on Public Transport, also noted a significant increase in metropolitan fare evasion between 2009 and 2011 due to the decline in effective enforcement during the transition to myki. However, the estimated fare evasion on Melbourne's public transport network has since dropped to its lowest level since 2008.

Given the significant delay, additional costs and challenges in delivering myki, the former government commissioned a review of the system in late 2010 to determine its future. Following this review it decided to proceed with implementing myki but with a reduced scope. The contract with the myki operator was subsequently amended in November 2011 and March 2013 to reflect the revised approach.

The current myki contract service term is due to expire on 30 June 2016. PTV is currently planning to retender the operation of the myki system.

1.3 Reviews of public transport ticketing

Previous performance audits

Our 1990 audit Met Ticket, and 1998 audit Automating Fare Collection: A major initiative in public transport, examined the state's previous public transport ticketing solutions.

Both of these audits found that fast tracking the development of these ticketing systems led to successive delays and cost blowouts in implementation, which exposed the state to major risks of not achieving system objectives.

Specifically, our 1990 audit Met Ticket found that a compressed time frame for implementing Met Ticket meant that certain desirable project management principles were bypassed. Specifically, multiple variations to the initial proposal occurred during the rollout, which complicated implementation and resulted in delays. The audit also found that, consequently, substantial effort and cost was incurred with only minimal benefit to public transport users and the government.

Despite the lessons from the Met Ticket experience noted in our 1990 report, in 1998 we identified similar weaknesses with the implementation of Met Ticket's successor, Metcard. In particular this audit found that the decision to fast track Metcard's implementation resulted in a failure to properly analyse the system's costs and benefits, and set unrealistic milestones, leading to delays.

Our 2007 audit New Ticketing System Tender examined the tender process for myki, including the lessons arising for the future management of major tenders. The report concluded that TTA established an 'innovative tender process and largely achieved its objectives for the procurement phase' of the project. It also identified some areas where probity management could be improved, in particular, ensuring that the probity protocols work to minimise the risk of actual or perceived conflicts of interest.

Other reviews of myki

State agencies, Parliament and the Victorian Ombudsman have also previously reviewed the status and implementation of myki. Key reviews include:

• Project review of the myki ticketing system, DTF, February 2011
• Own motion investigation into ICT-enabled projects, Ombudsman Victoria, November 2011, conducted in conjunction with VAGO
• Inquiry into Effective Decision Making for the Successful Delivery Of Significant Infrastructure Projects, Public Accounts and Estimates Committee, December 2012
• Lessons learned review of the myki ticketing system, DTF, April 2014

These reviews have all identified significant issues with the governance and contractual arrangements that were established:

• The roles and responsibilities of key governance bodies were not clearly defined. Consequently, this made it difficult to determine which agencies were responsible for different aspects of the project and this resulted in ineffective communication between key parties.
• The original contract was large, complex and outcomes-based. This meant that the functional performance requirements of the system were not clearly defined. This ultimately led to confusion over the responsibilities of the contractor, incurring delays and budget increases.

1.4 Institutional arrangements and responsibilities

Transport Ticketing Authority

The TTA was established in June 2003 to manage the Metcard ticketing system and to procure and manage a new replacement ticketing system, now known as myki. The TTA completed its role in the delivery of myki and the management of the Metcard system in December 2012, with PTV assuming responsibility for all TTA functions from January 2013.

Public Transport Victoria

PTV's primary objective under the Transport Integration Act 2010 is to plan, coordinate, provide, operate and maintain a safe, punctual, reliable and clean public transport system consistent with the vision statement and transport system objectives contained in the Act. This includes providing and operating a public transport ticketing system, managing the public transport ticketing system contract, and the collection and distribution of farebox revenue.

Department of Economic Development, Jobs, Transport & Resources

The Department of Economic Development, Jobs, Transport & Resources (DEDJTR) is responsible for leading strategic policy, planning and improvements relating to the transport network.

DEDJTR is currently a member of the myki Ticketing Advisory Group—which advises the PTV board on myki matters—and the myki Ticketing Services Retender steering committee—which provides advice and guidance relating to the retender of myki.

Department of Treasury and Finance

DTF's role and involvement in the project has varied over its life cycle. This has included observer membership of the former TTA board, current membership of the Ticketing Advisory Group, participation in the Ticketing Services project steering committee and reviewing associated progress reports. From 2010 DTF expanded its monitoring and review function as the project is now monitored under the High Value High Risk framework. DTF also led negotiations with the vendor following the review of the project, which was initiated by the former government and completed in 2011.

1.5 Audit objective and scope

The objective of the audit was to examine the operational effectiveness of the myki ticketing system by assessing whether:

• arrangements in place for the system's implementation and ongoing management are effective
• expected outcomes and benefits from the introduction of myki for users, operators and the state have been, or are on track to being, achieved.

The audit focused on how well PTV is managing and operating the myki ticketing system. Specifically, it examined whether governance and contractual arrangements including frameworks for monitoring and evaluation, assure myki's operational effectiveness. It also included DEDJTR due to its role in working with PTV on future contract planning, and DTF due to its role in overseeing and monitoring the project.

1.6 Audit method and cost

The audit was conducted in accordance with the Australian Auditing and Assurance Standards. Pursuant to section 20(3) of the Audit Act 1994, unless otherwise indicated any persons named in this report are not the subject of adverse comment or opinion.

The cost of the audit was $415 000.

1.7 Structure of the report

The report has three further parts:

• Part 2 examines whether governance and contractual arrangements support effective implementation and operation of myki.
• Part 3 examines:
  • the performance of the myki system and adequacy of arrangements for monitoring the contractor's performance
  • whether expected outcomes and benefits from the introduction of myki for users, operators and the state have been, or are on track to being, achieved.
• Part 4 examines whether planning for the retender for managing myki's ongoing operation is soundly based and designed to deliver a value-for-money outcome.
• Appendix A shows the extent of myki use on the regional train, coach and town bus network.

2.1 Introduction

The myki project has been complex and large-scale from its conception and through its development and procurement. Sound governance and oversight of such projects is essential for assuring risks to delivery are effectively managed, and that intended benefits are achieved.

This requires effective mechanisms for project monitoring and risk management to support timely decision-making in response to emerging issues.

It also requires clearly defined deliverables and responsibilities, and effective contract management to ensure responsible parties meet their respective obligations and achieve the contract's objectives.

This Part of the report examines whether myki's governance and contractual arrangements support its effective implementation and operation.

2.2 Conclusion

The myki system has experienced significant delays and cost increases. This is largely due to deficiencies in the original contract and governance arrangements.

The roles and responsibilities of key agencies initially charged with myki's development were neither well defined nor effectively implemented. This led to uncertainty and delays in decision-making—particularly during myki's initial build and rollout.

The ambitious initial scale and poorly defined functional performance requirements for myki also contributed to significant delays in its implementation. Collectively, these challenges compromised the viability of myki's initial scope and contract, and precipitated six major contractual amendments as the project encountered significant difficulties.

While these amendments strengthened the contract, they also substantially reduced myki's scope and extended its delivery time frames by seven years. This compromised achievement of the project's original objectives and expected benefits.

Since it was established in April 2012, Public Transport Victoria (PTV) has improved governance and contractual arrangements to support the oversight and operation of myki.

2.3 Governance arrangements supporting the delivery and operation of myki

The governance arrangements for the myki project have been scrutinised by several reviews over the past four years. These reviews consistently identified significant weaknesses that needed to be addressed.

Department of Treasury and Finance's 2011 Project review of the myki ticketing system

This review examined the initial myki contract and rollout, including the government's options for proceeding with or discontinuing myki.

The review found that myki's original communication, decision-making and accountability arrangements were 'troublesome'. In particular, it noted there was a lack of clarity regarding whether the former Department of Transport (DOT) or the Transport Ticketing Authority (TTA) was the ultimate decision-maker on issues, and that poor engagement between the myki contractor and public transport operators contributed to a difficult working relationship that negatively impacted the project.

The review also found that:

• the TTA board and public transport operators were excluded from the governance arrangements
• TTA did not have clear accountability and authority to directly manage the contractor
• the role and authority of all project-related groups was unclear and lacked integration.

The review noted that these arrangements needed to be clarified and strengthened if the state decided to proceed with the project.

Victorian Ombudsman 2011 Own motion investigation into ICT‑enabled projects

This investigation examined 10 major ICT-enabled projects, including myki, to determine whether they were on budget and on time, and if they met the needs for which they were designed.

The Ombudsman was critical of the fact that TTA's board initially did not include a representative from DOT, given that DOT had policy responsibility for ticketing issues and TTA was reliant on it for promptly addressing its requests. The Ombudsman also questioned if the board had the requisite experience and skills to implement an ICT project of the size and complexity of myki.

Similarly, our 2007 audit New Ticketing System Tender found that, for most of the tender period, the board consisted of only two members and that a larger board was warranted given the size and complexity of the tender.

Public Accounts and Estimates Committee's 2012 Inquiry into Effective Decision Making for the Successful Delivery Of Significant Infrastructure Projects

The Public Accounts and Estimates Committee's (PAEC) 2012 inquiry into the decision-making underpinning successful infrastructure projects examined six projects, including myki, to identify lessons learned.

The inquiry found myki lacked continuity in project management and governance, and that TTA did not possess the capability and capacity needed to effectively manage the project. It also noted that significant changes to key personnel exposed the project to greater risks than if there had been a consistent, high-quality team running it.

Department of Treasury and Finance 2014 Lessons learned review of the myki ticketing system

This review examined critical issues affecting myki's delivery within the originally approved time frame and budget.

The review found that the original governance structure made it very difficult to identify which agency had overall accountability for the project and what aspects of the project each agency was responsible for. Consistent with the Ombudsman's 2011 investigation, the review noted that the relationship between TTA and the former DOT was not clearly defined. Specifically, while DOT had policy responsibility for ticketing issues, it did not have a representative on the TTA board and therefore no visibility of policy implementation.

The review further noted that ineffective communication between the two agencies compounded the structural deficiencies—with TTA having to wait for DOT to respond to policy enquiries that may have had time-critical implications. Because of this reliance on DOT to clarify policy issues, TTA did not have complete control over meeting prescribed project deadlines. DTF therefore concluded that the design and operation of the governance arrangements was ineffective. DTF also noted that, had there been a closer working relationship between DOT and TTA, or if DOT had membership on the board, TTA's functioning with regard to policy issues may have been greatly improved.

Revised governance arrangements

The governance arrangements for the myki project were revised following PTV's establishment in 2012. The new arrangements have substantially addressed the previous deficiencies. In particular, PTV has now assumed the functions of managing the public transport system and myki, which were formerly separate functions under DOT and TTA respectively.

This fundamental change has, in effect, resolved the previous disconnect in responsibilities for public transport ticketing policy development and implementation that hampered TTA's management of myki. It has also clarified PTV's role, decision‑making authority and accountability for managing the myki contractor.

PTV's updated governance structure for myki includes arrangements designed to strengthen:

• cross-agency coordination
• engagement between PTV, the myki contractor and public transport operators
• PTV's expertise and capability for managing the contract.

Figure 2A outlines the revised governance arrangements.

Figure 2A

Current governance arrangements

Source: Victorian Auditor-General's Office.

In addition to these arrangements, Figure 2B outlines other initiatives recently introduced by PTV to improve its engagement with public transport operators and the myki contractor.

Figure 2B

Operator engagement initiatives

Source: Victorian Auditor-General's Office.

2.4 Contractual arrangements for myki

Previous reviews of myki have also found significant problems with the original contractual arrangements. These problems impacted the build, initial implementation and operation of myki and were linked to:

• poorly defined functional performance requirements
• a lack of flexibility in the initial contract to address underperformance
• unrealistic time frames
• poor financial viability of the initial contract.

Poorly defined functional performance requirements

The Ombudsman's 2011 investigation noted that the original myki contract was large and complex with over 13 000 pages and 40 schedules. It also found that the contract was difficult to manage as it was outcomes rather than requirements based, meaning it was unclear whether certain functional requirements were within or outside the scope of the contract. This led to disputes with the contractor about costs and priorities.

DTF's 2014 review similarly found that the contract lacked sufficient specificity as the required functional performance of the system was not clearly defined.

Additionally, PAEC's 2012 inquiry noted that the state and contractor would have been better served 'if more time had been invested in the beginning' to explore and get a common understanding of the requirements.

Lack of flexibility to address contractor underperformance

DTF's 2014 review found that the structure of the contract covering the project's build and operations phase did not provide the flexibility to suspend or exit the contract to address underperformance. This complicated the system's rollout, including the state's relationship with the contractor, as the operations phase was allowed to commence before the build was complete and related issues were resolved.

In particular, this meant that outstanding build issues had to be managed by the contractor in parallel with the commencement of operations which compromised the contractor's capacity to meet prescribed milestones and, ultimately, the project's delivery.

Unrealistic time frames

Early decisions in the planning phase of the myki project caused subsequent challenges that necessitated contractual amendments. In particular, the expiration of the former Metcard contract in 2007 was the catalyst for introducing myki and seeking to implement it within the ambitious time frame of two years.

This time line, stipulated in the initial contract, proved to be unrealistic and affected the contractor's ability to meet milestones and receive milestone delivery payments. Since myki commenced operating in 2009, contract payments of $27 million were withheld from the contractor for its failure to achieve milestones.

TTA advised the 2012 PAEC inquiry that the time frame had 'been the biggest single cause of project difficulties' with the contractor consistently failing to meet milestones during the initial build phase of the project. The contract was subsequently amended, which reset the project's milestones.

The complexity of the project was significantly underestimated and DTF's 2014 review noted that, 'no other jurisdiction in the world had developed and delivered a ticketing procurement strategy that included an outcomes-based specification through an open architecture approach in the two year time frame committed. The initial project planning did not take into account international experience on the implementation of new ticketing systems'. According to the review, if the contract had been appropriately benchmarked against projects around the world, which were similar in complexity, size, budget and time, the majority of changes subsequently incorporated in the contract may not have been necessary.

Financial viability of the contract

The 2011 DTF review noted that neither the contractor nor TTA felt the operating phase of the contract was commercially viable. Scope extensions in multiple areas including cash collection, card distribution and system maintenance had increased costs beyond what could be borne by the contractor. The contractor also highlighted that there were cost implications stemming from commencing operations in parallel with the delivery phase in 2009, rather than at the end of delivery.

A further issue affecting the commercial viability of the contract was its failure to accommodate the increased operating costs for the system arising from the 30 per cent growth in public transport patronage that occurred during its implementation. Contractual amendments had addressed the increased capital costs associated with patronage growth, but not the operations and maintenance costs.

Key themes

These issues with the myki project identified by the Ombudsman and PAEC are consistent with the key themes emerging from their related examination of other major ICT projects. These themes highlight that:

• poorly defined governance structures with unclear roles and responsibilities have hampered project management
• inadequate planning has led to significant delays, cost increases and failures to achieve original objectives
• overly-ambitious and poorly-scoped projects have unnecessarily heightened project complexity and risk
• inadequate ICT capability within agencies has contributed to poor project management.

2.4.1 Contractual amendments

Since the initial contract was awarded, there have been numerous variations and six major contractual amendments directed at addressing these issues. Figure 2C summarises the contractual amendments agreed between the parties.

Figure 2C

Summary of major amendments to the myki contract

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

The extent of changes to the contract indicates that the initial planning underestimated the complexity of implementing a new ticketing system. It also suggests that more time should have been spent developing the contract to more clearly define requirements and critical milestones.

Impact of key amendments

The most significant contractual changes were Amending Deeds 5 and 6, which addressed the unrealistic time lines and financial viability of the contract.

Amending Deed 5 ensured the deliverability of myki by reducing its scope to enable it to be rolled out by the revised due date of December 2012 when Metcard was to be switched off.

Amending Deed 6 was implemented to address the risk that the contractor may minimise or discontinue support for the system because the project was financially unviable. This amendment fundamentally changed the contract and how the state would pay for myki. The key changes included:

• the adoption of a 'cost plus' reimbursement model, rather than a fixed price arrangement—this meant the total cost of the system under the contract is no longer capped and ensured it was financially viable for the contractor
• a redefined scope of services, with some services no longer performed by the contractor—including call centre operations
• changed key performance indicators, encompassing financial incentives and abatements.

The introduction of the 'cost plus' reimbursement model and financial incentives for sustained good performance has improved the commercial viability of the contract. However, shortcomings with PTV's approach to verifying the contractor's performance data are compromising its capacity to assure the integrity of performance-based payments. This is further discussed in Part 3 of the report.

3.1 Introduction

Public Transport Victoria (PTV) needs an effective performance measurement and reporting framework in order to ensure that the system operates effectively, and delivers the intended benefits.

Such a framework depends on the availability of sufficient, appropriate and reliable performance data and on PTV regularly monitoring critical aspects of performance. This includes monitoring whether services are being delivered in accordance with the contractual standards so that timely action can be taken to address emerging issues.

This Part of the report examines:

• the operational performance of myki and adequacy of arrangements for monitoring the contractor's performance
• whether expected outcomes and benefits from the introduction of myki for users, operators and the state have been, or are on track to be, achieved.

3.2 Conclusion

PTV has enhanced the quality of the performance measures and incentives in the myki contract, and their potential to drive improved performance.

However, the effectiveness and integrity of the new performance regime is currently compromised by critical shortcomings in PTV's approach to verifying the accuracy of related results reported by the contractor.

PTV needs to urgently address these issues as they reduce assurance that current incentive payments are appropriate and justified.

Further, the absence of arrangements for monitoring myki's overall effectiveness, efficiency and achievement of benefits, means there is currently no evidence to demonstrate its impact on the performance and management of the public transport system. However, the significant delays, budget increases and substantial reductions in myki's scope over time strongly indicate that it has failed to fully meet its original objectives.

3.3 Measuring myki's performance

3.3.1 Issues with previous monitoring arrangements

The performance regime in the initial myki contract was complex, onerous and, with 81 measures, difficult to apply in practice. Further, the performance measures only applied once myki was fully operational. The Department of Treasury and Finance (DTF) had raised concerns with the performance regime in a number of ministerial briefings. For example, a 2012 DTF briefing to the Treasurer noted that the performance measures in the contract were 'difficult to interpret and apply' and that a new regime was needed to drive 'appropriate behaviours' from the contractor. A subsequent 2013 DTF briefing to the Treasurer similarly noted that 'more measurable and enforceable key performance indicator targets' were needed.

A contractual amendment introduced in March 2013—Amending Deed 6—sought to address these shortcomings by establishing a new performance regime. Key changes to the contract included:

• deleting obsolete measures relating to services that were removed from the contract—this substantially reduced the measures from 81 to 27
• establishing an incentive and abatement regime to drive improvements in customer service and system reliability, by linking the contractor's performance against defined standards to financial incentives and penalties.

3.3.2 New performance measures

The new performance regime includes 27 measures aligned with seven service categories specified in the contract:

• distribution services—the supply, distribution and stock management of smartcards
• cash collection—the collection of cash from card vending machines and ticket offices and the replenishment of cash, smartcards and consumables
• scheme administration—the administration and maintenance of data and products
• device management—the implementation, updating, operation and maintenance of devices
• financial management—the time lines of financial settlement, reconciliation and investigation of associated discrepancies
• technical support—the management of the operational, functional and performance aspects of the back office systems, and system connectivity with devices
• operational support—response to queries from transport operators, myki retailers and PTV.

The new regime has improved the clarity and focus of performance measures in the contract, and its potential to drive improved performance. In particular, the new regime has heightened the focus on service aspects that previously emerged as critical performance issues. Specifically, new measures have introduced incentives for the contractor to promptly address faults with myki devices and related software upgrades, which previously disrupted system operations. The new regime has also increased the contractor's focus on assuring the accuracy of the financial information recorded by the system.

3.3.3 Limitations of the current performance regime

Limited focus on equipment availability

While the contract amendments improved the performance measures in the original contract, fundamental issues remain.

Specifically, the performance regime is focused on availability rather than performance. That is, it does not adequately measure how well the equipment operates as distinct from the length of time for which it is functioning and available. For example, the speed of transaction processing—the response times of myki fare readers, including entry/exit gates—is not measured.

This is a critical omission given that a key intended benefit of myki is fast and convenient ticketing for the travelling public. PTV's testing of transaction speed at entry and exit points on all device types during the implementation of myki reinforces the importance of such measures, as it showed that this ranged from 1.8 to 4.2 seconds against a design target of 1.2 seconds.

Current measures also fail to distinguish performance problems with equipment at busy train stations during peak periods, an obviously critical performance issue, from similar issues with equipment at infrequently used stations during off peak periods.

Lack of system-wide and benefits monitoring

There is no overarching framework for monitoring the effectiveness and efficiency of the myki ticketing system.

The new performance regime is limited to monitoring the contractor's performance and does not enable assessment of the extent to which myki's broader intended benefits and value for money have been achieved.

This issue is discussed further in Section 3.7.

3.3.4 Limitations of other monitoring activities

PTV produces a number of monitoring reports that supplement the performance information about myki it derives from the contract's performance regime. While these reports provide useful insights, they do not provide a complete view of myki's performance, nor are they sufficient to verify all the performance information submitted by the myki contractor.

Specifically, these reports include:

• operational performance monitoring reports which focus on myki device availability, transaction capture by the system, the condition of selected devices and the status of maintenance works
• device management dashboard reports which focus on the status of maintenance works for all myki devices
• operator dashboard reports for trains and trams which similarly focus on maintenance issues as well as tracking the number of card sales and 'touch-ons'
• metro trains device downtime reports, which focus exclusively on the myki system's 'availability' across the train network.

A key limitation of the above reports is that, like the contract's performance regime, they mainly focus on the 'availability' and condition of myki devices, rather than on how well the system performs.

While PTV's operational performance monitoring reports examine aspects of myki's performance, this is limited to assessing a sample of bus, tram and train card vending machines and if they are functioning, and the responsiveness of selected myki card readers on a sample of trams and buses.

Other limitations of the operational performance monitoring reports include:

• they make no reference to the standard used for assessing the responsiveness of devices and only describe changes in performance relative to previous reports, meaning the basis of the assessment is unclear
• they do not assess the performance of myki entry/exit gates and card readers across the train network, despite initial PTV testing during the rollout showing the system performed worse than the contractual standard
• the reports do not provide up-to-date intelligence on how the system is currently performing
• the reports do not cover the range of measures reported on by the contractor for determining incentive payments and abatements, meaning they cannot be reliably used to verify all monthly results reported to PTV by the contractor.

PTV also advised that it maintains a register of incidents, such as equipment faults, that have impacted the myki system which it considers when assessing the contractor's performance in relation to the incentive and abatement scheme. However, the information in this register is limited as it depends on the vigilance of staff and customers to report such incidents to PTV. Unless PTV is informed, it is unlikely to be aware of and record the incident. Consequently, there is a risk that the real level of equipment faults may be understated.

3.3.5 Incentives and abatement regime

The new performance regime now incorporates performance-based payments linked to 12 of the 27 performance measures. The contractor has the opportunity to receive incentive payments for sustained positive performance against these measures. The incentive scheme is separated into two categories—customer service and system reliability. Incentive payments are calculated separately for each category.

The performance regime also provides for abating the contractor's monthly service charge for sustained poor performance across 27 measures. Figure 3A provides an overview of the incentive and abatement regime.

Figure 3A

Overview of the incentive and abatement regime

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

A deficit point system is used to determine whether the contractor is eligible to receive an incentive payment or if the contractor's monthly service charge should be abated. Deficit points are accrued when the contractor does not meet a measure's target. Each measure has a different method for calculating how deficit points are accrued.

Under the regime, the contractor is eligible for an initial incentive payment of $150 000 per category when the total number of accrued deficit points for the category remain at, or below, 1 000 for three consecutive months. For each subsequent month that deficit points remains below the threshold, a further incentive payment of $50 000 per category is awarded. If the number of deficit points exceeds the threshold in a particular month, no incentive is awarded for the category, and the three-month period is reset. This means the contractor will need to achieve another three months of sustained good performance to qualify for an incentive payment.

The contractor's monthly service charge can be abated for poor performance. The abatement figure is calculated from the deficit point system, with each deficit point equalling $1. If the number of deficit points accrued across all 27 measures within a month is below 2 000, no abatement is applied. However, once this threshold is exceeded, the full abatement amount is enforceable. This abatement amount cannot exceed 10 per cent of the monthly services charge of approximately $5 million.

The contractor is able to receive an incentive payment for one or both service categories and still be liable for an abatement if performance in one or more of the 27 measures is poor.

As at December 2014, PTV had paid $1.4 million in performance-linked incentives, and had applied abatements totalling $325 000 since the regime commenced in April 2013. Figure 3B shows the net incentive payment for each month during this period.

Figure 3B

Net incentive payment, April 2013 to December 2014

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

Our analysis of data supplied by PTV shows that apart from the first three months of the new incentive and abatement regime, the contractor has consistently received incentive payments for meeting or exceeding performance requirements.

Although these results indicate sustained good performance, it is not possible to assess whether current standards—including incentives and abatement thresholds—have been set appropriately, as PTV did not document the rationale underpinning current performance benchmarks and how they equate to satisfactory performance.

PTV needs to review these measures prior to awarding the new myki contract, including whether the standards and related financial incentives are set at appropriate levels.

3.4 Monitoring of myki's performance

The myki contract requires the operator to prepare a monthly operational report on its performance across the 27 indicators. This information is used by PTV to assess the contractor's performance.

3.4.1 Verification of contractor's monthly reports

The integrity and effectiveness of the performance regime depends heavily on the availability of accurate and reliable data. PTV reviews the monthly performance report submitted by the contractor and prepares a verification report highlighting any discrepancies. PTV advised that if a discrepancy is detected, the contractor is informed of the issue and, where agreed and deemed appropriate, relevant modifications are made to the contractor's report. If an agreement cannot be reached, the issue can be referred for dispute resolution to the Operational Control Group. Evidence provided by PTV shows that this process has previously led to changes in the contractor's reports following the detection of such discrepancies.

However, PTV's verification activities are limited and do not provide sufficient assurance that the source data used by the contractor to compile performance reports is accurate. This means that PTV cannot reliably verify the accuracy of the results reported by the contractor and therefore assure that all incentive payments are appropriate.

PTV does not currently have all the information needed to sufficiently assess and validate the contractor's reported results. Under the contract, PTV can conduct an audit which would give it access to the performance data and systems maintained by the contractor. However, PTV has not done so. In the absence of follow-the-dollar powers, we have not been able to verify what information the contractor holds and whether it supports information reported to PTV.

We found that on most occasions PTV was unable to verify the accuracy of the following 18 performance measures, nine of which directly related to incentive payments reported by the contractor from April 2013 to July 2014:

• For all six technical support measures, two-thirds of the PTV verification reports we examined stated 'PTV receives no data to be able to qualify (the contractor's) reported data for the month' or there is simply no information recorded.
• For three of the four operational support measures, 87 per cent of PTV's verification reports had no verification analysis, or similarly state that it has no data to verify the contractor's claim that no notable critical incidents have occurred.
• For all three of the cash collection measures, no verification assessment occurred in 75 per cent of the reports.
• For two of the four device management measures, there is similarly no verification analysis in 90 per cent of the reports we examined. In three instances the reports note that 'there is a very large discrepancy between the data supplied by the contractor and that calculated by PTV'.
• For one of the smartcard management measures, no verification assessment occurred in 87 per cent of the reports.
• PTV's verification reports similarly state that the performance thresholds for the three scheme administration measures are disputed by the contractor, meaning that agreed time frames are not clarified, preventing verification of whether the abatement standard has been met.

PTV needs to urgently address these issues as they are impacting on the effectiveness and integrity of the performance regime.

3.4.2 Issues impacting reliability of myki performance measures

Although PTV's verification activities are limited, they have nevertheless exposed several issues with the contractor's performance data. These issues further reduce assurance over the accuracy and reliability of the data upon which performance incentives are based.

For example, PTV's comparison of data about the availability of myki devices reported by the contractor to that arising from its own spot checks, has revealed material differences that can impact payments arising from the incentive and abatement scheme. Figure 3C summarises this data for the period January 2012 to April 2013.

Figure 3C

Per cent of myki devices available, comparison between contractor data and PTV data

Source: Victorian Auditor-General's Office from the Public Transport Victoria, Solution Performance Monitoring Cycle 4.

Although the contractor's reports show in excess of 99 per cent availability for devices, figures recorded from PTV's spot checks ranged between 95 and 98 per cent. The minimum availability standard across all devices to qualify for incentives is 99.5 per cent, therefore if PTV is correct it could disqualify the contractor from an incentive payment. A DTF-commissioned review of these results noted that the reason for the differences is 'quite complex and could be due to how the data is generated and compiled'. The DTF review found that the contractor generates device availability data based on the reported incidents which are recorded when an automated alarm is activated by devices.

However, it is possible that devices may be out of service for short periods without generating an alarm. For example, a device may have software freezes that prevent customer interactions, while still maintaining connectivity with the back office and giving the impression of being fully in service. Consequently, actual device availability is not always reflected, and poor performance may be underreported and not adequately considered when assessing eligibility for incentive payments.

3.5 Snapshot of myki's operational performance

Figure 3D summarises the performance data reported by the myki contractor from April 2013 to December 2014. Although PTV cannot assure its reliability, the data indicates the contractor has met the performance target 86 per cent of the time over this 21‑month period.

Figure 3D

myki operational performance, April 2013 to December 2014

(a) Customer service measures part of incentive regime.

(b) System reliability measures part of incentive regime.

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

Figure 3E shows the bottom four performing measures for the period April 2013 to December 2014 and the percentage of months in which the measures did not meet the performance standard. Specifically, it suggests that the performance regime needs adjustment to increase the contractor's focus on addressing these issues as they have the potential to reduce the revenue collected by myki.

Figure 3E

Bottom four performing measures, April 2013 to December 2014

(a) Customer service measures part of incentive regime.

(b) System reliability measures part of incentive regime.

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

PTV advised that when the contractor breaches the minimum performance standards they are required to submit a Corrective Action Plan with the details of the breach and how it will be addressed.

We found that six Corrective Action Plans were submitted to address breaches of minimum standards for the 'Return terminal location to service' measure during the period July 2013 to April 2014. The Corrective Action Plans reveal a steady decline in the service levels provided by a sub‑contractor following the non-renewal of their contract. However, this measure has consistently improved since the commencement of a new sub‑contractor in April 2014.

3.6 Complaints about myki

PTV, the Victorian Ombudsman and the Public Transport Ombudsman (PTO) all receive and investigate complaints regarding public transport services, including myki. PTV does not report publically on myki complaints.

As shown in Figure 3F, data reported by PTO demonstrates that the highest percentage of public transport complaints during 2013–14, 35 per cent, related to myki. However, this has been steadily decreasing from 42 per cent in 2012–13 and 46 per cent in 2011–12. Similarly, data maintained by PTV on the number of myki complaints and related enquiries shows that they are trending down with around 140 a day in 2013–14, compared to 190 a day in 2012–13.

Figure 3F

Public transport complaints and enquiries, 2013–14

Source: Victorian Auditor-General's Office based on information from the Public Transport Ombudsman.

Overcharging of public transport users

The most common issues raised with PTO were associated with account overcharging, and refunds and reimbursements. These two categories comprised around 45 per cent of myki complaints. This has increased steadily from 29 per cent in 2011–12 and 32 per cent in 2012–13.

The myki system is designed to automatically calculate the lowest fare possible according to the zones travelled. However, in some instances public transport users are overcharged. Figure 3G illustrates that, overall, bus users are the most commonly overcharged customers.

Figure 3G

Number of reimbursements for overcharges, January 2013 to March 2015

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

While there has been a gradual decrease in the number of reimbursements for non‑bus users—from a high of approximately 2 100 in August 2013, to around 60 in December 2014—the overcharging of bus users has remained an ongoing issue, although it declined sharply in early 2015.

This was first identified as a systemic issue in 2011 by PTO where some bus users were being charged for travel in Zone 1 when travelling wholly in Zone 2 and vice versa. This occurred in areas where the zones overlapped. Some consumers were being incorrectly charged the higher zone fare, rather than myki correctly identifying the lowest fare. Similarly, other users with a myki pass permitting travel in one zone were being charged myki money for a different zone.

Figure 3H shows how the myki system was designed to work on buses and when overcharging can occur.

Figure 3H

myki bus overcharging

How the myki system is designed to work on buses:
Where a bus operates in Route A the myki console device can only identify Route A bus stops
	A driver logs onto the myki system when starting a bus route and enters Route A details into the console. This filters out bus stops which are not applicable to Route A.
When an overcharge can occur:
When a bus operates in Route B, but is logged in as Route A myki can only identify Route A bus stops
	If a driver logs on to an incorrect route, or does not update route details when beginning the next service, the myki system will pick up GPS points on the route that is logged into the system. This can result in consumers being charged for the wrong zone.
When a bus operates Route A, but the system identifies GPS bus stop points from Route B
	If a driver logs in, but does not select route details, or fails to log on to the console, the system operates in default mode and locates the nearest GPS points—some of which could be in a different route and an incorrect zone fare applied.

Note: The diagram is designed to provide an indicative perspective only and does not represent the complexity of the bus network which comprises multiple routes intersecting Zone 1 and Zone 2. Road maintenance detours may also have an impact.

Source: Victorian Auditor-General's Office based on information from the Public Transport Ombudsman, Annual Report 2014.

The former Transport Ticketing Authority (TTA) put in place a process for reimbursing affected customers, pending changes to the myki bus consoles, which it believed would address the issue. However, successively revised target dates to address this issue in 2011, 2012 and 2013 were not met.

PTV has since taken a number of steps to address the problem. It identified that one of the issues was bus drivers having difficulty remembering and/or correctly entering the various route codes into the myki console. PTV worked with bus operators to upload daily shift data for each driver into the bus myki system. This meant the drivers were able to simply select 'next trip' in the console, rather than having to key in specific route codes. It also made the process of logging in less open to human error.

PTV advised that when this process was trialled, occurrences of buses operating in 'default mode' or logged into the incorrect route reduced significantly. PTV also engaged two temporary staff to expedite the uploading of shift data for metropolitan bus operators by March 2014.

Despite PTV's efforts, PTO continues to receive complaints from consumers about overcharging due to Zone 1 and 2 overlap. PTV has committed to providing additional training and information to bus operators, to ensure that drivers understand their obligations and how to operate the system. In addition, it is providing regular reports to bus operators to indicate where drivers are not interacting with the system properly and may need additional training.

Changes to fares introduced from January 2015 for travel across both Zone 1 and Zone 2 are expected to partly address this issue as customers will only incur a Zone 1 fare for such trips. However, these changes will not address the problem for Zone 2 only trips, which may continue to be overcharged.

3.7 Benefit realisation and value for money

The myki project has not met its original objectives as it has experienced significant delays, budget overruns and a substantial reduction in project scope over time. Figure 3I illustrates the budget increase and reduction in project scope over time.

Figure 3I

Summary of key myki budget and scope changes since initial project approval

Source: Victorian Auditor-General's Office.

3.7.1 Expected outcomes and benefits

The original myki business case set out the customer, operator, and government objectives. These were developed following a review of prior experience of ticketing systems, stakeholder consultation regarding their needs, and customer research. Figure 3J summarises these objectives for the proposed ticketing solution.

Figure 3J

Customer, operator and government objectives

Source: Victorian Auditor-General's Office based on A New Ticketing Solution: The Business Case, April 2004, Transport Ticketing Authority.

These stakeholder objectives were then synthesised into overarching objectives to develop a ticketing solution that:

• facilitates efficient, reliable and safe transport operations that encourages passenger growth and enhances the community image of public transport in Melbourne
• provides the optimum value-for-money solution for all stakeholders and the lowest whole-of-life cost for the required ticketing functionality
• supports relevant government policy objectives
• is operational at, or shortly after, the expiry of the Metcard contract
• provides a platform for potential future usage in other state and commercial applications.

In addition to the project objectives, the business case considered the broader economic, social and environmental benefits of a smartcard ticketing system. These benefits were expected to deliver around $6.3–$10.8 million per year in benefits of which $4–$5 million was described as relatively tangible. From an economic perspective, the major expected benefits included:

• increased attractiveness of the public transport system and therefore increased patronage
• reduced fare evasion and concession fraud
• increased ticket validations due to improved reliability
• operating cost savings due to reduced bus boarding times.

3.7.2 Monitoring myki's impact

While the business case identified all stakeholder objectives, it did not clearly demonstrate the link between the objectives and the contractual performance regime through measurable indicators. Additionally, myki's objectives have not been updated following modifications to its scope since it was first approved. No benefits monitoring and evaluation process has been established to assess whether myki has achieved its expected outcomes and benefits—including value for money.

The Victorian Ombudsman's 2011 Own motion investigation into ICT-enabled projects recommended that an independent post-implementation review of the myki project against its business case objectives be undertaken to assess this. TTA and DTF accepted this recommendation.

In 2012, the former Department of Transport (DOT), TTA and DTF also advised the Public Accounts and Estimates Committee that a review assessing the extent to which benefits had been realised would be undertaken following the completion of myki's implementation. This has yet to occur.

The scope of the review recommended by the Ombudsman is similar to the benefits review recommended by DTF's Gateway Review process. This review is mandatory for projects assessed as High Value High Risk, such as myki. In October 2013, DTF sought the Treasurer's exemption from performing the benefits review required by the Gateway Review process on the basis that:

• the original business case could not be accessed given it was a Cabinet document of the former government
• the business case had not been updated since the project was first approved
• a DTF led lessons learned review of the factors contributing to myki's stress would fulfil the state's commitment to the Ombudsman and PAEC given the business case could not be accessed to conduct a benefits review.

This advice was deficient. While DTF correctly noted the constraint to accessing the business case posed by existing Cabinet conventions, it did not advise the Treasurer that this impedes:

• the effective governance of myki, and of all major state investments requiring a full business case to be approved by Cabinet
• DTF's oversight of these projects under the High Value High Risk framework.

It also did not advise the Treasurer of the need to resolve this barrier to access given that the lifecycle of most major state investments extends beyond a government's term in office.

This was a missed opportunity, particularly given that DTF previously raised this issue with the Department of Premier and Cabinet (DPC) in 2012, and it has yet to be addressed.

Barrier to effective governance caused by Cabinet conventions

DTF recognises that the current barrier to access caused by existing Cabinet conventions poses a significant challenge to the effective management of such projects, particularly following a change of government. In 2014 it updated its Investment Lifecycle Guidelines to encourage agencies to extract key information from full project business cases—in a form not subject to Cabinet-in-confidence conventions—to mitigate this issue. It also advised it has been encouraging agencies to develop benefit management plans to permit evaluations in the event business cases are not accessible.

While these strategies will assist with mitigating the access issue, they rely on the vigilance of agencies to extract the relevant information prior to the business case being surrendered under Cabinet conventions. If this does not occur, as was the case with myki, the existing conventions do not enable a benefits review to be conducted. DTF advised that it has not yet assessed agencies' compliance with this guideline.

DTF wrote to DPC in 2012 recommending that Cabinet conventions be amended to exempt full business cases for approved projects so that key information is permanently available to relevant agencies to support effective project implementation, monitoring and evaluation. DTF's letter also highlighted that this would permit central agencies continued access to this critical information, which they need to effectively carry out their roles.

However, DTF advised that it received no response to this letter from DPC which is the custodian of Cabinet conventions.

Urgent action is required to address this issue. Currently there is a risk that a business case is reduced to simply being a hoop agencies jump through to get the initial approval, which they can then disregard following a change of government.

Rationale for exempting myki from a benefits review

DTF's request for an exemption on the grounds that the business case had not been updated was also unsound. This approach risks creating an undesirable precedent that benefits reviews should only be reserved for projects that go according to plan, whereas projects that experience significant challenges and scope changes, such as myki, should be exempt because they have gone so far off the rails that they no longer reflect everything they set out to do.

DTF and PTV asserted that a benefits review of myki against the original business case would be of limited value given the retender, unlike the original project, does not involve the development of a new system.

However, the original myki contract established the objectives and benefits expected from both the initial build and ongoing operation of the system. While the retender focuses exclusively on myki's continued operation, a benefits review could have provided valuable insights on how to refine and update myki's original value proposition and assumed benefits, particularly in light of the changes to myki since the project was first approved.

The significant challenges experienced during myki's initial build and rollout heighten the need for this to occur.

The original business case justified the state's decision to invest almost $1 billion of taxpayer funds to develop and implement myki on the basis that it would deliver significant benefits to Victorians.

It is therefore critical for PTV and DTF to assess the benefits achieved so as to understand the nature and extent of any residual or new benefits that remain achievable going forward. It is also needed to assure full accountability and transparency of the achievements from this significant expenditure of taxpayer funds.

Such a review would also provide PTV with important insights on the actions it should take to optimise achievement of these benefits in the current retender and next iteration of myki.

Ongoing need to address the Ombudsman's 2011 recommendation

Any constraints in accessing the original business case could have been initially mitigated by TTA had it developed a benefit management plan for the project as recommended by a 2004 Gateway Review of the procurement strategy. This did not occur.

While DTF subsequently prepared a 'lessons learned review' which identified valuable insights into the issues that affected myki's delivery—including the appropriateness of governance and contractual arrangements—the review did not assess whether any benefits had been achieved. Hence, it did not adequately address the Ombudsman's 2011 recommendation and DTF's 2012 commitment to PAEC.

Consequently, there is no evidence that myki has satisfactorily achieved any of its intended benefits or value for money, relative to the original business case objectives. However, the substantial reductions in the project's scope over time, coupled with significant increases in the original project time line and budget, mean that it is highly unlikely that the system has fully achieved any of its original objectives.

3.7.3 Expenditure on myki

A key determinant of value for money is the cost of myki. The total revised budget for myki is $1 550.5 million. This represents an increase of approximately $551.6 million, or 55 per cent, on the original budget of $998.9 million, and an increase of $809.7 million, or 109 per cent, on the business case estimate of $741.9 million. There have been three significant budget revisions since 2003 when the initial budget was developed.

A breakdown of the original and revised budget, and actual expenditure as at 30 June 2014 is provided in Figure 3K.

Figure 3K

Original budget, revised budget and actual costs of the myki system as at 30 June 2014

(a) For the period to 2017.

(b) For the period to 2019.

(c) For the period to 2016.

Source: Based on information from Public Transport Victoria and the Auditor-General's Reports on the Annual Financial Report of the State of Victoria, 2010–11, 2011–12 and 2013–14.

The expenditure of the myki system includes the capital and operating costs for the contractor to deliver and operate myki until 2016. It also includes the cost of supporting the Metcard system, customer education, marketing and TTA during the initial transition period, as well as ongoing costs for PTV operations.

As shown in Figure 3L the major drivers of increased costs since the original budget have been additional contractor payments, increased costs for civil works relating, in part, to installing additional myki devices, and because of the need to continue operating Metcard for five years longer than expected. Further, the project contingencies did not include the cost of negotiating an extension of the Metcard contract. This meant the state had little leverage to negotiate reasonable extension terms with the Metcard contractor. According to DTF's 2014 Lessons learned review of the myki ticketing system, this resulted in significantly higher costs to keep Metcard in place as delays to myki occurred.

Figure 3L

Comparison of original to current myki budget

Source: Based on information from Public Transport Victoria and the Auditor-General's Reports on the Annual Financial Report of the State of Victoria, 2010–11, 2011–12 and 2013–14.

New cost reimbursement regime

Under the new cost reimbursement regime the state now reimburses the contractor for direct and indirect costs incurred in operating the myki system. This makes the contract more viable for the contractor, however, it exposes the state to a risk of increased costs. Cost driven volume parameters such as device maintenance and card distribution and collection activities—where the contractor previously bore the costs—present the greatest risk to the state.

It is not possible to assess the project's final cost. However, DTF's and PTV's financial analyses indicate that the myki project is expected to come in within its revised budget. Payments under the contract are also currently within budget. Additionally, Amending Deed 6 introduced the following risk mitigation measures to control costs and support agreed service levels:

• more measurable and enforceable key performance indicator targets with new financial incentives for the contractor to outperform these—with an increased focus on areas of service deemed to be critical
• a sustained poor performance regime, with abatement penalties
• a gain-share regime, where the benefits resulting from reduced myki operating costs are shared with the contractor
• establishment of the Operational Control Group which provides PTV with enhanced visibility of the contractor's activities through its input into key planning decisions.

Recommendations

That Public Transport Victoria:

1. strengthens its performance monitoring arrangements for myki prior to awarding the new contract by:

• assessing the adequacy of existing performance measures and standards for driving improvements in performance
• developing new measures addressing how well the equipment operates as distinct from the length of time for which it is available
• reviewing on at least an annual basis and, where necessary, adjusting performance incentives to support further improvements in performance or achievement of emerging service priorities
• developing a broader framework to assess myki's efficiency and effectiveness and its impact on improving performance and management of the public transport system

2. uses its right under the contract to audit and verify the performance data provided by the contractor
3. seeks access to the original myki business case in consultation with the Department of Premier and Cabinet and:

• conduct a post-implementation review of the myki project against its original objectives and benefits
• incorporate relevant lessons into the new myki contract as soon as possible and in any future subsequent procurement of public transport ticketing services.

That the Department of Treasury and Finance coordinate with the Department of Premier and Cabinet to:

4. advise the government on the impacts of current Cabinet conventions on the Gateway Review and High Value High Risk framework and related benefits reviews.

4.1 Introduction

The current myki contract expires in June 2016 with an option to extend it for a further six months at the state's discretion.

Public Transport Victoria (PTV) has commenced planning the myki retender, known as the Ticketing Services Retender (TSR), to ensure myki continues to operate beyond the current contract.

The scale, complexity and significance of myki to the state means the retender is a High Value High Risk project that will be subject to oversight by the Treasurer and the Department of Treasury and Finance (DTF).

This Part assesses whether the planning for the retender is soundly based and designed to deliver a value-for-money outcome. It specifically examines:

• the adequacy of procurement planning
• whether appropriate governance arrangements have been established for the tender
• whether lessons learned from the myki contract have been considered.

As the retender is at an early stage, we have examined the procurement planning undertaken at the time of the audit.

4.2 Conclusion

The myki retender is occurring under significant time pressure due in part to previous delays in implementing myki.

The virtual absence of any contingency in the retender schedule means the tender outcome may well be resolved after the current contractual term expires. If this occurs, it risks exposing the state to significant additional costs.

PTV is taking steps to mitigate the risk of this occurring and is leveraging the lessons, where relevant, from the previous myki contract. PTV's effective implementation of related initiatives will be critical to achieving this.

The expected outcomes and benefits from the retender process have not been clearly defined. PTV needs to address this issue as it reduces the transparency of the new contract's impact, and accountability for PTV's related performance.

4.3 Procurement planning

The Victorian Government Purchasing Board (VGPB) sets policies and minimum standards for departments' procurement of non-construction goods and services. VGPB has recently moved to more principles-based policies under the procurement reform process, and PTV has committed to meeting these standards.

VGPB guidance indicates that sound procurement planning involves:

• developing a business case which establishes the need for the goods and/or services to be procured and identifies options for acquiring them, and the risks, costs and benefits associated with each option
• developing a procurement conduct plan and related strategy for implementing the preferred option—for example, a single or multi-stage tender
• establishing a sound probity framework to achieve an impartial, transparent and accountable tender process
• developing a risk management strategy detailing the mitigation strategies for managing significant risks to the procurement.

4.3.1 Developing a business case

PTV has not developed a business case for the TSR project. Instead it determined, with agreement from DTF, that the key elements of a business case would be incorporated in a detailed procurement strategy. This is because the retender is for an existing service whose need has previously been established, and does not require new infrastructure. This approach was reasonable.

4.3.2 Developing a procurement strategy

As a High Value High Risk project, the TSR procurement strategy was subject to a Gateway Review in August 2014. The review noted that the strategy was lacking in key areas. In particular, it highlighted that the identified critical success factors and benefits needed to be more specific and measurable.

PTV subsequently refined its strategy in an effort to address these issues, however, it still fails to adequately address these key business case requirements. In particular, identified benefits remain vague, and the associated measures are indicators of outputs rather than of outcomes or benefits. Consequently, the proposed measures are not sufficient for assessing the outcomes from the retender process, including implementation of the resulting contract.

Project time lines

PTV has adopted a high-risk procurement strategy as the retender is being implemented under significant time pressure.

Delays in planning and commencing the retender process have forced PTV to exercise its option to extend the current contract for a further six months in order to accommodate the tender evaluation and transition period. This has virtually eliminated any contingency in the retender schedule.

The current procurement schedule is set out in Figure 4A. The transition period from the current contractor to new contractor is planned to occur from July 2016 to November 2016 which leaves only one month—December 2016—as a contingency.

Figure 4A

Ticketing service retender project time line

Source: Victorian Auditor-General's Office based on Public Transport Victoria's Ticketing Services Retender Procurement Strategy.

PTV's procurement strategy acknowledges that the time line is challenging but considers it is achievable provided scope expectations are effectively managed. DTF's 2014 Gateway Review similarly found that the current schedule is tight and a number of activities 'appear to be behind schedule or are yet to be resourced'.

The limited contingency means there are significant risks in meeting the transition time line. PTV's procurement strategy acknowledges this issue largely affects the latter stages of the procurement—in particular the RFT response and evaluation phase, and contract negotiation and award—as there is little scope in the schedule to respond to emerging issues.

It is evident that PTV is actively monitoring the project time lines and has met its first two major milestones with the approval of the procurement strategy in October 2014, and the release of the EOI in February 2015.

Any slippage in the current schedule, however, risks incurring significant additional costs for the state as the current contract provides no option for further extensions. This means there is no certainty that this can feasibly occur. Any further extension of the transition and handover period beyond December 2016, if a new contractor is appointed, is likely to require extensive negotiations with the incumbent operator and significant additional funds to cover unanticipated costs.

4.3.3 Management of risks

PTV has prepared a Contract Strategy Framework with transition options including fall back options in the event that procurement time lines are extended beyond the current contract. These options if activated will result in substantial costs for the state if a new contractor is appointed and PTV cannot negotiate a satisfactory transition with the incumbent provider.

PTV advised that the TSR project risks and issues are managed in line with its Risk Management Framework, which is based on the Victorian Government's Risk Management Framework 2011 and consistent with AS/NZS ISO 31000:2009 Risk Management – Principles and Guidelines.

We observed that risks, including those impacting time lines, have been tabled with the TSR Steering Committee at every meeting. However, it will be critical for them to be effectively managed going forward in order to mitigate the risk of any future slippage in the tender schedule.

4.3.4 Probity

PTV has established probity procedures that largely conform to the guidelines set out in VGPB's policy for the Conduct of Commercial Engagements. This policy requires:

• accountability and transparency of the procurement process
• fairness and impartiality in carrying out the procurement
• management of actual, potential and perceived conflicts of interest
• maintaining the confidentiality and security of tender documentation and information
• attaining value for money.

Specifically, PTV developed a procurement conduct plan which reinforces these standards of conduct, and establishes clear processes for identifying and managing conflicts of interest.

PTV also engaged a probity auditor to independently assess and report on the conduct of the procurement process. However, the probity auditor also provided advice on the development of the procurement conduct plan.

Although VGPB policy and associated guidelines allow both the probity auditor and probity advisor roles to reside with a single entity, VAGO considers combining these roles at the very least, weakens the perception of independence.

PTV has subsequently appointed a separate probity advisor for the remainder of the procurement process, which is in line with better practice.

4.4 Governance arrangements

Project governance should set a firm framework to guide project success, create transparency and confidence in decision-making, and provide clarity around roles and responsibilities.

The TSR project has a well-defined governance structure. PTV has clearly documented the roles and responsibilities of key groups and individuals working on the project.

Figure 4B outlines the governance structure for the TSR project.

Figure 4B

Project governance structure

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

4.5 Lessons learned

The previous reviews of myki outlined in Part 2 of this report identify a range of issues that impacted the delivery of myki, particularly in the early project procurement phase.

The difficult history of public transport ticketing procurement in Victoria reinforces the need for the TSR project to consider the lessons from these previous procurements.

While many aspects of the issues highlighted in Part 2 relate to a project with a significant build component, it is evident that PTV has considered many of the relevant lessons to date.

Figure 4C outlines the strategies PTV advised it is implementing to avoid a reoccurrence of previous issues.

Figure 4C

PTV's consideration of lessons learned

Source: Victorian Auditor-General's Office based on information from Public Transport Victoria.

Recommendation

5. That Public Transport Victoria clarifies the retender benefits and intended outcomes and develops measurable associated indicators.

RESPONSE provided by the Chief Executive Officer, Public Transport Victoria

RESPONSE provided by the Secretary, Department of Treasury and Finance

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources