Applying the High Value High Risk Process to Unsolicited Proposals

Conclusions

DTF has not been effective in consistently and comprehensively applying the HVHR process to unsolicited proposals.

DTF's guidance falls short of ensuring a rigorous assessment of the likely benefits of unsolicited proposals and providing the transparency needed to enable stakeholders and the wider community to understand the full impacts of these proposals. These gaps are significant and evident—to different degrees—in the two unsolicited proposals examined.

Applying the HVHR process to the CityLink Tulla proposal has not resulted in the information underpinning this project meeting DTF's better practice guidelines consistently or comprehensively. When making the decision to support this proposal, government was not fully informed and DTF missed the opportunity to advise about the risks to realising the proposal's benefits and how to mitigate these.

The additional HVHR process scrutiny partly or fully assured the costs, time lines, risks, governance and project management, and procurement of CityLink Tulla. However, there was weak assurance about the deliverability of the proposal's benefits, inadequate assessment of the alternative funding options and inadequate engagement with stakeholders about the likely impacts.

In contrast, the way in which the HVHR process was applied to the Cranbourne Pakenham interim offer was much better. The proposal was subject to rigorous assessment of the costs, time lines, benefits and risks, and DTF fully advised government. This is partly because the evidence-based, long-term plans for the rail network and the Dandenong corridor provided a clear perspective on the problems that needed to be addressed and how the proponent could best contribute to improving services.

However, it remains unclear how the absence of elements essential for defining and realising benefits—such as a cost-benefit analysis and benefit management plan—would have been addressed in the Cranbourne Pakenham final offer.

The often inherent lack of competitive tension in these proposals, the likely long-term costs to the community and the significant impacts on specific stakeholders mean that DTF needs to improve:

• the level of rigour applied to assessing and realising the intended benefits
• the transparency of these proposals by requiring them to clearly communicate value for money, better justify the preferred funding approach and convey how affected stakeholders will be impacted by the proposal's costs and benefits.

The experience with Cranbourne Pakenham reinforces the importance of assessing proposals in the context of a well-developed, long-term planning framework.

Findings

Advice and guidelines on applying the HVHR process

The way that unsolicited proposals are currently developed is unlikely to consistently and comprehensively inform HVHR assessments. While DTF has improved the guidance for unsolicited proposals, it does not provide a comprehensive and adequate basis for applying the HVHR process to them.

The February 2015 Market-Led Proposals Interim Guideline marked a step forward with its commitment to greater transparency and developing a better understanding of the benefits. However, it was not underpinned by the type of detailed guidance and measures needed to do this. This is especially critical where government does not have a longer-term plan that adequately defines need and allows DTF and other agencies to see where a proposal might appropriately fit.

In terms of transparency, government has yet to finalise how it communicates the costs, funding, rationale and expected benefits of committed unsolicited proposals. Current approaches to reporting on infrastructure projects do not adequately convey this information to the community.

DTF publishes more information for public private partnerships (PPP) than for other infrastructure projects in the form of a project summary. However, this summary does not adequately explain the full costs and impacts on stakeholders nor provide sufficient assurance about the rigour of the options assessment. DTF needs to advise government how to better communicate this information for all infrastructure projects including unsolicited proposals.

For example, the private sector proponent will contribute $850 million (in September 2014 dollars) to CityLink Tulla and is expected to recover an equivalent toll revenue stream worth approximately $3.2 billion up to 2035. Providing sufficient detail to stakeholders—including the public—about the funding approach, the nature of payments and how these are calculated, and the impact on different stakeholders are critical aspects of transparency.

In terms of the rigour of advice, the guidelines are unlikely to produce proposals that provide the clarity and depth of information needed to rigorously and consistently assure that proposals' stated benefits are deliverable.

DTF needs to advise government on how to improve the unsolicited proposal guidance, its application and the public sector's role in assessing proposals, in order to:

• ensure that the essential information needed to inform government about the deliverability of a project is produced and verified
• more fully assure Parliament and the community about the rigour of accepted proposals
• better convey the financial and other impacts of the preferred option on government and other stakeholders.

Application to the CityLink Tulla proposal

The application of the HVHR process to CityLink Tulla had significant gaps which DTF needs to address for similar proposals in the future. The lack of sufficient information on the project's benefits, the absence of a full funding analysis and weaknesses in the approach to stakeholder engagement are fundamental gaps which compromise the quality of advice government is entitled to receive.

Benefits

The information provided on the project's benefits and the risks that threaten to undermine their realisation is inadequate. In reviewing the July 2014 draft business case, DTF found the discussion of benefits to be superficial and lacking validation. We agree with this assessment and did not find any material improvement in the information provided in the final April 2015 business case.

The material underpinning the business case identified potentially significant risks to the full realisation of the project benefits, and these risks were not described or mitigated in the final business case.

One area of recurrent concern is the absence of a structured and effective approach to assuring the quality of the transport modelling and the economic appraisal approach and results. Applying these models is complex and checking that assumptions are reasonable and properly applied is essential.

CityLink Tulla motorway. Photograph courtesy of VicRoads.

Over the past six months the Department of Economic Development Jobs Transport & Resources (DEDJTR) has put a governance framework in place to ensure that there are consistent assumptions underpinning the transport modelling and economic appraisal of projects, and that these have been properly applied. The formal processes to check that assumptions have been appropriately applied were agreed in June 2015 and therefore were not applied to the CityLink Tulla proposal.

It is critical that DEDJTR evaluate the effectiveness of these processes in improving the level of rigour and assurance applied to these areas for transport infrastructure projects.

Funding analysis

The value-for-money assessment for CityLink Tulla was narrow and did not incorporate a comprehensive and thorough analysis of the costs and benefits of alternative funding options. During the development process, government requested the assessment of a range of tolling options, in terms of their impacts on transport outcomes and toll revenue, while acknowledging that toll levels should reflect the benefits obtained by travellers and should avoid disproportionate impacts.

The audited agencies have been unable to justify the proposed tolling of goods vehicles as the preferred funding approach. They have not provided a full and objective assessment of a range of alternative funding options.

Stakeholder engagement

The approach to stakeholder engagement has not been satisfactory and the advice to government on stakeholder concerns has been incomplete. A stakeholder engagement strategy was not formed until after the funding approach had been decided.

DTF advised government that there was cautious industry support for funding the project through the proposed goods vehicle toll increases because the industry understands the intended benefits. VicRoads advice to us during the course of the audit was consistent with this DTF advice.

This advice is not consistent with the documentary evidence, especially the strong and public opposition by the major Victorian freight and logistics industry association—the Victorian Transport Association—to the increased tolls. Other freight industry stakeholders publicly supported the project but were not clear that they supported the funding arrangements.

Agencies need to form a stakeholder engagement strategy for government approval early in the unsolicited proposal process. This will formalise decisions on when, how and by whom stakeholders are engaged, while taking account of the risks.

Application to the Cranbourne Pakenham proposal

DTF effectively applied the HVHR process to inform government's decision about whether to proceed with this unsolicited proposal. The assessment described the progress made by the proponent but also the areas where it fell short of meeting the HVHR criteria.

DTF concluded that without an additional 12 to 16 weeks beyond the end of March 2015 deadline for the final offer, it was unlikely to pass the value-for-money and HVHR assessments. In this case, the HVHR assessment clearly fulfilled its role of informing government about whether to proceed with the proposal.

However, the process for developing this unsolicited proposal did not include a cost-benefit analysis or allocate responsibility for developing and applying a benefit management plan. These are mandatory for state-initiated projects and essential for informing government decisions and for tracking and managing benefits after delivery. It is unclear from the current guidelines how these practices should be applied to unsolicited proposals.

The major lessons to draw from Cranbourne Pakenham are:

• the ongoing importance of government decisions being informed by rigorous assessment and thorough information and advice
• the value of proposals' assessment taking place in the context of a comprehensive plan for addressing defined problems in the short, medium and long term
• the dangers of the current ambiguity about the inclusion of essential information in proposals—this needs to be better defined and the responsibility for doing this rests with DTF.

Recommendations

1. That the Department of Treasury and Finance reviews the current market-led proposals guidelines and advises government how to:

• address deficiencies in the scope and quality of information provided in unsolicited proposals where this is insufficient to fully inform High Value High Risk assessments at relevant assessment stages
• ensure the reporting requirements communicate the full impact of proposals to Parliament and the community, and provide them with greater assurance about the rigour of the analysis informing government decisions.

2. That the Department of Economic Development, Jobs, Transport & Resources evaluates the effectiveness of the governance framework recently introduced to assure the quality of transport forecasts and economic appraisals.
3. That the Department of Treasury and Finance:

• improves the level and depth of scrutiny applied to verifying benefits of all unsolicited proposals that proceed to full development
• fully assesses the viable, alternative proposal funding options for unsolicited proposals
• requires that relevant agencies define stakeholder engagement strategies for proposals that proceed to full development.

Submissions and comments received

We have professionally engaged with the Department of Treasury and Finance, the Department of Economic Development, Jobs, Transport & Resources, VicRoads and Public Transport Victoria throughout the course of the audit. In accordance with section 16(3) of the Audit Act 1994 we provided a copy of this report, or relevant extracts to those agencies and requested their submissions or comments.

We have considered those views in reaching our audit conclusions and have represented them to the extent relevant and warranted. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

Effective infrastructure planning and delivery are critical to the state's future prosperity and liveability. Government introduced the High Value High Risk process (HVHR) to address systemic weaknesses undermining agencies' performance in developing and investing in major projects.

This Part of the report:

• outlines the purpose and nature of the HVHR process
• summarises VAGO's previous audit on the HVHR process as applied to traditional, state-initiated projects
• explains how the process has been modified for unsolicited proposals
• describes the scope and conduct of this audit with its focus on how well agencies applied the HVHR process to two major unsolicited proposals.

1.2 The HVHR process

The goal of the HVHR process is to achieve more certainty over the delivery of the intended benefits of infrastructure projects to planned budgets and time lines through more rigorous review of business cases and procurements.

In introducing the HVHR process, the Department of Treasury and Finance (DTF) advised government that the priority was to:

• enforce the requirement for a robust business case with clear project objectives, well-defined benefits, a rigorous appraisal of options, selection of appropriate procurement methods and appropriate governance and management
• clearly articulate a tender proposal, appointment approach and contract management framework that appropriately allocates and manages risk, delivers benefits and effectively manages scope and cost.

For a state-initiated project, HVHR involves DTF providing additional assurance to the Treasurer at key stages of the project to address these priorities.

Figure 1A shows, for a state-initiated project, how HVHR is integrated with:

• departmental, quality-assured documentation—justifying the investment and its procurement and implementation approaches
• DTF review and advice to government—advising on the value and complexity of each investment, determining the depth and type of review applied
• the Gateway Review process—examining investments at key decision points and providing advice to the project's senior responsible person.

Figure 1A

Investment review process

Source: Department of Treasury and Finance, Investment Lifecycle and High Value/High Risk Guidelines—Overview, 2014.

1.2.1 VAGO's June 2014 HVHR audit

VAGO first audited the HVHR process and its application to five infrastructure projects in June 2014. The report, Impact of Increased Scrutiny of High Value High Risk Projects, concluded that applying the process had made a difference to the quality of the business cases and procurements of government's infrastructure investments. However, these improvements had not improved practices to the point that they consistently and comprehensively met DTF's better practice guidelines. Significant gaps remained and consequently the government was still exposed to the risk of projects failing to deliver the intended benefits on time and within approved budgets.

A key issue identified in the 2014 audit was a lack of clarity around whether all relevant projects were being included in the HVHR process, including high-risk projects valued between $5 million and $100 million, projects funded outside the typical budget process and projects arising from unsolicited proposals.

VAGO recommended that DTF improve how it:

• administers the process, addressing specific records management issues
• selects projects for inclusion in HVHR, especially for risky projects valued at less than $100 million and for projects generated as unsolicited proposals
• assesses the rigour of project business cases, especially in relation to project benefits
• evaluates the impact of HVHR and communicates learnings to departments.

DTF accepted all recommendations, with completion dates ranging from September 2014 to June 2015 for six recommendations.

In response to VAGO's recommendation that unsolicited proposals valued over $100 million be subject to HVHR processes, DTF committed to review the guideline and advise government by September 2014 on how HVHR should be applied.

1.3 Developing unsolicited proposal guidelines

An unsolicited proposal is one made to government by the private sector to build infrastructure and/or provide services. It involves the private sector seeking government support for proposals based on basic project specifications.

Government wants to realise the benefits of innovative ideas and unique options that offer great value, while continuing to apply rigorous assessment processes to prove these proposals are viable, and be assured that the expected benefits are deliverable within forecast costs and time lines.

The support being sought in these proposals is typically financial but may also include regulatory or other forms of assistance. Government may agree to advance the proposal in exclusive negotiations if the proponent can demonstrate that it alone can best deliver value for money and benefits that are valuable to government but it may also decide to go to a competitive tender process.

Proposals are assessed for uniqueness which might involve a proponent:

• having a unique idea or the use of Intellectual Property critical to the proposal
• holding a unique position or advantage, for example holding the rights under an existing contract, owning land, technology or software, or having exclusive access to or control over strategic assets.

DTF published the first Unsolicited Proposal Guideline in February 2014. The government elected in late 2014 replaced this with the Market-led Proposals Interim Guideline in February 2015. These, together with an August 2014 update about how the HVHR process should be applied to these proposals, are described in the following Sections.

1.3.1 February 2014 Unsolicited Proposal Guideline

This guideline aims to provide 'a transparent framework that sets out a clear and consistent process for project assessment and contracting where private sector parties approach government with an unsolicited proposal'.

The government's objectives for these proposals include:

• developing innovative ideas with the private sector where appropriate
• ensuring an open, transparent and fair process that involves a high standard of probity and public accountability
• incorporating open competition wherever possible
• ensuring value for money for government is achieved
• ensuring projects' benefits for government are measurable and can be maximised
• ensuring the private party's intellectual property is respected and that private parties are fairly compensated as part of the process.

It also stipulates minimum requirements that a proposal will only be pursued where:

• there is a need and appetite for the project that is consistent with government policy objectives and in the public interest
• it is financially, economically and socially feasible and capable of being delivered
• it has a degree of uniqueness that means it could not reasonably be delivered by another private party or achieve the same value for money through a competitive process within acceptable time frames
• it represents value for money and provides benefits to government.

Figure 1B summarises the guideline's five-stage process for achieving these things.

Figure 1B

Summary of Unsolicited Proposal Guideline process

Source: Victorian Auditor-General's Office.

1.3.2 August 2014 incorporation within HVHR process

Following VAGO's recommendation that unsolicited proposals over $100 million be subject to HVHR, DTF proposed and government endorsed how unsolicited proposals should be integrated with the HVHR and Gateway Review processes.

The HVHR process should be applied to unsolicited proposals that:

• have a total estimated government investment of more than $100 million, or
• are identified as high risk using an approved risk assessment tool, noting that the focus would be on risk to government and residual risk, or
• government determines warrant the rigour of increased oversight.

DTF published a diagram showing the integration to supplement the February 2014 guidance and we examine the quality of this guidance and the analysis underpinning it in Part 2 of the report.

One of the challenges examined in Part 2 is how to apply HVHR requirements designed for conventional state-initiated projects to the process for developing unsolicited proposals. For state-initiated projects, portfolio agencies develop business cases and then tender documentation to inform the HVHR assessment.

In contrast, proponents of unsolicited proposals provide this information in their proposals, with government oversight agencies playing a greater role in helping them define the investment logic and the proposal's benefits. It is intended that the full set of information for the HVHR assessment and to support government decisions is built up throughout the stages both by the private sector and the public sector and generally is developed later in the process than for a conventional state-initiated project.

1.3.3 February 2015 Market-led Proposals Interim Guideline

The Unsolicited Proposal Guideline was updated in February 2015 by the new government which was elected to Parliament in November 2014.

The updated guideline provides more detail on the process but does not fundamentally change government's approach to unsolicited proposals. The objectives are similar and the five-stage approach has been retained.

The government wanted to provide additional transparency by explaining in more detail what the process involved and making some changes to its operation. These changes include:

• encouraging proponents to conduct a pre-submission meeting with government to discuss requirements and information expectations
• more clearly mandating whether a competitive approach should be pursued during the stage 2 preliminary assessment of the proposal
• providing more detail on the criteria used to determine uniqueness and on how value for money would be assessed through both quantitative and qualitative assessments of costs, risks and intended benefits
• more detailed requirements around probity planning
• earlier public disclosure of the unsolicited proposals government is considering.

1.4 Applying HVHR to two unsolicited proposals

May 2014 Budget Papers identified that government was progressing two unsolicited proposals that would be subject to the HVHR process—the CityLink Tulla Widening project (CityLink Tulla) and the Cranbourne Pakenham Rail Corridor project (Cranbourne Pakenham).

Figure 1C summarises the development of these proposals alongside the dates when DTF published its guidelines.

Figure 1C

Time line of process and project development

Note: DEDJTR = the Department of Economic Development, Jobs, Transport & Resources.

Source: Victorian Auditor-General's Office based on DTF documentation.

1.4.1 CityLink Tulla Widening project

This project aims to build capacity, boost performance and improve safety on one of Melbourne's busiest roads.

The government received an unsolicited proposal in August 2012 to widen the CityLink freeway up to Bulla Road, to be funded through increased tolls. The government expanded the project to incorporate state-initiated works which expand the upgrade to cover a total of 23.8 kilometres of the freeway between the Burnley Tunnel and Melbourne Airport.

In October 2014, government entered into a contract for the proposal. Following the November 2014 election, government revised the proposal to take into account the cancelled East West Link project. In April 2015, government signed a contract with the proponent to fund and deliver works between the Burnley Tunnel and Bulla Road and to partly fund works between Bulla Road and Melrose Drive. The section of freeway between Melrose Drive and the airport is jointly funded by the Commonwealth and the state.

CityLink Tulla, Bolte Bridge. Photograph courtesy of Nils Versemann/Shutterstock.com.

Figure 1D includes the unsolicited work—section 3—and state-initiated works—sections 1 and 2.

Figure 1D

CityLink Tulla scope

Project boundaries Project scope Works Create extra lanes in each direction through a combination of new construction and the use of emergency lanes (section 3). Introduce a freeway management system used to control ramp access and lane management. Junction upgrades to improve capacity and merging. Unsolicited proposal funds sections 2 and 3 with the proponent delivering section 3. VicRoads delivering sections 1 and 2. Costs and funding The $250 million section 1 costs funded by Commonwealth and state government contributions. Section 1 is outside of the scope of the unsolicited proposal. The costs at May 2015 were: section 2—$214.7 million section 3—$657.5 million. Of the total of $872.2 million, $851.1 million comes from toll revenue and $21.1 million, from the government. The proponent will recoup this outlay by: raising toll charges for commercial vehicles extending the period of time when tolls can rise by more than CPI accommodating more traffic as a result of the improvements. Project development and assessment Sections 1 and 2 are similar to normal state-initiated projects. Both are developed through approval of a business case subject to the HVHR process. The approval of section 3 was based on review of a full proposal. The scope and costs were revised early in 2015 following the East West Link cancellation. Final approval came in late April 2015.

Source: Victorian Auditor-General's Office based on information from VicRoads.

1.4.2 Cranbourne Pakenham Rail Corridor project

In March 2014, the government announced the Cranbourne Pakenham Rail Corridor project as the first project to be progressed under the new unsolicited proposal guidelines. The project was a $2 to 2.5 billion—approximately $5 billion in total service payments—initiative to increase capacity on the rail corridor. The proponent under the Project Finalisation Agreement comprised the same entities as the shareholders of the current metropolitan rail franchise.

Figure 1C shows that, a year later in March 2015, government rejected this unsolicited proposal based on an evaluation of the interim offer which raised issues about the project's alignment with the government's broader transport policy commitments.

Figure 1E describes the proposal scope.

Figure 1E

Cranbourne Pakenham scope

Cranbourne and Pakenham lines Project scope Works The design, construction, financing and maintenance of: 25 new high capacity trains a new train maintenance depot at Pakenham East upgraded conventional signalling across the corridor and initial deployment of high capacity signalling on the Cranbourne to Dandenong section. The design, construction, financing and transfer to the state of: power upgrades removal of four level crossings at Murrumbeena, Koornang, Clayton and Centre roads minor modifications to assets along the corridor. Procurement The special purpose vehicle comprising the same entities as the shareholders of the current metropolitan rail franchise. Competitive processes for packages overseen by the special purpose vehicle. Public private partnership type arrangement over a 15-year period and regular payments. Cost $2.5 billion estimated cost discounted to today. $2.4 billion estimated procurement capital costs. $4.2 billion to $5.2 billion total service payments over the life of the project. Project development and assessment Developed through a series of proposals from the consortium. Approval conditional on proponent meeting gates and milestones set out in the Project Finalisation Agreement. HVHR and Gateway Reviews to inform government's decision to proceed to contract. Value for money and HVHR assessment of interim offer, final offer and contract.

Source: Victorian Auditor-General's Office based on information from DTF.

1.5 Audit objective, scope and criteria

The objective of this audit was to assess whether the HVHR process has been effectively applied to unsolicited proposals identified as HVHR projects by examining:

• DTF's advice to government about how to apply HVHR to unsolicited proposals
• the application of HVHR to the CityLink Tulla and Cranbourne Pakenham unsolicited proposals.

Government identified these proposals as subject to the HVHR process in the May 2014 Budget. We viewed the application of HVHR to these proposals as critical given their size, innovative development and procurement approach, which involved negotiations with single proponents.

In examining CityLink Tulla, we extended the scope of this audit to cover the application of the HVHR process to both the unsolicited proposal and state-initiated works because these components, and the material supporting their development and funding, are so closely connected. The audit time frame spanned the period of proposal negotiation and assessment conclusion for both unsolicited proposals.

The government agencies engaged in this audit were the Department of Treasury and Finance, the Department of Economic Development, Jobs, Transport & Resources, VicRoads and Public Transport Victoria.

1.6 Audit method and cost

The audit team reviewed the documents held by the audited agencies and those generated by the government's consideration of the projects. The team also interviewed personnel responsible for the review and oversight of these projects.

The audit was conducted in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

Pursuant to section 20(3) of the Audit Act 1994, any persons named in this report are not the subject of adverse comment or opinion.

The total cost of the audit was $485 000.

1.7 Structure of the report

The report is structured as follows:

• Part 2 examines the advice provided to government by DTF in applying VAGO's June 2014 recommendation about how unsolicited proposals should be subject to the HVHR process
• Part 3 examines the HVHR process as applied to the CityLink Tulla proposal
• Part 4 examines the application of the HVHR process to the Cranbourne Pakenham proposal.

2.1 Introduction

This Part examines whether the Department of Treasury and Finance (DTF) has adequately responded to VAGO's June 2014 recommendation to subject unsolicited proposals over $100 million to the High Value High Risk (HVHR) process, by providing government with clear advice based on a comprehensive analysis of the issues.

In August 2014, government approved that the criteria used to include state-initiated projects in the HVHR process should be applied to unsolicited proposals. DTF presented government with a diagram to show how the unsolicited proposal, Gateway Review and HVHR processes would be integrated and this is included as Figure 2A.

In February 2015, the new government asked DTF to revamp its guidelines and communicated them in the Market-led Proposals Interim Guideline. Figure 1A in the Background Part of this report describes the major changes.

This Part of the report describes the challenges of applying the HVHR process to unsolicited proposals and assesses the clarity and comprehensiveness of the changes made to the guideline to incorporate the HVHR process.

2.2 Conclusion

The way that unsolicited proposals are currently developed is unlikely to provide all the information needed to inform an HVHR assessment. Unsolicited proposals do not typically define their intended outcomes in terms of the service benefits delivered to the community, prove that these benefits will be achieved, or encompass the full range of risks that are relevant to government. This contrasts with a full business case for a conventional, state-initiated project, which requires all of this information.

While DTF acted swiftly in August 2014 to implement VAGO's June 2014 audit recommendation, it missed the opportunity to properly address this issue when advising how best to integrate unsolicited proposals with the HVHR process.

The February 2015 Market-Led Proposals Interim Guideline represents an important step in the right direction. It places more emphasis on developing a good understanding of the benefits, however, it fails to explain how this is to be achieved. There is a lack of clarity about who is responsible for identifying benefits and there is a continued risk that government decisions will not be adequately informed by detail of the expected benefits.

DTF needs to advise government how to address this gap and supplement its guidance so that agencies and potential proponents are aware of the requirements and their responsibilities.

2.3 Challenges in applying the HVHR process

The requirements for informing government decisions on supporting projects should be the same irrespective of whether it is a government-funded project developed by a department through the budget process or an unsolicited proposal initiated outside of government. Government needs to have confidence that a project will deliver clearly defined benefits for Victorians for the estimated cost and within expected time lines, while effectively managing the risks.

Over the past decade DTF has worked to improve the way state-initiated projects are developed because this is critical to fully informing government decisions and delivering successful projects. The Investment Lifecycle Guidelines and more recently the HVHR process are designed to help agencies better define, develop and deliver predictable and valuable projects.

The private sector is likely to be focused on its own commercial interests in generating a proposal. This creates the risk of a diminished focus on understanding, defining and realising the service outcomes (or benefits) critical to the community. A recurrent finding across a wide range of VAGO audits is that agencies are poorly demonstrating the benefits of projects.

DTF advised that it is aware of these issues and does not expect proponents to incorporate the same breadth of information in the early stages of the project development process. Private proponents are likely to be wary of investing too much up front when it is uncertain whether government will support their proposals. Instead DTF understands that more of the essential groundwork will happen later in the process for these proposals compared to conventional state-initiated projects.

DTF is also aware that proponents are likely to focus on the costs, risks and timing of the outputs they plan to deliver. DTF sees a role for portfolio agencies in helping shape and clarify the connection between these outputs and the improved service outcomes that are the final goal and of most importance to government. In turn, there is a role for DTF in undertaking HVHR reviews of proposals to ensure that this work has taken place and that the service outcomes and benefits are clearly demonstrated before government commits its support.

The highest standards of transparency need to be applied to proposals because:

• project ideas are generated by private companies that are commercially driven
• confidentiality concerns mean these ideas may be developed without the same level of stakeholder engagement as conventional state-initiated projects have.

DTF has not yet finalised the reporting requirements. However, simply replicating the information provided in recent summaries for public private partnership projects is unlikely to adequately convey the full costs and impacts on stakeholders, or provide assurance about the rigour of the option's assessment.

The unsolicited proposals included in this audit experienced these challenges and we expected DTF to advise government about these issues and provide clarity about how applying HVHR would address them.

2.4 August 2014 update of the guidelines

DTF updated its guidance in August 2014 by publishing Figure 2A, showing the integration of the unsolicited proposal and HVHR requirements. However, it could not demonstrate that this work included a comprehensive analysis of the risks and issues around the completion of HVHR assessments for unsolicited proposals. The advice to government did not adequately convey how these risks should be mitigated.

2.4.1 Analysis underpinning DTF's advice

The documents describing DTF's work to address VAGO's June 2014 recommendation are insufficient for us to see that it comprehensively analysed the issues.

DTF advised that it considered the risks and practical challenges in applying the HVHR process to unsolicited proposals and how to overcome them, by exploring:

• how to best align the HVHR and Gateway Review processes with the stages of assessment and delivery in the Unsolicited Proposal Guideline
• criteria to classify unsolicited proposals as HVHR, including by the size and type of contribution from government and by the scale and type of risks that government would be exposed to.

From the documentary evidence provided, DTF could demonstrate that it:

• consulted with the DTF HVHR committee in June 2014
• consulted with the Unsolicited Proposals Interdepartmental Committee(IDC) in July 2014, presenting a draft diagram for integrating the Gateway Review, HVHR and unsolicited proposal review processes
• further consulted the IDC to confirm a revised approach as shown in Figure 2A
• presented to and gained government's approval for this approach in August 2014
• published Figure 2A on its website where it remains a current document.

However, these documents did not show that DTF examined the practical challenges of applying the HVHR process to unsolicited proposals.

We did not find evidence showing how DTF determined that the existing criteria for including state-initiated projects in the HVHR process should be applied to unsolicited proposals. The effectiveness of this approach depends on the thorough application of a risk assessment tool for proposals requiring less than $100 million in state funding as these are not automatically subject to HVHR processes.

2.4.2 Advice to government

DTF provided advice to government in August 2014 recommending an approach to apply HVHR assessment to unsolicited proposals. This advice took the form of Figure 2A together with a brief providing the background and rationale for the proposed approach.

The main problem with the advice is that it did not raise the practical challenges of applying the HVHR criteria to unsolicited proposals and did not describe how DTF would address these challenges.

Figure 2A

Integration of the unsolicited proposals, HVHR and Gateway Review processes

Source: Department of Treasury and Finance.

2.5 February 2015 market-led proposals guideline

Changes in the 2015 Market-led Proposals Interim Guideline have the potential to address the issues we have identified by providing more direction on building a robust case for investment and how these will be assessed. However, the guidance still lacks detail around defining the intended outcomes and allocating clear responsibility between the proponent and government agencies for their achievement.

Under the revised guidelines:

• proponents are required to articulate the benefits of interest to the state, including how the proposal is in the public interest, at Stage 1—the initial proposal
• government must complete a qualitative and quantitative assessment of value for money that includes verifying these benefits at Stage 4—negotiating and finalising a binding offer.

An HVHR assessment of an unsolicited proposal should provide assurance about the deliverability of the intended benefits and the risks that threaten their realisation. It remains unclear who is responsible for generating the information needed to gain this type of assurance.

In presenting the revised guideline to government for approval, DTF did not specifically raise the significant difficulties in practically translating aspects of the HVHR assessment to unsolicited proposals. The advice highlighted the following key lessons from administering the previous guidelines with the need to better:

• explain to proponents how to show their proposals are unique, offer value for money and align with government priorities
• manage proponents' expectations, and draw their attention to the possibility that proposals may not proceed
• consult with government prior to submission to improve proponents' process understanding.

DTF recognises the need to learn from the early unsolicited proposals and improve the process to address clear gaps and weaknesses. As at 30 June 2015 DTF advised us that of 45 proposals received, only five proposals have proceeded past Stage 2 and only one proposal has progressed to contract award at Stage 5. As a result, there have been limited opportunities to learn lessons from the early implementation.

However, for those proposals that do progress, DTF recognises that it needs to address the issues around the clarity and assurance of the benefits that will be delivered to the community. Unsolicited proposals, like conventional state-initiated projects, need to:

• clearly define the investment logic, linking the proposal to the problems that need addressing and the benefits to the community that are valued by government
• develop an understanding of these problems and benefits to a depth and quality equivalent to what is meant to be provided in a final business case.

DTF needs to advise government about how best to do this so that quality information is available in time to inform its decisions throughout the proposal development.

Recommendation

1. That the Department of Treasury and Finance reviews the current market-led proposals guidelines and advises government how to:

• address deficiencies in the scope and quality of information provided in unsolicited proposals where this is insufficient to fully inform High Value High Risk assessments at relevant assessment stages
• ensure the reporting requirements communicate the full impact of proposals to Parliament and the community, and provide them with greater assurance about the rigour of the analysis informing government decisions.

3.1 Introduction

This Part examines whether the High Value High Risk (HVHR) process has been effectively applied to the CityLink Tulla Widening project (CityLink Tulla) to provide rigorous and comprehensive advice to government about whether and how the proposal should proceed.

In this Part we assess the quality of the advice given about the proposal's deliverability and procurement, including whether government can be assured the proposal will be:

• delivered on budget
• delivered on time according to realistic time frames
• successful in delivering and sustaining the intended benefits
• comprehensively risk managed
• governed and managed effectively
• procured in a way that represented value for money by delivering intended benefits according to planned time lines and costs.

Our analysis covers both the unsolicited proposal—focused on the southern tolled section of the freeway—and the state-initiated works—between Bulla Road and Melbourne Airport—because of the connections and dependencies linking these components.

In October 2014, the previous government decided to proceed with the unsolicited proposal based on advice that it met the HVHR deliverability and procurement criteria. The change of government in November 2014 stopped the execution of this decision. However, in April 2015 the new government decided to proceed with the proposal and the state-initiated works based on a final business case, updated value-for-money assessment and a final HVHR assessment confirming the project's deliverability.

3.2 Conclusion

Applying the HVHR process has not resulted in the information underpinning the project's approval consistently and comprehensively meeting the Department of Treasury and Finance's (DTF) better practice guidelines. While the additional HVHR scrutiny improved assurance about costs, time lines, risks, governance and project management, and procurement, significant gaps remain.

These gaps and shortfalls in information include:

• the lack of sufficient information on the project's benefits, reliability and the risks that threaten to undermine its sustainability and realisation
• the narrowness of the value-for-money assessment and the lack of a comprehensive analysis of the costs and benefits of alternative funding options
• an inadequate approach to stakeholder engagement and the lack of a clear, coherent and endorsed engagement strategy early in the proposal's development.

These gaps are significant and mean that agencies have not fully informed government of the full range of issues and risks relevant to its decision-making.

3.3 Overall assessment

We tested the quality of the assurance provided by the HVHR review of CityLink Tulla against the criteria set out in relevant Investment Lifecycle and High Value/High Risk Guidelines criteria. Our assessment is based on the documentation provided by DTF, VicRoads and the Department of Economic Development, Jobs, Transport & Resources (DEDJTR). DTF assessed the HVHR criteria as assured and Figure 3A describes our assessment against these criteria.

Agencies applied the greatest amount of scrutiny and achieved partial or full assurance for project budgets, time lines, governance and project management, and procurement. In contrast, there are unaddressed issues that prevent us from being fully assured about the deliverability of benefits. These issues vary in their significance and are described in greater detail within the body of this Part.

However, we found a lack of assurance for the delivery of intended benefits. The documentation used to justify the project did not adequately explain the benefits, their distribution across stakeholder groups or the risks to their scale and sustainability. DTF applied insufficient scrutiny to test the project benefits and so the deficiencies of earlier business cases were not addressed in the final advice that informed government's decision to approve the project in May 2015.

Apart from the project benefits, there were other significant gaps in analysis and advice, including the inadequate:

• assessment of alternative funding options and communication of the impacts for stakeholder groups and government's wider policy objectives
• approach to consultation in terms of defining a clear approach to stakeholder engagement at the outset of the process, and adequately consulting affected groups on the impacts of locked in decisions.

Figure 3A

VAGO's assessment of final offer and business case against HVHR requirements

Note: ✔✔✔ = Fully assured—Project documentation and advice to government fully addresses Investment Lifecycle and HVHR Guidelines and provides a high level of assurance.

✔= Partly assured—Evidence that HVHR reviews improved the level of assurance while not fully addressing all issues. The significance of outstanding issues varies from relatively minor—business case is not updated with additional evidence—to more significant—where material issues are not raised or addressed. We provide a commentary on the issues.

✘ = Not assured—Insufficient evidence to be assured about the deliverability of the project.

Source: Victorian Auditor-General's Office.

3.4 Deliver on budget

For a project to pass this HVHR criterion, DTF needs to be confident that estimated costs are based on a verified and detailed understanding of base costs, with evidence-based allowances for contingencies and risks. Costs should be:

• appropriately structured, clearly describing capital and ongoing whole-of-life costs and distinguishing base cost, base risk and contingency risk allocations
• reliable by being:
  • based on a clear and appropriately defined project scope and design
  • estimated by competent estimators and checked against benchmark data
  • estimated using a recognised and rigorous approach to costing
  • subject to a robust review process.

Figure 3B shows that we found that reviews of the project's costs had improved assurance. We found the level of assurance to be greater for the state-initiated works than for the unsolicited proposal. Figure 3B describes the outstanding issues.

Figure 3B

Findings on delivering on budget

Source: Victorian Auditor-General's Office.

3.5 Deliver on time

The Investment Lifecycle and HVHR Guidelines require the business case to include a detailed and realistic project implementation schedule that encompasses key milestones, decision points and delivery events. It also requires information on competing priorities, dependency analysis, required skills and capabilities, the availability of agency staff and on the public communication of time lines.

Figure 3C shows that we rated the reviews as providing full assurance for all three sections of the project. The combined procurement of sections 1 and 2 puts pressure on achieving the target completion date for section 2. However, we are assured that sufficient mitigation measures are in place, including clear requirements embedded in the contract that allocate the risk of achieving this completion date to the contractor.

Figure 3C

Findings on project time lines

Source: Victorian Auditor-General's Office.

3.6 Achieving the intended benefits

3.6.1 Overall assessment

Delivering benefits to the community is the key reason for investing in public infrastructure. Managing projects to realise, sustain and maximise benefits is therefore essential. This requires an evidence-based understanding of the expected benefits, their distribution and the risks to their realisation, together with clear advice about how best to address these risks.

There is sufficient evidence for us to see that this project is likely to deliver significant, net benefits to car travellers. However, proving the intended benefits and comprehensively assessing and advising government about the risks, are the weakest parts of the HVHR assessment. This replicates the findings from our 2014 HVHR audit, meaning that assuring the delivery of intended benefits continues to be the weakest link in the application of the HVHR process.

Figure 3D shows a 'not assured' rating for the HVHR review of benefits in relation to both the state-initiated works and the unsolicited proposal. Further details on these findings are in Sections 3.6.2 and 3.6.3.

Figure 3D

Overall findings on project benefits

Source: Victorian Auditor-General's Office.

3.6.2 Understanding the problems and benefits

The Investment Lifecycle and HVHR Guidelines require proponents to define and measure the problems being addressed and the intended benefits. Agencies need to:

• show that they understand the causes and effects of the problems driving the investment, including the groups affected and the nature of these problems
• clearly define the project's benefits and any likely disadvantages, describing the impacts on different stakeholders and how they intend to measure and report on the achievement of the project's objectives.

Agencies have not demonstrated a thorough understanding of the problems and expected benefits essential to informing decisions about if and how the project should proceed and providing confidence that the outcomes will be monitored and managed.

Understanding the problems driving the project

The definition of the problem in the draft and final April 2015 business cases is at too high a level to fully convey the evidentiary basis for the project and its expected positive and potentially negative impacts.

We acknowledge that detailed documentation supporting the business case, such as the detailed traffic impact assessment for section 3, partly fills this gap. However, the key findings from this analysis have not been adequately conveyed in the business case. It does not describe the congested locations, on or adjacent to the project, where the assessment identifies the risk that current problems may persist or intensify after implementing the project.

Figure 3E illustrates this lack of detail, using road safety as an example.

Figure 3E

Scrutiny of the current and expected safety problems

Source: Victorian Auditor-General's Office based on material from VicRoads.

Defining and monitoring the project's expected benefits

Agencies did not address DTF's July 2014 review findings that the draft business case:

• discussion of anticipated benefits was superficial and lacks validation through the provision of supporting evidence
• did not include a benefit management plan that adequately explained how expected benefits would be monitored and managed.

We agreed with these findings and the April 2015 final business case did not address them. Despite this, DTF did not raise them in advising government on the project's performance against the HVHR criteria.

The April 2015 business case is inadequate because:

• benefits are presented as aggregate, discounted monetary sums with insufficient detail on the travel time, operating cost, reliability and safety changes underpinning these estimates and the impacts on different stakeholders
• it is unclear how long it is likely to be before rapid traffic growth significantly erodes the travel time and reliability benefits provided by the project
• it does not describe the potential disadvantages of:
  • greater traffic volumes affecting congestion upstream or downstream of the project
  • making commercial vehicles bear the majority of the project cost through increased tolls and how this affects commercial vehicle productivity
  • the operation of ramps and junctions with ramp metering and increased traffic.

The final business case includes an expanded benefit management plan. However, we are not assured of its quality because we are not convinced that:

• measures are appropriate—for example, measuring average travel times and travel time variability over a three- to four-hour period for trips traversing the entire project will not be sufficient to understand and manage emerging impacts
• targets are soundly based—they are not explained and stop after five years
• targets and measures are comprehensive—absence of measures for individual ramps and sections to show localised problems and bottlenecks
• a detailed plan is in place for how data will be generated, analysed and reported
• sufficient information will be collected to understand the safety impacts.

3.6.3 Reliability of the traffic and benefit estimates

VAGO's 2011 audit on the Management of Major Road Projects concluded that VicRoads had fallen short of the standards required to reliably forecast traffic and estimate projects' economic benefits when informing government decisions. We viewed addressing these weaknesses as critical because decisions had been reached without a full appreciation of the likely consequences.

While VicRoads has improved its approach to forecasting traffic and estimating economic benefits, it still lacks a structured and transparent approach to quality-assuring the results. In addition the CityLink Tulla business case did not explain how significant forecasting uncertainties are being managed. DTF did not scrutinise these issues.

Quality assurance

None of the versions of the business case adequately explain how the application of the transport model and the economic appraisal framework were quality assured.

The review comments from DTF and the former Department of Transport, Planning and Local Infrastructure reveal a superficial examination of the traffic forecasts and economic benefits. This was insufficient to verify these have been:

• based on reasonable approaches consistent with national and state guidelines
• applied and calculated as intended.

In past audits on the management of major road and rail projects, the absence of structured and rigorous quality assurance by portfolio and central agencies allowed significant errors and inconsistencies to go undetected.

Transport modelling and economic benefit uncertainties

The information presented to government did not convey the significant uncertainties around the scale and sustainability of benefits. These issues, and options to address them, should have been communicated. Instead the business case and HVHR assessment conveyed a positive and unambiguous view of the benefits.

Past examples of road upgrades which did not meet short-term performance expectations illustrate the potential consequences of ignoring these risks:

• The 2007 Pakenham Bypass upgrade was designed to provide enough capacity for 20 to 25 years but had reached this limit by 2010.
• The 2009 upgrade of the Bolte Bridge and West Gate Freeway junction led to peak traffic queues across the Bolte Bridge that did not meet community expectations.

We identified material that merited inclusion in the business case and scrutiny by DTF in providing a balanced commentary to government on the deliverability of the project's benefits. The key documentary sources included:

• a VicRoads commissioned traffic impact assessment of sections 2 and 3 of the project identifying a range of capacity, safety and other performance issues
• a 2014 review of the strategic transport model used for CityLink Tulla commissioned by the former Department of Transport, Planning and Local Infrastructure as part of the East West Link project, which identified significant sources of uncertainty relevant to the traffic forecasts.

Figure 3F summarises the issues raised in these reports. The approach to forecasting traffic for CityLink Tulla is an improvement on the past because it takes into account induced traffic—extra traffic to a road generated by the improvements. However, significant sources of uncertainty remain and these have not been examined and communicated to government.

The capacity provided is likely to be exhausted well before the end of the 30-year evaluation period. The business case provides no advice about when congestion will again become critical and what government should do to prepare for this.

The April 2015 Gateway Review, Gate 2, identified that traffic forecasts may be conservative, with the risk that actual traffic will result in saturation earlier than anticipated. It advised that care is needed in communicating the expected project benefits.

In our discussions with DEDJTR, VicRoads and DTF there was a sense of inevitability about reaching this saturation point and an acceptance that agencies and government would have determined what to do next in advance of this.

Figure 3F

Overall findings on project benefits

Source: Victorian Auditor-General's Office based on information provided by VicRoads and East West Link documentation.

3.7 Comprehensively manage risks

DTF's Investment Lifecycle and HVHR Guidelines require the business case to:

• clearly identify and assess key risks associated with the proposed investment
• describe the proposed approach to ongoing risk monitoring and management
• document a risk management strategy.

The guidelines describe a broad range of risks to which these requirements apply.

Risk management is rated as not assured for the risks that threaten the realisation of the project's intended benefits and as providing partial assurance for the remaining risks in light of the residual gaps that need to be addressed.

3.7.1 Risks to intended benefits

Section 3.6 described the lack of HVHR scrutiny and advice in relation to the risks to intended benefits. Agencies did not adequately advise government about potentially significant risks. This explains our not assured rating.

3.7.2 Assessing assurance for other sources of risk

Figure 3G shows partial assurance for sections 1 and 2 and full assurance for section 3 because of contractual arrangements with the proponent.

It also summarises DTF's HVHR assessment and the remaining gaps that reduce the level of assurance. Section 3.7.3 provides more detail on these remaining gaps.

Figure 3G

Findings on risk management

Source: Victorian Auditor-General's Office and DTF documentation.

3.7.3 Further detail on risk management gaps

The gaps described below were not adequately communicated in the final business case, HVHR assessments and briefings provided to government in April 2015.

Project and risk management documentation

The final business case includes an extensive risk register, summarising a range of risks and short descriptions of proposed mitigations. The depth of this information falls short of the requirement for a comprehensive risk management plan (or strategy).

The April Gateway Reviews for Gates 2 and 4 noted the absence of a formal project plan and emphasised the urgent need for a detailed plan and sub-plans to be developed given the proximity of the project delivery phase.

These reviews also found that the multiple risk registers reviewed for components of the project needed to be integrated at a whole-of-project level. The reviews recommended the engagement of an experienced practitioner to administer the risk management process during project delivery.

Approach to stakeholder engagement

The approach to stakeholder engagement has not been satisfactory and the advice to government about stakeholder concerns has been incomplete. Based on the proponent's consultation with, the major Victorian freight and logistics industry association, the Victorian Transport Association (VTA), RACV, Taxi Services Commission, Australian Logistics Council, Toll Group and Linfox, DTF's April 2015 HVHR assessment of the unsolicited proposal states that:

'In general the reaction has been a cautious support to the proposed Heavy Commercial Vehicle toll changes as a funding mechanism for the Project as the industry understands the intended benefits'.

Our review of the available evidence suggests this statement is potentially misleading because it is critical that agencies form a stakeholder engagement strategy early in the unsolicited proposal process so that decisions on when, how and by whom stakeholders are engaged are formalised based on an objective analysis of the risks.

VicRoads documented an engagement strategy and started consulting on the project in April 2014. At this stage many critical decisions, such as the funding arrangements, had been set. This risked alienating stakeholders and not understanding the implications for the community and for government's policy objectives.

VicRoads advice to us during the course of the audit was consistent with this DTF advice that there was cautious industry support for funding the project through the proposed goods vehicle toll increases because the industry understands the intended benefits. This contradicts the documentary evidence, especially the strong and public opposition by VTA to the increased tolls. VicRoads brought to our attention announcements by other freight industry stakeholders publicly supporting the project but these were not clear in their support of the funding arrangements.

Figure 3H

Freight industry views on project funding

Source: Victorian Auditor-General's Office, VicRoads data and VTA media releases.

3.8 Applying sound governance and management

DTF's Investment Lifecycle and HVHR Guidelines require:

• demonstration of an appropriate governance structure for the proposed solution
• people appointed to key positions such as a steering committee to be identified
• a robust and up-to-date project management strategy including deliverables, dependencies between activities and monitoring and review arrangements
• identification of a senior responsible officer for the project.

CityLink Tulla motorway. Photograph courtesy of VicRoads.

Figure 3I shows that there is partial assurance for sections 1 and 2 and full assurance for section 3 because of the contractual arrangements with the proponent.

Figure 3I

Findings on project governance and management

Source: Victorian Auditor-General's Office.

3.9 Procuring for best value for money

The Investment Lifecycle and HVHR Guidelines require a range of procurement options to be considered with sufficient information provided to demonstrate the basis for selecting the preferred procurement approach. Documentation should describe:

• the process, method and analysis for selecting the preferred procurement option
• the recommended procurement option, clearly explaining why it offers the best value for money
• a high-level procurement strategy to implement and deliver the works.

This Section also examines the value for money. While the scrutiny applied has resulted in improved assurance, this area is not rated as fully assured because of some specific procurement issues and a narrow value-for-money analysis.

3.9.1 Procurement assessment

Figure 3J shows that the HVHR assessment improved the level of assurance for the procurements of both the state-initiated works and unsolicited proposal. The table identifies the issues that prevented a rating of fully assured.

Figure 3J

Findings on project procurement

Source: Victorian Auditor-General's Office.

3.9.2 Value-for-money assessment

Overview

The Investment Lifecycle and HVHR Guidelines define value for money as 'an assessment of...the cost of obtaining goods and services against the benefits they provide'.

The guidance applying to the business case emphasises the importance of considering a range of options and justifying the recommended project solution as the 'best value for money way to respond to the problem and deliver the expected benefits'.

The guidance on procurement explains that the value-for-money assessment should:

• balance the costs of a good or service against a range of attributes, including quality, performance standards, suitability, risk exposure, policy alignment, timeliness, convenience, resource use, and social and environmental impacts
• be assessed on a 'whole of life' or 'total cost of provision' basis, where the long-term costs and viability and sustainability of an investment are considered in conjunction with the immediate budget implications.

Figure 3K summarises the value-for-money analysis. The most significant gap is the absence of a comprehensive analysis of alternative funding options. The way this project is funded is likely to significantly affect the scale and distribution of benefits and affect the economic, social and environmental outcomes.

The evidence does not show that the options were adequately assessed and the impacts for specific stakeholders communicated to government.

Figure 3K

The state's approach to measuring value for money

Source: Victorian Auditor-General's Office, VicRoads data and VTA media releases.

Assessing the impact of alternative funding options

In 2014 government endorsed the following tolling principles to inform the East West Link tolling strategy and the setting of tolls on other Melbourne toll roads:

• 'improve transport outcomes by optimising asset utilisation and balancing traffic flows across the transport network'
• 'maximise the value of toll revenue to secure the reliability, quality and saleability of the toll revenue stream, whilst ensuring that toll levels are reflective of the benefit obtained by the user and avoiding distortionary impacts'.

Government also endorsed investigating the impacts of a range of potential changes including:

• differential time-of-day tolling
• maintaining consistency in pricing structures between East West Link and CityLink
• introducing an additional heavy vehicle class
• differential pricing for particular ramps or sections
• flexible pricing.

Figure 3L shows the contribution of toll-related changes to the overall project cost in the final April 2015 contract.

These revenue sources are identical to those included in the contract signed by the previous government in October 2014 except for the exclusion of a freeze on tolls during construction to compensate road users for disruption during construction. This was withdrawn to cover in part the increased cost of the project with the transfer of works that were previously to be completed as part of the East West Link.

Figure 3L

Proponent's project funding contribution

Source: Victorian Auditor-General's Office based on VicRoads value-for-money analysis.

Figure 3L shows that commercial vehicles are the heaviest contributors to the unsolicited proposal funding, contributing $1.5 billion over 20 years. Advice to government justifying this remained constant since October 2014, advising that:

• the changes will bring the ratio of commercial vehicle to car tolls closer to those on other Australian toll roads and are more likely to produce a balanced network outcome and more efficient and fairer infrastructure pricing
• freight operators are able to pass costs on to customers and the expected level of diversion from the freeway to other roads is small
• the changes are not expected to significantly impact economic growth.

None of the agencies in this audit has been able to justify the substance of these arguments by providing a full and objective assessment of the performance of a range of potential funding options against the government's endorsed tolling principles.

Transparency to Parliament and the community

As with reporting on public private partnerships, government commits to releasing project summaries for unsolicited proposals which are high-level summaries of projects' scope, discounted costs, funding, need, rationale and expected benefits.

This level of reporting fails to provide sufficient assurance around the rigour of the options assessment, sufficient detail on the costs of the project over time, or a clear articulation of the value for money this represents.

The significant impact that funding CityLink Tulla will have on specific stakeholders, and the magnitude of this investment warrants a greater degree of transparency.

Recommendations

1. That the Department of Economic Development, Jobs, Transport & Resources evaluates the effectiveness of the governance framework recently introduced to assure the quality of transport forecasts and economic appraisals.
2. That the Department of Treasury and Finance:

• improves the level and depth of scrutiny applied to verifying benefits of all unsolicited proposals that proceed to full development
• fully assesses the viable, alternative proposal funding options for unsolicited proposals
• requires that relevant agencies define stakeholder engagement strategies for proposals that proceed to full development.

4.1 Introduction

This Part examines whether the High Value High Risk (HVHR) process has been effectively applied to the Cranbourne Pakenham Rail Corridor project (Cranbourne Pakenham) to provide rigorous and comprehensive advice to government about whether and how to proceed. It does this by determining whether the Department of Treasury and Finance's (DTF) ratings against the deliverability criteria reflect the underlying evidence and have been accurately and fully communicated to government.

The proponent submitted an interim offer on 31 January 2015. The Department of Economic Development, Jobs, Transport & Resources (DEDJTR) completed an interim evaluation in early March 2015, which underpinned DTF's HVHR assessment and the Minister for Public Transport's submission to government in late March. Government considered the interim proposal and this advice, and decided not to proceed with this project.

The proposals covered in this audit were at different stages of completion when they underwent their HVHR assessments and this affected how agencies assessed and reported on them:

• The CityLink Tulla Widening project (CityLink Tulla) final offer passed the HVHR review because DTF assessed it as substantially meeting all of the HVHR deliverability criteria.
• The review of the Cranbourne Pakenham interim offer had a different purpose. It was not a pass or fail test, but aimed to inform the proponent and government about issues that could prevent the final offer from passing an HVHR assessment. This interim assessment exceeded the requirements of the Market‑led Proposals Interim Guideline, which requires a single HVHR assessment of the final binding offer.

VAGO's review was complicated because the HVHR criteria defined for this interim offer review did not completely align with the categories in DTF's HVHR guidelines. While in total they covered the standard deliverability criteria, we had to translate the findings to address the same criteria used for CityLink Tulla.

4.2 Conclusion

DTF effectively applied the HVHR process to inform government's decision about whether to proceed with this unsolicited proposal to a final binding offer. The assessment described the progress made by the proponent but also the areas where its interim offer fell short of meeting the HVHR deliverability criteria.

DTF concluded that without a further 12 to 16 weeks beyond the agreed end of March 2015 deadline for the final offer, the proponent's final offer was unlikely to pass the value for money and HVHR assessments. In this case the HVHR assessment clearly fulfilled its role of informing government's decision about whether to proceed with the proposal.

However, our examination of this proposal illustrates weakness identified in Part 2 of this report. Specifically, the development process did not include a cost-benefit analysis or allocate responsibility for developing and applying a benefit management plan. These are mandatory for state-initiated projects and essential for informing government decisions and tracking and managing benefits after delivery. It is unclear from the unsolicited proposal guidelines how these practices should be applied to proposals.

4.3 HVHR assessment and advice to government

DTF effectively completed the HVHR assessment of the interim proposal and adequately communicated the findings to government.

4.3.1 Overall assessment

Figure 4A shows VAGO's assessment against the HVHR deliverability criteria.

VAGO's assessment of DEDJTR's and DTF's examination of the interim proposal concluded that it did not fully meet any of the HVHR criteria and the significance of the outstanding issues meant rating the level of assurance as 'not assured' or 'partly assured'. Even for the partly assured criteria, significant improvement was required for government to accept a final offer.

VAGO's assessment is consistent with DTF's HVHR assessment and confirmed that DTF applied sufficient scrutiny in its assessment to inform government about the issues critical to deciding whether to progress the proposal to a final, binding offer.

Figure 4A

VAGO assessment of the interim offer against the HVHR requirements

Note: ✔✔✔ = Fully assured—Project documentation and advice to government fully addresses Investment Lifecycle and HVHR Guidelines and provides a high level of assurance.

✔= Partly assured—Evidence that the HVHR review improved the level of assurance while not fully addressing all issues. The significance of outstanding issues varies from relatively minor—business cases is not updated with additional evidence—to more significant—where material issues are not raised or addressed. We provide a commentary on these issues throughout this Part.

✘ = Not assured—Insufficient evidence to be assured about the deliverability of the project.

Source: Victorian Auditor-General's Office.

DTF completed its HVHR assessment of the interim proposal in March 2015, and drew heavily on the work of DEDJTR's evaluation, which rated the interim proposal against the proposal-specific HVHR criteria as either:

• strong performance—demonstrated strengths with few errors, risks, weaknesses or omissions, if any
• average performance—existence of some errors, risks, weaknesses or omissions that are capable of resolution—including through the implementation of alternative strategies
• poor performance—numerous errors, risks, weaknesses or omissions that may be difficult to satisfactorily resolve.

We reviewed these ratings in forming our assessment of the proposal's level of readiness and determining whether government had been accurately and comprehensively informed about the issues. DEDJTR's interim evaluation was based on a structured, intensive and comprehensive review of the interim proposal.

4.3.2 Advice to government on the interim offer

DTF adequately communicated to government the outstanding issues and risks that the proponent needed to address in its final offer, as summarised in Figure 4B.

Figure 4B

Summary of outstanding issues and risks

Source: Victorian Auditor-General's Office based on DTF material.

These issues, together with advice about the extent to which the proposal would meet government's updated policies, provided the information needed to decide if and how to proceed with the unsolicited proposal. Government accepted DTF's recommendation not to proceed, but to instead use a conventional state-initiated project to deliver its policy commitments for the corridor.

4.4 Deliver on budget

Passing this HVHR criterion requires DTF to be confident that the project budget is based on a structured, detailed and verified understanding of the costs.

DTF concluded that the cost of the project was likely to exceed the budget allocation because of expected increases in the proponent's and the state's costs. This was supported by a DEDJTR project funding forecast suggesting an additional $559 million was required.

The interim evaluation rated performance for this criterion as average, taking into account the allowance for changes in scope but also the risks of cost escalation because separate work packages had not yet been integrated.

DTF expressed significant concerns about the project being delivered on budget. We agree with the finding that the proponent would have to significantly improve the quality of the cost estimate included in its final offer for it to pass this HVHR criterion. Most importantly it needed to fully integrate the separate work packages and update cost estimates in light of this. Accordingly we rate this criterion as not assured.

4.5 Deliver on time

Passing this HVHR criterion requires a detailed and realistic project implementation schedule which encompasses key milestones, decision points and delivery events. It also requires information on competing priorities, dependency analysis, required skills and capabilities, and staff availability.

DTF concluded that the proponent was unlikely to meet the time lines set out in the binding agreement it entered into with government, and that it had significant concerns about this.

The proponent's delivery plan was rated as poor because it had fallen behind the agreed schedule for reaching acceptance of a final offer and commencing delivery. The evaluation team judged that the proponent needed an additional 12 to 16 weeks to have a chance of its final proposal passing the HVHR criteria.

The nomination of proven contractors as preferred tenderers indicated that with sufficient time, the proponent was likely to develop an effective, integrated program. However, DTF believed that it was unlikely to meet the scheduled start of the delivery phase in July 2015 because it would need, at a minimum, April to June to try and address the HVHR deliverability issues identified in the interim offer.

DTF's comments and the interim evaluation rating are consistent with the evidence and align with our not assured rating.

4.6 Achieving the intended benefits

Delivering benefits to the community is the key reason for investing in public infrastructure. Managing projects to realise, sustain and maximise benefits is therefore essential. This requires an evidence-based understanding of the expected benefits, their distribution and the risks to their realisation, together with clear advice about how best to address these risks.

4.6.1 The state's intended benefits and policy settings

Figure 4C describes the state's objectives which frame the intended benefits.

Figure 4C

State's project objectives

Source: Victorian Auditor-General's Office based on the Project Finalisation Agreement.

The Project Scope Requirements documented the specific technical requirements required to deliver on these objectives. The proponent had to demonstrate how it had met these detailed requirements to provide assurance that the project, if implemented, would meet the state's objectives.

Objective 1 has been derived from an analysis of the problems facing the metropolitan rail system and from short-, medium- and long-term plans for meeting demand growth while maintaining a specific level of service. The problems, intended outcomes and operational implications are described in the December 2012 Network Development Plan – Metropolitan Rail and in the September 2014 Dandenong Line Concept of Operations.

The 2014 document sets out the measures needed to accommodate growth on the Dandenong corridor, with the Cranbourne Pakenham project designed—as part of this long-term plan—to extend capacity to meet demand until 2021.

The proposal included four major grade separations—building paths, roads and railways at different heights where they cross, so that traffic does not intersect. The analysis and outcome definition underpinning the choice of these four was not clear. While it is clear that growing road and rail traffic is intensifying delays at these major level crossings, we have not seen the same type of strategic analysis underpinning their selection.

The change in state government further complicated the decision-making context because it committed to grade separating nine level crossings on the corridor and running services with a newly assembled fleet of 37 high capacity trains. The proposal included 25 new trains and 12 existing trains.

4.6.2 DTF's HVHR assessment

While DTF confirmed that it had not identified any issues that would, without further work, prevent the achievement of the state's objectives, it found a range of issues where the interim proposal fell short of meeting the Project Scope Requirements. In addition, the proposal did not fully align with the new government's updated policy commitments for the corridor.

The issues DTF raised about scope and benefits included that the interim offer:

• would result in a number of services being more overcrowded than expected because of the proposed service pattern
• did not comply with the local manufacturing content requirement, although this item was considered separately to the HVHR assessment
• may restrict changes to the rail network in the future
• did not meet all the Project Scope Requirements with most of the noncompliance being in relation to the signalling system, with additional concerns about the proposed depot design.

The current scope is not fully consistent with updated government policy:

• It did not allow for the purchase of 37 new trains to operate the proposed service pattern. The interim offer included the purchase of only 25 new high capacity trains.
• It did not allow for the removal of nine level crossings consistent with government's election commitments. The Project Scope Requirements specified the removal of four major level crossings.

DTF rated the performance of the interim offer, in terms of delivering the benefits, as average. This was a combination of:

• a strong rating in relation to the state's objectives—noting that the proponent had issues to resolve but that with sufficient time resolution could be achieved
• an average rating in relation to meeting the scope and requirements.

4.6.3 VAGO assessment

Our assessment aligns with DTF in finding the deliverability of the intended benefits is only partly assured. DTF advised government of the critical issues in relation to the intended rail project benefits and this assessment was underpinned by extensive scrutiny of the proponent's conformity with the scope requirements.

An appreciation of broader network and corridor problems and priorities provides critical context for making decisions about unsolicited proposals. In this sense the Cranbourne Pakenham proposal had an advantage over CityLink Tulla where there is not the same contextual information.

Our residual concerns involve the absence of:

• guidance on the need and responsibilities for developing a cost-benefit analysis and a benefits management plan, as are required for conventional state-initiated projects
• information on the problems and strategic priorities driving the level crossing removal component of the proposal.

4.7 Comprehensively manage risks

DTF's Investment Lifecycle and HVHR Guidelines require a risk management strategy, with the clear identification and assessment of key risks, and a documented approach to ongoing risk monitoring and management.

DTF concluded that further work was required to mitigate key project risks. Specifically:

• the signalling and Dandenong rail corridor work packages needed to be better integrated to deliver an acceptable level of risk
• the overall project delivery program requires further integration and greater coverage of the risks to time lines and costs from this underdevelopment.

The proponent had accepted the state's proposed risk allocation and this included risks relating to the relationship between the proponent and the rail franchisee, which was seen as one of the unique strengths of the proposal.

The interim evaluation's rating of average and DTF's assessment are consistent with our rating of partly assured for this criterion.

4.8 Applying sound governance and management

The Investment Lifecycle and HVHR Guidelines require:

• evidence of an appropriate governance structure—including that people appointed to key positions have been identified
• a robust and up‑to‑date project management strategy—including deliverables, dependencies between activities and monitoring and review arrangements
• the identification of a senior responsible officer for the state.

Murrumbeena station level crossing. By Melburnian (Own work (digital photograph by author)) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons

DTF agreed with the interim evaluation rating of average for the proponent's capability and capacity and the strong rating for the public private partnership style organisation structure proposed.

We have taken this as an overall average rating for governance and translated this to a partly assured rating. While the proposed structures appeared sound:

• the proponent needed to further develop its organisational structure and identify key personnel
• DTF's HVHR assessment did not examine DEDJTR's plan to oversee project delivery—this information was still being developed and was to be assessed with the binding offer.

In addition we note, in common with CityLink Tulla, there was a lack of preparedness to adequately engage relevant stakeholders and the broader community. This again highlights the need for a clear articulation by the portfolio agency early in the unsolicited proposal process of the expectations around the timing, depth, scope of, and roles and responsibilities for, stakeholder consultation.

4.9 Procuring for best value for money

The Investment Lifecycle and HVHR Guidelines require a range of procurement options to be considered. Documentation should describe the process and method for selecting a preferred form of procurement and explain why the choice is the best value for money.

The interim evaluation and DTF confirmed the progress made by the proponent. The project was to be procured as seven discrete work packages through a select tender process. DTF found that, as a consequence, competitive prices had been obtained at the subcontractor level. The proponent had identified preferred bidders for all but the train depot maintenance and had started to integrate the delivery of these packages.

The specialist contractor employed for the purpose concluded that the interim offer demonstrated value for money by outperforming the state cost benchmark, while comparing well in terms of its qualitative outcomes.

DTF believe that this opinion needs to be tempered by the preliminary nature of the assessment and in particular the unresolved issues described throughout this Part.

Accordingly the interim evaluation and DTF rated the proposal's value for money as average which corresponds with our partly assured rating. We reached this assessment because the approach to procuring the packages appeared robust and competitive, but it lacked integration and there were uncertainties about the benefits.

4.10 Lessons learned from this proposal

The major lessons learned from Cranbourne Pakenham are:

• the importance of a structured and rigorous evaluation and the full reporting of its significant findings for informing government decisions
• the value of being able to place a proposal in the context of a wider, well informed perspective of system-wide problems and seeing where the proposal might fit into a comprehensive plan for addressing these problems in the short, medium and long term
• the dangers of the current ambiguity about the inclusion of essential elements in the development of a business case, such as completing a cost-benefits analysis and developing a benefit management plan. This needs to be better defined and the responsibility for doing this rests with DTF, as it responds to the recommendation made in Part 2.

RESPONSE provided by the Secretary, Department of Treasury and Finance

RESPONSE provided the Acting Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE by provided by the Chief Executive, VicRoads