Impact of Increased Scrutiny of High Value High Risk Projects

Background

In Victoria, the total value of public sector capital investments underway or commencing in 2014–15 is around $72 billion, including projects delivered as public private partnerships. This includes a commitment made in the 2014–15 Budget of up to $27 billion to build new road and rail infrastructure such as the $8 billion to $10 billion stage 2 of the East West Link and the $8.5 billion to $11 billion Melbourne Rail Link.

Previous VAGO and Ombudsman reviews have found that significant flaws with project business cases and procurement processes have led to cost and time overruns and diminished benefits.

The government adopted the High Value High Risk (HVHR) process in late 2010 because of past cost overruns of over $2 billion that it attributed to the inadequate management of project business cases and procurements. The goal of the HVHR process is to achieve more certainty in the delivery of intended benefits in line with planned costs and time lines.

The HVHR process applies to all public sector infrastructure and information and communications technology investments that are likely to draw on Budget funding that:

• have a total estimated investment greater than $100 million, or
• are identified as high risk using an approved risk assessment tool, or
• aredetermined by government as warranting the rigour of increased oversight.

HVHR investments are subject to extra scrutiny by the Department of Treasury and Finance (DTF), together with additional Treasurer's approvals at key milestones covering the business case and procurement, and monitoring after the procurement decision. HVHR reviews started in 2011 to cover projects in the 2012–13 Budget cycle.

This audit examined the effectiveness of the HVHR process in improving project business cases and procurements so that they provide an adequate basis for delivering intended project benefits within approved time lines and costs. We assessed whether the process has:

• been applied as intended to assess business cases, procurement approaches and contractual and governance arrangements for all relevant HVHR projects
• resulted in improved business cases that provide a comprehensive and rigorous basis for government project decisions
• resultedin improved procurements and, ultimately, better outcomes.

We examined the application of the process for five projects valued at $936 million:

• the registration and licensing system (RandL)
• Mitcham Road and Rooks Road rail grade separations
• Royal Victorian Eye and Ear Hospital redevelopment
• Western Highway duplication from Beaufort to Buangor
• regionalrolling stock purchase.

Towards the end of the audit we examined a further three projects to assess the impact of DTF updating its business case assessment template in late 2012.

Conclusion

Based on our review of these projects, DTF's increased scrutiny applied through the HVHR process has made a difference to the quality of the business cases and procurements underpinning government's infrastructure investments. However, DTF has not yet started formally evaluating the impacts of the application of the HVHR process on project outcomes.

These improvements have not lifted practices so that they consistently and comprehensively meet DTF's better practice guidelines. This means government is still exposed to the risk that projects fail to deliver intended benefits on time and within approved budgets. A robust HVHR process is critical to effectively managing this risk.

DTF's scrutiny and assurance of HVHR projects was not applied comprehensively or consistently across all the 'deliverability' criteria. Its performance was best when assuring project costs, time lines and agencies' approaches to risk management. It was more mixed for procurement, governance and project management, and clearly inadequate when assuring the expected project benefits. Of particular concern was the lack of substantive HVHR review of project benefits, given that none of the business cases adequately justified these.

This audit identified gaps and inconsistencies in DTF's approach to identifying projects that should be subject to HVHR review, specifically in relation to small value projects that may have high risks and projects that were delivered outside of the general government sector.

It also identified the need for more comprehensive assessment guidance for HVHR analysts, the application of a standardised file structure and comprehensive central register for managing reviews and the need for better identification and management of potential conflicts of interest.

The RandL project is one of the projects this audit examined in detail. It is the type of complex project that HVHR is meant to deal with and an example of a project where significant, negative deliverability risks have materialised. While the RandL project commenced well before the HVHR process started, the updated 2011 business case was subject to an HVHR type review and the project has been monitored under HVHR since late 2011.

However, neither the management nor oversight applied to the RandL project, including HVHR monitoring, was effective in detecting or controlling risks before they materialised to significantly delay the project and increase the estimated cost of delivery. A key reason for this was the lack of skills and depth of experience needed to manage and oversee a complex project of this type.

The government has now decided to pause the project until February 2015. It is currently running at least 18 months late, has cost around $102 million and VicRoads has proposed additional expenditure of up to $135 million more than the approved budget of $158 million.

Findings

Management of the HVHR process

There is clear scope for DTF to improve how the HVHR process is managed, to make it more efficient, transparent and accountable. DTF's May 2012 internal audit and VAGO's 2013 audit of the Gateway Review Process recommended changes that DTF has not yet fully addressed.

DTF's HVHR process captures projects valued above $100 million that rely on funding from the State Budget. However, the audit identified management weaknesses that DTF needs to address, including:

• acting on VAGO's previous findings about gaps in its approach to assessing whether projects valued between $5 to $100 million should be in the HVHR process
• providing criteria for assessing whether non-Budget-funded projects typically funded by public corporations should be captured by the HVHR process
• extending the detailed guidance and templates provided to DTF analysts for business case assessments to cover the entire HVHR process
• better administering the process by applying a standard documentation and record-keeping approach to all reviews, and maintaining a master register which comprehensively documents review activities
• improving the collation and communication of lessons learned and better practices observed across all HVHR reviews
• applyingan evaluation framework to reliably measure the outcomes of the HVHR process and understand further areas for improvement.

Improving the quality of reviews

HVHR business case reviews raised substantive issues and agencies frequently modified submissions to address these issues. However, this did not consistently occur across all projects nor for all the criteria that DTF uses to test a project's deliverability.

For four of the five business cases we looked at in detail we saw evidence of:

• substantive reviews following an HVHR review template
• DTF identifying and constructively communicating issues to departments
• responses to DTF's comments providing greater assurance about the rigour of the project and in some cases resulting in changes to business cases to reflect this information
• adviceto the Treasurer which accurately reflected review findings and conclusions.

However, there is considerable room for DTF to improve its administration and oversight of the HVHR process. Some improvements do not require much change—for example, making sure that business cases are updated where additional significant information is provided. However, other improvements, such as adequately testing the reliability of project benefits, require a step change in approach and acquisition of the skills needed to do this.

Recommendations

That the Department of Treasury and Finance:

1. improves its approach to selecting projects for inclusion in the High Value High Risk process by:

• systematically reviewing projects between $5 million and $100 million to determine whether they should be subject to the High Value High Risk process through documented reviews of project risk assessments
• clarifying the criteria for selecting public projects that do not require Budget funding for inclusion in the High Value High Risk process
• recommending that projects over $100 million selected under the government's unsolicited bids policy be subject to High Value High Risk processes

2. develops assessment guidance and templates covering all High Value High Risk stages to improve the consistency, rigour and transparency of High Value High Risk reviews
3. improves its administration of the High Value High Risk process by:

   applying a standardised file structure for managing assessment documents and supporting evidence

   developing and maintaining a comprehensive central register of High Value High Risk review activity by project and High Value High Risk process stage

4. identifies potential conflicts of interests of reviewers and documents how these are mitigated
5. develops and applies an evaluation tool to measure the extent to which the High Value High Risk process is affecting project outcome
6. provides greater assurance that High Value High Risk reviews comprehensively test compliance with its Investment Lifecycle and High Value High Risk Guidelines in areas critical for project deliverability
7. checks that for complex, risky projects—particularly those involving information and communications technology transformations—the specialist skills needed to successfully manage, oversee and quality assure these projects have been assessed and acquired.
8. improves how it communicates with and informs departments by:
9. developing a structured approach to collating and sharing the lessons from all High Value High Risk reviews
10. completing an annual satisfaction survey of agencies that have been subject to High Value High Risk reviews

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report was provided with a request for submissions or comments to:

• Department of Treasury and Finance
• Department of Transport, Planning and Local Infrastructure
• Department of Health
• VicRoads
• Public Transport Victoria.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

1.1.1 Infrastructure planning and delivery

Effective planning and delivery of major infrastructure projects is critical for governments to achieve their policy objectives. In Victoria, the total value of public sector capital investments underway or commencing in 2014–15 is around $72 billion.

The 2014–15 Budget commits up to $27 billion in new infrastructure including:

• $8.5 billion to $11 billion for the Melbourne Rail Link, which includes a tunnel linking Southern Cross and South Yarra stations and a new airport rail link
• $8 billion to $10 billion to complete the East West Link by extending the eastern section from the Tullamarine Freeway to the Western Ring Road
• $2 billion to $2.5 billion to upgrade the Cranbourne–Pakenham Rail Corridor
• $850 million to widen the Tullamarine Freeway approaches to the airport.

Southern Cross Station. Photograph courtesy of TK Kurikawa/Shutterstock.

This substantial investment requires effective oversight. However, reviews by VAGO and the Ombudsman found that significant flaws with project business cases and procurements have led to cost and time overruns as well as diminished or unclear benefits. These findings highlight the critical importance of the more stringent review processes examined here.

1.1.2 Goal of the HVHR process

In late 2010 the government introduced the High Value High Risk (HVHR) process to address issues related to inadequate project management that resulted in cost overruns of $2 billion. The goal of the HVHR process is to achieve more certainty in the delivery of intended benefits to planned costs and time lines through more rigorous review of business cases and procurements.

By 2008 the Department of Treasury and Finance (DTF) had developed Investment Lifecycle Guidelines that were mandated for investments over $5 million. Since 2010, these guidelines have been updated so that they support the development of business cases, which are mandatory for capital investments with a total estimated investment of $10 million or more. DTF stated to the Public Accounts and Estimates Committee Inquiry into Effective Decision Making for the Successful Delivery of Significant Infrastructure Projects that the 'HVHR processÖ is directed at ensuring the guidance and processes are thoroughly and consistently applied'.

In introducing the HVHR process, DTF advised government that the priority was to:

• enforce the requirement for a robust business case with clear project objectives, well-defined benefits, a rigorous appraisal of options, selection of appropriate procurement methods and appropriate governance and management
• clearly articulate a tender proposal, appointment approach and contract management framework that appropriately allocates and manages risk, delivers benefits and effectively manages scope and cost.

1.2 Victoria's approach to managing investments

1.2.1 Guidance

DTF's investment lifecycle framework guides agencies to derive maximum benefit from investments by helping them to:

• conceptualise an investment by establishing the need and defining the benefits
• prove a solution by assessing the costs, benefits and risks of likely options
• procure the investment by awarding a contract that best delivers the solution and provides value for money
• implement the solution to realise benefits and manage costs and risks
• realise the benefits and measure the success of the investment.

Figure 1A shows the guidelines, tools and templates DTF makes available so that agencies can effectively navigate this pathway. It also shows the role of the Gateway Review Process—or 'Gateway'.

Figure 1A

Investment lifecycle and HVHR guidelines, tools and templates

Source: Department of Treasury and Finance, Investment Lifecycle and High Value/High Risk Guidelines—Overview (2014).

1.2.2 Review and oversight of government investments

The mechanisms for reviewing the rigour and progress of investments from concept through to benefit realisation and informing government funding decisions are:

• departmental, quality-assured documentation—justifying the investment and its procurement and implementation approaches
• DTF review and advice to government—the value and complexity of each investment determining the depth and type of review applied
• the Gateway Review Process—examining investments at key decision points and providing advice to the project's senior responsible person.

Figure 1B shows what the review process looks like for a complex, risky or high value project where all these review layers are applied. The HVHR process involves strengthening DTF's review of investments up to and including implementation and the following Section defines the scope and content of HVHR reviews.

Figure 1B

Investment review process

Source: Department of Treasury and Finance, Investment Lifecycle

1.3 Scope and content of the HVHR process

1.3.1 Investments captured by HVHR

The HVHR process applies to all general government sector infrastructure and information and communications technology investments that are likely to draw on Budget funding and:

• have a total estimated investment greater than $100 million, or
• are identified as high risk using an approved risk assessment tool, or
• are determined by government as warranting the rigour of increased oversight.

The general government sector includes approximately 200 agencies that directly provide public services such as departments, public hospitals and further education institutes.

While the process is primarily applied to Budget-funded investments, it can—at the government's discretion—be applied to investments by state-controlled corporations that sit outside the general government sector. These are the seven public financial corporations, e.g. the Treasury Corporation of Victoria, and around 80 public non‑financial corporations, e.g. the Port of Melbourne, water corporations and Alpine Resorts.

These reviews are discretionary because investments in these entities are usually not funded from the State Budget but made on a commercial basis from their own resources.

DTF's public statements around this issue are not uniformly consistent and clear, with reference to the inclusion of all investments over $100 million 'regardless of the funding source' on its website. The audit examines this clarity issue in Part 2 of this report.

1.3.2 Structure and content of the HVHR reviews

The HVHR process builds on DTF's existing Investment Lifecycle Guidelines and Gateway reviews. HVHR investments are subject to extra scrutiny by DTF and additional Treasurer's approvals at the following stages—for the business case deliverability assessment, at critical milestones during procurement and for significant variations after contract sign off.

Within the HVHR process, DTF also monitors critical recommendations arising from Gateway reviews on 'Concept and Feasibility' (Gate 1) to 'Tender Decision' (Gate 4).

Preliminary business case

DTF advises government on strategic merit and need and whether the proposal should be developed into a full business case.

This review is clearly part of the HVHR assessment because DTF's guidance documentation indicates that only HVHR projects must submit a preliminary business case. Non-HVHR projects instead submit a less extensive strategic assessment.

The key decision is whether a project should proceed to a full business case based on a preliminary assessment of strategic merit, an early options assessment and the project's likely costs, benefits and risks—including a high level cost-benefit analysis.

Full business case

DTF provides advice to inform the Treasurer's decision on whether the case is sufficiently robust to be submitted to government for funding by advising whether it is confident that the project's intended benefits can be delivered on time and within budget. The government's assessment—also informed by DTF and the Department of Premier and Cabinet—goes beyond deliverability to also review the policy merit, affordability, scale and alignment of the proposal.

DTF has a template for its analysts on the scope, approach and documentation of HVHR full business case reviews. This template was initially developed in 2011 and updated in late 2012.

DTF's assessment focuses on providing government with assurance that projects are 'deliverable'. Figure 1C outlines this definition. At this stage, DTF needs to establish whether a well thought through plan has been developed to deliver the project as outlined in the business case.

Figure 1C

Project deliverability definition

An assessment of project deliverability is DTF's judgement on whether the business cases can achieve the: nominated benefits proposed in the business cases, on budget and on time according to the specific scope of the business cases.

Source: Victorian Auditor-General's Office from the Department of Treasury and Finance.

After it does this assessment, DTF can recommend to the Treasurer that the project:

• proceed for funding consideration—i.e. DTF assesses the project as deliverable
• proceed for funding consideration subject to further refinement of identified issues—i.e. DTF assesses the project as deliverable if the final business case submitted for funding is updated to address specific issues
• not proceed—i.e. DTF assesses the project as unlikely to be deliverable.

Figure 1D shows how DTF addresses deliverability by assessing the project as conveyed in the business case across seven key areas. Analysts should cover these areas when assessing business cases for HVHR projects, supporting responses with a clear rationale and sufficient evidence to justify their assessments.

The final DTF recommendation about whether to proceed is based on a balanced assessment, taking account of the responses in each of these areas.

Figure 1D

Business case assessment areas

Deliver on budget—assess the comprehensiveness, logic and level of detail of the project costs, the incorporation of risks if alternative funding forms part of the business case and, where there are multiple options, whether these have been soundly costed. Deliver to time lines—assess whether project time lines and milestones are sound and supported by sufficient evidence and adequate plans to deliver as intended. Determine whether the time lines are consistent with those used for similar projects. Deliver benefits—assess whether the business case adequately demonstrates that the project can deliver the intended benefits, and compare the predicted benefits with those achieved by similar projects. Comprehensively manage risks—assess the comprehensiveness of the risk management plan included in the business case and determine whether there are any gaps and whether there is a clear strategy to mitigate material risks. Apply appropriate governance—assess whether the governance structure includes: a clearly defined project owner a board or steering committee with an appropriate mix of skills and experience clear roles, responsibilities and decision-making processes appropriate plans to secure the required resources and manage stakeholders. Adequately justify the preferred procurement approach—assess whether all suitable procurement options have been considered and whether the business case demonstrates that the proposed procurement strategy is the most effective and value-for-money way to deliver the project. Substantiate the project management approach—assess whether the business case clearly substantiates that: the project is using an appropriate management strategy, is underpinned by sufficient project planning and has, or will develop, an appropriate change management plan and handover strategy from project team to end users the organisation has, or can, secure the capability and skills required to deliver the project and has demonstrated that it can successfully deliver similar projects.

Source: Victorian Auditor-General's Office based on Department of Treasury and Finance guidance.

Procurement

DTF provides advice to inform the Treasurer's approval of:

• procurement documentation prior to its release
• the decision on the preferred bid prior to its announcement
• the final contract, prior to signing.

Reviews of tender documentation, the preferred bid decision and the final contract prior to sign-off are meant to ensure that:

• the preferred procurement approach is effectively implemented and translated into a contractual framework that delivers on the best value-for-money approach
• the risks of diminishing this outcome through unintended scope creep or a contractual form that diminishes the outcomes for the state are managed.

Post contract

The HVHR process involves DTF monitoring project progress—including emerging issues and risks and any changes to the project scope and cost that was agreed to in the signed‑off contractual arrangements—and advising the Treasurer about significant changes. The Treasurer approves significant contract variations based on advice from DTF.

Within the HVHR process DTF also monitors projects post contract signing and during delivery through its quarterly reporting process to government and representation on project steering committees or equivalent.

1.4 Reviews relevant to this audit

1.4.1 VAGO audit—Gateway Review Process

VAGO's 2013 audit, Planning, Delivery and Benefits Realisation of Major Asset Investment: The Gateway Review Process, concluded that DTF had not systematically reviewed and verified the agency self-assessments of project risk that determine whether projects are subject to the Gateway reviews.

DTF also uses these assessments to identify risky or complex projects under $100 million that should be covered by the HVHR process.

Part of VAGO's recommendation on this issue required DTF to verify that all new projects had completed robust project risk assessments.

1.4.2 Department of Treasury and Finance internal audit reviews of HVHR

DTF commissioned two internal audit reviews of the HVHR process that reported in May 2012 and May 2014.

Figures 1E and 1F summarise the findings from these internal audits, and Part 2 of this report examines whether DTF has acted on these findings.

Figure 1E

Recommendations from HVHR May 2012 internal audit

#	Recommendation	Rating / action	Internal audit commentary
1.	Project business case assessment requirements should be structured and formalised.	Moderate 30 Sep 2012.	Lack of standards to guide the use of DTF's business case assessment tool. Variation across DTF analysts in detail required from agencies and evidence retained of approvals.
2.	Bring forward time frames for submission of preliminary/full business cases.	Moderate 30 Sep 2012.	Time lines for the provision and review of business cases during 2011 and early 2012 for the 2012–13 Budget cycle were difficult and not always met.
3.	A standard file structure should be implemented to assist HVHR process.	Low 30 Sep 2012.	DTF did not maintain documentation received from departments in a consistent or standardised way, impacting on the ability to obtain a clear audit trail of work undertaken.
4.	HVHR stakeholder engagement/communications activities should be enhanced.	Low 30 Sep 2012.	Communication issues around the transition to the then new HVHR process primarily around DTF's information requirements.
5.	The relationship between HVHR and Gateway may require further clarity.	Improvement opportunity No date set.	Some confusion for departmental project teams on the links and distinctions between the Gateway review and HVHR processes.

Source: Victorian Auditor-General's Office.

The May 2014 audit was conducted as a separate exercise to the May 2012 audit and did not follow up on the earlier audit's recommendations. It is unclear why the later audit did not examine these earlier recommendations.

Figure 1F shows the recommendations, ratings and commentary of the later report.

Figure 1F

Recommendations from May 2014 HVHR internal audit

#	Recommendation	Rating/action	Internal audit commentary
1.	Outcomes from the lessons learnt database should be incorporated into the HVHR business case review process.	Low 30 November 2014.	A comprehensive Gateway lessons learnt database has been developed allowing for user access within DTF and by other key users, such as departments. This database includes key lessons for projects based on independent gateway review recommendations, search capabilities and top 10 risks by project type. This top 10 risk functionality is not currently used in HVHR deliverability assessments analysis so there is no formal linkage of these lessons learnt to current assessment processes to ensure that common project risks are adequately considered for HVHR projects. There is the risk that available data is not effectively used to drive robust HVHR deliverability assessments.
2.	Impacts to project benefits realised should be tracked throughout the project lifecycle.	Improvement opportunity 30 November 2014.	There is no ongoing monitoring or reporting against project benefit metrics. A template to track broader project benefits upon project completion against those proposed in the business case for all asset investments—i.e. HVHR and non‑HVHR investments—has recently been made available to agencies.

Source: Victorian Auditor-General's Office.

1.5 Audit objectives, scope and criteria

This audit examined the effectiveness of the HVHR process in improving project business cases and procurements so that they provide an adequate basis for delivering intended project benefits within approved time lines and costs.

It did this by assessing whether the HVHR process has:

• been applied as intended to assess business cases, procurement approaches and contractual and governance arrangements for all relevant HVHR projects
• resulted in improved business cases that provide a comprehensive and rigorous basis for government project decisions
• resulted in improved procurements and ultimately improved delivery and outcomes.

1.5.1 Applying the HVHR process as intended

Part 2 assesses whether DTF's management and application of the HVHR process has been:

• comprehensive—in terms of identifying HVHR projects and completing required reviews
• consistently high quality—in terms of employing capable reviewers and applying a consistent, quality-assured approach to deliver high quality reviews
• underpinned by adequate monitoring and reporting—firstly to government and also to agency stakeholders
• evaluated in terms of its emerging impacts—by developing and applying a framework to measure improved timeliness, cost and benefit realisation.

1.5.2 Impacts on business cases and procurements

Part 3 examines HVHR reviews to determine their impacts on the quality of business cases and procurements and how this is likely to affect project outcomes.

For a sample of business cases we assessed whether HVHR assessments of the deliverability of intended benefits according to planned costs and time lines were robust. We did this by examining whether they provided adequate assurance that projects would:

• deliver on budget
• deliver to planned time lines
• deliver the intended benefits
• comprehensively manage risks
• apply appropriate governance
• apply a procurement strategy that represented value for money
• apply an appropriate and effective project management approach.

We also examined whether the procurement reviews were undertaken as required and contributed to assurance that the procurement approach approved as part of the business case was effectively implemented.

1.6 Audit method and cost

The audit team gathered evidence by examining the application of the HVHR process for five projects, namely:

• Registration and licensing system—this project involves replacement of VicRoads registration and licensing IT systems and changes to supporting processes. The initial estimated cost was $173 million, with completion targeted for April 2013.
• Mitcham Road and Rooks Road rail grade separations—this project is part of the Metro Level Crossing Blitz program and involves separating the rail from the road by lowering the rail tracks beneath Mitcham Road and Rooks Road, in Mitcham. The total estimated cost is $197 million and a target completion date of mid-2014.
• Royal Victorian Eye and Ear Hospital redevelopment—this project involves redeveloping the hospital at a cost of $165 million with a target completion date of December 2017.
• Western Highway duplication from Beaufort to Buangor—this project is part of the $505 million Western Highway duplication project from Ballarat to Stawell, funded by the Commonwealth and state governments on an 80/20 basis. The project budget for the duplication from Beaufort to Buangor is around $140 million. It was contracted in April 2014 and construction is expected to commence in mid-2014.
• Regional rolling stock—this investment involved the procurement of additional rolling stock for the regional rail network with total approved funding of $261 million and an expected final delivery date of August 2016.

These projects were selected to give coverage of projects of different type, scale and complexity and from more than one sector that had been subject to substantial HVHR review and had also moved into implementation.

V/Line VLocity train.

Towards the end of audit conduct we added three projects to our sample to test whether the introduction of an updated business case assessment template in late 2012 had improved the quality of reviews.

The additional three projects were:

• Blackburn Road grade separation
• Kilmore Wallan bypass
• Southland station.

Audit evidence was gathered from DTF and the following departments and agencies responsible for planning and delivering the sample projects:

• Department of Health
• Department of Transport, Planning and Local Infrastructure
• Public Transport Victoria
• VicRoads.

The audit was conducted in accordance with section 15 of the Audit Act 1994 and the Australian Auditing and Assurance Standards.

Pursuant to section 20(3) of the Audit Act 1994, any persons named in this report are not the subject of adverse comment or opinion.

The total cost of the audit was $370 000.

1.7 Structure of the report

The report is structured as follows:

• Part 2 examines the management of the HVHR process
• Part 3 examines the impact of the HVHR process.

2.1 Introduction

Effectively managing the High Value High Risk (HVHR) process involves establishing a review capacity, applying clear policies and procedures to guide and record reviews, applying rigorous quality assurance, clearly communicating with stakeholders and monitoring and reporting on the process and its outcomes.

We assessed whether the Department of Treasury and Finance's (DTF) overall management and application of the HVHR process had been:

• comprehensive—capturing all high value and risky projects within the HVHR process and completing all prescribed reviews (Section 2.3)
• consistently high quality—employing capable reviewers, applying a consistent, quality-assured approach to deliver high quality reviews (Section 2.4)
• underpinned by adequate monitoring and reporting—firstly to government and also to departments so they could understand the process and respond to the lessons learned across all reviews (Section 2.5)
• evaluated in terms of emerging impacts—by applying a framework to measure improved timeliness, budgeting and benefit realisation (Section 2.6).

We have also drawn on VAGO's May 2013 audit Planning, Delivery and Benefits Realisation of Major Asset Investment: The Gateway Review Process and on DTF's internal audit findings and recommendations reported in May 2012 and May 2014.

2.2 Conclusion

There is considerable scope for DTF to improve how the HVHR process is managed to make it more efficient, transparent and accountable. Its May 2012 internal audit and VAGO's audit of the Gateway Review Process (Gateway) recommended changes that DTF has not yet fully addressed.

DTF effectively captures within the HVHR process projects valued over $100 million which rely on funding from the State Budget. However, the audit identified management weaknesses that DTF needs to respond to, including:

• addressing VAGO's previous findings about gaps in DTF's approach to assessing whether projects valued between $5million and $100 million should be in the HVHR process
• providing criteria for assessing whether non-Budget funded projects typically funded by public corporations should be captured by the HVHR process
• extending detailed guidance and templates provided to DTF analysts for business case assessments to cover the entire HVHR process
• better administering the process by applying a standard documentation and record-keeping approach to all reviews and maintaining a master register which comprehensively documents review activities
• improving the collation and communication of lessons learned and better practices observed across all HVHR reviews
• applying an evaluation framework to reliably measure the outcomes of the HVHR process and understand further areas for improvement.

2.3 Comprehensively applying the HVHR process

The HVHR process aims to capture Budget-funded infrastructure and information and communications technology (ICT) investments that are valued at more than $100 million or are assessed as high risk or that the government identifies as requiring more intensive review and oversight.

The five projects we reviewed in detail had been subject to the required HVHR reviews and approvals, with one exception—the regional trains rolling stock project.

For this project the most significant exception was the lack of evidence that the Treasurer approved the request for proposal document. However, DTF adequately justified this on the basis that the Treasurer had approved the procurement strategy for a single supplier to bid. There is no evidence that the absence of approval for the request for proposal document compromised the integrity of the procurement process.

In terms of coverage, DTF has included Budget-funded projects over $100 million within the HVHR process since it was first used in the 2012–13 Budget cycle. However, it is not clear that the HVHR process has been applied to all intended projects, specifically for projects:

• valued between $5million and $100 million that are high risk, because DTF cannot show that it has reviewed risk profile reports when deciding on their HVHR status
• over $100 million that are funded outside of the Budget by public corporations, because there is no systematic assessment process and inclusion is activated only through a specific request by government.

The government announced funding commitments of up to $27 billion as part of the 2014-15 Budget for infrastructure projects, with most ($24 billion) designated as HVHR projects that have not progressed through the HVHR process. These included projects funded under government's unsolicited bids policy and multi-billion dollar projects for a rail tunnel linking Southern Cross and South Yarra stations and stage 2 of the East West Link.

It is currently unclear how the HVHR process will be applied to unsolicited bids and to major projects that have been fast tracked to a funding commitment without any HVHR reviews.

The absence of a comprehensive central, up-to-date register to provide a consolidated record of risk assessments and subsequent reviews for HVHR projects makes it difficult for DTF to demonstrate that the HVHR process is comprehensively applied.

This is an essential management tool and DTF should develop and maintain a comprehensive register.

2.3.1 Identifying HVHR projects

This Section provides the evidence on our overall findings.

Total estimated investment greater than $100 million

DTF has effective processes in place to capture within the HVHR process projects worth more than $100 million that are funded through the State Budget. Most of these projects are for assets in the general government sector. This sector includes approximately 200 agencies directly providing public services such as departments, public hospitals and further education institutes. It excludes seven public financial corporations, such as the Treasury Corporation of Victoria and, around 80 public non‑financial corporations, such as the Port of Melbourne Corporation, water corporations and Alpine Resorts.

Public corporations outside of the general government sector typically rely on their own finances, not the State Budget, to fund major projects. The major exception is transport investments, which are normally funded through the State Budget but are held by a public corporation, VicTrack.

DTF does not usually include in the HVHR process projects that are funded outside the State Budget unless requested to do so by government. There is also a lack of clarity about whether projects under the recently announced unsolicited bids policy will be subjected to HVHR review.

We found that projects over $100 million listed in the State Budget since the 2011–12 Budget cycle had been reviewed under the HVHR process, with the exception of:

• projects started before the application of HVHR review
• projects funded outside of the State Budget by, for example, water corporations
• expenditure related to a program of projects rather than a single project.

The Public Accounts and Estimates Committee's (PAEC) Report on the 2012–13 Financial and Performance Outcomes of May 2014 identified 33 projects in the 2012–13 Budget Papers valued at more than $100 million that were not designated as HVHR.

DTF confirmed that it does not include non-Budget‑funded projects in the HVHR process unless specifically requested to do so by government. It has advised that HVHR exists primarily to provide Budget-related assurance, and public corporations have a commercial-based oversight overlay to manage investment risks.

However, we note that DTF did not advise PAEC of this. Instead, DTF advised that these projects were not included in the HVHR process because they had started before its application or because the expenditure related to a program of projects rather than a single project.

DTF needs to provide greater clarity by defining the criteria it uses to decide when non‑Budget‑funded projects should be captured by the HVHR process.

DTF also needs to clarify whether proposals received under the government's new unsolicited proposal process, such as the $2.5 billion Pakenham and Cranbourne rail line upgrade, will be captured by the HVHR process. The value of this bid and the absence of a competitive bidding process make the application of the HVHR process to projects like this essential.

DTF advised that it will propose to government that the Unsolicited Proposal Guideline should be updated to provide clarity on the role of the HVHR process.

Identifying high risk projects under $100 million

In 2013, VAGO concluded that DTF had not systematically reviewed and verified agencies' self-assessments used to decide which projects fall within the Gateway Review Process. DTF uses these same assessments to decide whether to include projects in the HVHR process.

Figure 2A summarises our 2013 findings and recommendations to DTF. In summary, VAGO recommended that DTF improve its processes for systematically documenting its reviews of agency risk assessments.

DTF accepted the recommendation but indicated that it was already undertaking risk assessments of projects independent of the Gateway Project Profile Model risk assessment tool.

However, 12 months later, DTF has not provided evidence to show that it now verifies the quality and comprehensiveness of risk assessments for projects under $100 million or undertakes its own risk assessments as part of the HVHR process.

Figure 2A

VAGO risk profiling recommendation and DTF's response

VAGO's 2013 audit on the Gateway Review Process (GRP) concluded that DTF: 'has not systematically reviewed and verified agency self-assessments of project risk, which determine whether projects are subject to the GRP. In many cases, responsible departments and agencies have not undertaken the required GRP risk assessment'. VAGO recommended that DTF: 'systematically validate whether projects should be subject to Gateway review, by verifying that robust project risk assessments are completed for new projects'. DTF's formal response to the recommendation stated that: 'As part of the High Value High Risk (HVHR) framework, DTF already undertakes risk assessments of projects to determine which projects should be classified as HVHR and subject to the Gateway Review Process (GRP). This assessment takes into account risk assessments undertaken by departments (including Project Profile Models) as well as DTF's own independent assessment of risk. DTF will examine further options to improve this process, including enhanced documentation outlining the reasons for DTF's risk assessment of projects.'

Source: Victorian Auditor-General's Office, Planning, Delivery and Benefits Realisation of Major Asset Investment: The Gateway Review Process, May 2013.

2.3.2 Application of HVHR process to significant projects announced in the 2014–15 State Budget

The 2014-15 Budget commits up to $27 billion in new infrastructure including up to $21 billion on the following two projects:

• $8.5 to $11 billion for the Melbourne Rail Link, which includes a tunnel linking Southern Cross and South Yarra stations and a new airport rail link
• $8 to $10 billion to complete the East West Link by extending the eastern section from the Tullamarine Freeway to the Western Ring Road (stage 2).

These projects are listed as HVHR projects in the Budget Papers but have not progressed through any steps in the HVHR review process normally undertaken for funded projects. This is not how the HVHR process is usually applied to projects over $100 million, which requires the Treasurer's approval to proceed to funding based on a deliverability assessment of the full business case.

Government's funding decisions on the Melbourne Rail Link and East West Link stage 2 projects require the development of a full business case in line with the HVHR process. However, DTF has advised that it intends to apply a tailored HVHR assessment process to these projects to enable project risks to be considered while supporting accelerated time frames. This tailored HVHR process is yet to be defined.

Given the significance and scale of these projects it is critical that DTF determine how it will effectively apply the HVHR process within these truncated timelines to ensure that the project costs, benefits and time lines have been robustly determined.

2.3.3 Applying the HVHR process

There are 11 steps in the HVHR review process covering project development, procurement and post-implementation variations and monitoring, as outlined in Figure 2B. None of these steps are optional. However, during the introduction of the HVHR process in 2011, where well advanced projects were brought into the process, the early stages of review were not retrospectively applied.

This audit examined application of the HVHR process to five projects, namely:

• the registration and licensing system (RandL)
• Mitcham Road and Rooks Road Rail grade separations
• Royal Victorian Eye and Ear Hospital redevelopment
• Western Highway duplication from Beaufort to Buangor
• regional rolling stock procurement.

Figure 2B maps the 11 major HVHR steps against each of the five projects we selected for detailed review.

This shows that mandated HVHR reviews have been completed, except for RandL and regional trains. This is because the HVHR process was introduced in mid-2011, after RandL had already passed through these stages. DTF did not provide evidence that the regional trains contract review or procurement document approval steps were undertaken.

The lack of a comprehensive master register showing all HVHR review activities and a structured and consistent approach to record-keeping made it difficult to extend our process analysis beyond this small sample of projects.

Applying these improvements will provide greater assurance that DTF completes required HVHR reviews.

Figure 2B

Completion of mandatory HVHR review and approval steps

#	HVHR Step	RandL	Mitcham	Eye & Ear	RRS	W. Hwy
1.	Review preliminary business case	N/A	✔	✔	✔	✔
2.	Review full business case	N/A	✔	✔	✔	✔
3.	Approval of full business case	N/A	✔	✔	✔	✔
4.	Review procurement documents	N/A	✔	✔	✔	✔
5.	Approval procurement documents	N/A	✔	✔	✘	✔
6.	Review preferred tender	N/A	✔	✔	✔	✔
7.	Approval of preferred tender/bid	N/A	✔	✔	✔	✔
8.	Review of contract	N/A	✔	✔	✘	✔
9.	Approval of contract	N/A	✔	✔	✔	✔
10.	Review of project performance	✔	✔	✔	✔	✔
11.	Approval of major variations	✔	N/A	N/A	✔	N/A

Note: 'Mitcham' refers to the Mitcham Road rail grade separation, 'Eye & Ear' refers to the Eye and Ear Hospital redevelopment, 'RRS' refers to the regional rolling stock procurement project, and 'W Hwy' refers to the Western Highway duplication project.

Source: Victorian Auditor-General's Office.

2.4 Supporting consistently high-quality reviews

We assessed whether DTF had deployed the processes and resources needed to support consistent, high-quality reviews providing:

• sufficient and well-trained analysts and supervisory staff
• clear guidance about assessment criteria and documenting reviews
• quality assured reviews that are consistent, transparent and free from role conflict and conflict of interest.

2.4.1 Capacity—analysts and supervisory staff

The HVHR process is managed within DTF's Infrastructure Advice and Delivery group. DTF recruited 11 analysts during 2011 to enhance its capacity to complete HVHR reviews. These new staff complemented existing resources and represented a significant increase in DTF's capacity to conduct reviews.

DTF committed to develop and train its analysts to improve the consistency and depth of reviews. There is evidence that analysts are provided with a range of relevant training opportunities. However, DTF is yet to develop and implement structured development and training activities for HVHR analysts.

We acknowledge the critical role of on-the-job coaching, mentoring and feedback about performance. However, a structured program is also needed to provide a consistent level of base training, together with tailored, technical training to address needs and gaps identified through regular assessments of HVHR reviews.

We have not seen evidence of DTF taking a structured approach to reviewing emerging needs and dealing with them using a range of mechanisms, such as a formal training program.

2.4.2 Guiding and documenting HVHR assessments

Developing a comprehensive set of review templates

DTF has developed comprehensive guidance for agencies to develop, procure and implement capital projects in the form of the Investment Lifecycle and High Value/High Risk Guidelines. This material has been progressively expanded and improved.

Achieving consistent, rigorous reviews across multiple HVHR projects using different reviewers represents a significant challenge. DTF recognises that review guidelines and templates should cover every stage of the HVHR process.

However, only the business case review process currently has a template that provides detailed questions addressing the review objectives and requires analysts to consistently document the results of each review. There is room to improve the current business case template to better align it with the requirements of the Investment Lifecycle and High Value High Risk Guidelines that fall within the scope of the HVHR business case review.

DTF also needs to extend the templates to other stages, including preliminary business case review and procurement.

Better documenting and managing HVHR review records

The May 2012 internal audit of HVHR found that 'DTF did not maintain documentation received from departments in a consistent or standardised way, impacting on the ability to obtain a clear audit trail of work undertaken'. It recommended that, 'A standard file structure should be implemented to assist the HVHR process'.

Standardising documents and files would provide a clear audit trail and make the HVHR process more efficient by making it easier for different analysts to understand past review stages when doing the next review stage.

DTF committed that by September 2012 it would develop 'a standardised file structure and assessment documentation to be used for all HVHR projects'.

DTF designed a file structure for this purpose in late 2012, but it has not been consistently applied. The project files we reviewed were not standardised or easily navigable, with significant differences in the type and detail of documentation retained.

This made it more difficult to assess reviews and led to follow-up requests during the audit because information was not on file. DTF should apply a standard file structure and assessment documentation for all stages of the HVHR process.

2.4.3 Quality assurance

When introducing HVHR, DTF committed to applying the process consistently and impartially. This is important, given the significant role of the HVHR process in providing government with assurance about the robustness of project planning and delivery.

We have seen evidence that senior DTF staff review all HVHR review briefings. However, these quality reviews are not in a standard form and do not clearly conclude whether HVHR reviews have adequately addressed the key criteria for assessing material submitted by departments.

For example, with a business case, the quality review should confirm or clarify the HVHR assessment conclusions about whether the analysis and evidence supporting the project's time lines, budget and expected benefits adheres to the Investment Lifecycle and High Value High Risk Guidelines.

At present, the only consistently documented evidence of review is the signature of the reviewer on the relevant brief. This does not provide assurance on the scope and adequacy of the review and is exacerbated by the fact that templates are not in place for HVHR reviews, except for the business case assessment.

A review based on a comprehensive template would provide greater quality assurance and clearly indicate what evidence had been relied on to support the assessment and recommended action.

2.4.4 Understanding and managing perceived conflicts

Concerns about role conflict and confusion created by DTF representatives sitting on steering committees or project boards and also advising government about a project's performance have been previously raised by VAGO, the Ombudsman and PAEC.

Figure 2C summarises the concerns reported by the Ombudsman and PAEC.

Figure 2C

Previous Ombudsman and PAEC findings

Victorian Ombudsman The November 2011 report Own motion investigation into ICT-enabled projects found a lack of clarity about the role of DTF representatives on project steering committees and recommended that the terms of reference for steering committees include details of the role to be played by each person on a committee. Public Accounts and Estimates Committee The December 2012 report Inquiry into Effective Decision Making for the Successful Delivery of Major Infrastructure Projects included a finding that concerns had been raised by VAGO about role confusion for DTF, which is responsible for both assisting with project proposals and assessing those proposals on behalf of government. The committee proposed a revised approach that involved DTF focusing solely on providing assurance to the investor.

Source: Victorian Auditor-General's Office, based on Ombudsman and Public Accounts and Estimates Committee reports.

Quality reviews should verify the independence of advice by disclosing and addressing any actual or perceived conflicts of interest. We found evidence of DTF staff, and in some cases the same review analyst, having contributed to the submission development by sitting on a project steering committee. This was the case for three of the five HVHR projects examined during the audit.

DTF agreed that there could be a perception of conflict of interest as a result of fulfilling multiple roles but asserted that there is no actual conflict. It further advised that any perceived or potential conflicts are effectively managed as a result of the following review layers:

• HVHR analyst advice is subject to review by their manager, director and the Deputy Secretary and Secretary of DTF
• the Deputy Secretary Budget reviews DTF advice to the Budget and Expenditure Review Committee (BERC) of Cabinet on Budget issues but has no role in the HVHR process
• DTF's HVHR committee reviews DTF's role in the HVHR process.

These practices do not change VAGO's position that potential conflicts should be documented, together with DTF's approach to managing them, so that those receiving advice are aware of them and of how they are being managed.

2.5 Monitoring and reporting

2.5.1 Communicating with government

DTF committed to more proactive monitoring of investment delivery under the HVHR process when it advised government on its introduction.

DTF's quarterly reports to BERC on HVHR projects and its regular DTF HVHR accountability report are effective in informing DTF senior staff and government on project progress in terms of outputs, time lines and budgets, and about project risks.

However, these reports do not routinely address the achievement of intended project benefits, and projects are typically removed from these reports once implemented.

In addition, these reports have not comprehensively covered all HVHR projects from when they were designated as being subject to the review process. Our analysis shows a lag of over three months for some projects between the time they were designated as HVHR and when they were first included in major project monitoring reports to government and DTF.

The May 2014 internal audit identified the need for more robust monitoring of, and reporting on, progress against proposed benefits—financial and non-financial—throughout the project lifecycle.

DTF recently developed and released to agencies a template to track broader project benefits against those proposed in the business case for all asset investments—i.e. HVHR and non-HVHR investments.

2.5.2 Sharing lessons and better practice with agencies

Periodically reporting on progress and emerging lessons, as well as understanding agencies' views on the benefits and drawbacks from reviews, is critical to realising the potential of the HVHR process.

DTF previously committed to sharing and applying lessons learned and to completing an annual satisfaction survey across participating agencies.

DTF has not fully met these commitments and needs to act on these overdue actions. The evidence shows that:

• DTF manages HVHR reviews on a project-by-project basis and has not systematically communicated key lessons across the sector
• the May 2012 internal audit also recommended that DTF enhance stakeholder communication and engagement. While engaging constructively and intensively around individual project reviews, DTF does not have a structured approach to understanding and learning from agencies' combined experiences of the HVHR process because it does not formally survey them.

Notwithstanding this, DTF provided evidence of actions taken to share some lessons with agencies through presentations and ongoing informal communication. DTF has also progressively updated and enhanced the Investment Lifecycle and High Value High Risk Guidelines material and shared feedback on the quality of HVHR business cases with the Victorian Infrastructure Policy Reference Group. This group includes representatives from government departments.

We sought feedback from agencies on their experience of, and perspectives on, the HVHR process. This feedback was largely positive but agencies were consistent in their desire to see DTF communicate any common lessons or issues identified from distilling the results of HVHR reviews undertaken to date. Other commonly raised issues and comments included:

• a lack of clarity about the timing and scope of application of the HVHR process during its initial introduction
• a gradual improvement in clarity about information requirements for reviews
• time pressures in preparing business cases and adequately addressing issues raised by HVHR reviews
• some projects costing over $100 million are not necessarily complex or high risk because they have common characteristics and are routinely delivered successfully by agencies and so do not necessarily warrant additional scrutiny under the HVHR process.

2.6 Evaluating the impacts of the HVHR process

Like any government-funded program, the most important thing to understand is how well the investment has achieved intended objectives—i.e. making projects more predictable in terms of delivering on time and within budget and realising the benefits that underpin business cases.

DTF committed to undertaking an independent evaluation of the HVHR process and reporting the results to government by June 2014. This review has not been commissioned, and we have seen no evidence of progress towards this. DTF has not developed an evaluation framework, nor collected and processed the data needed to measure the outcomes attributable to the HVHR process.

This is a significant gap that DTF needs to address.

VAGO's recent audit on the Gateway Review Process found that in 2004 the Department of Premier and Cabinet developed a detailed review tool to be used over time to assess the impact of the Gateway process against its underlying objectives, outlined when the process was introduced. This tool was developed in consultation with DTF and departments. The Gateway Supervisory Committee and the government endorsed the review tool in August 2004

The review tool provides a sound basis for ongoing review of the impact of the Gateway Review Process. Despite this, and the government's endorsement, DTF informed us that it has not implemented this tool to evaluate Gateway, nor adapted the tool to evaluate the impact of the HVHR process.

This tool was developed for the Gateway Review Process and would be a good starting point for DTF to measure the impact of the HVHR process.

Recommendations

That the Department of Treasury and Finance:

1. improves its approach to selecting projects for inclusion in the High Value High Risk process by:

• systematically reviewing projects between $5 million and $100 million to determine whether they should be subject to the High Value High Risk process through documented reviews of project risk assessments
• clarifying the criteria for selecting public projects that do not require Budget funding for inclusion in the High Value High Risk process
• recommending that projects over $100 million selected under the government's unsolicited bids policy be subject to High Value High Risk processes

2. develops assessment guidance and templates covering all High Value High Risk stages to improve the consistency, rigour and transparency of High Value High Risk reviews
3. improves its administration of the High Value High Risk process by:
4. applying a standardised file structure for managing assessment documents and supporting evidence
5. developing and maintaining a comprehensive central register of High Value High Risk review activity by project and High Value High Risk process stage

4. improves how it communicates with and informs departments by:
5. developing a structured approach to collating and sharing the lessons from all High Value High Risk reviews
6. completing an annual satisfaction survey of agencies that have been subject to High Value High Risk reviews

5. identifies potential conflicts of interests of reviewers and documents how these are mitigated
6. develops and applies an evaluation tool to measure the extent to which the High Value High Risk process is affecting project outcomes.

3.1 Introduction

This Part examines the impacts of High Value High Risk (HVHR) reviews on the quality of business cases and procurements for the following five projects:

• registration and licensing system (RandL)
• Mitcham Road and Rooks Road rail grade separations (Mitcham Road)
• Royal Victorian Eye and Ear Hospital redevelopment (Eye and Ear Hospital )
• Western Highway duplication from Beaufort to Buangor (Western Highway)
• regional rolling stock procurement (regional trains).

Towards the end of the audit we examined a further three projects to assess the impact of the Department of Treasury and Finance (DTF) updating its business case assessment template in late 2012.

Royal Victorian Eye and Ear Hospital.

Section 3.3 examines whether HVHR assessments of business case deliverability were robust. Section 3.4 examines the procurement reviews and Treasurer's approval of tender documents, preferred bid decisions and the final contracts. Both use the relevant parts of DTF's Investment Lifecycle and High Value High Risk Guidelines as the basis for these assessments.

Section 3.5 summarises our findings for the RandL project, which is the one project that most clearly failed the deliverability test. We also use this section to explain why this happened and the lessons to be drawn from this.

3.2 Conclusion

DTF has improved the scrutiny and quality of business cases and procurements captured by the HVHR process. However, these improvements have not lifted practices so they consistently and comprehensively meet DTF's better practice guidelines. There are gaps and inconsistencies which DTF needs to address if it is to fully realise the potential benefits of the process.

HVHR business case reviews raised substantive issues and agencies modified submissions to address these issues. However, these positive findings were not consistently reflected across all projects nor for all the criteria that DTF uses to test projects' deliverability.

DTF's level of scrutiny and assurance needs to improve across all the 'deliverability' criteria. Its performance was best for assuring project costs, time lines and agencies' approaches to risk management, more mixed for procurement, governance and project management and clearly inadequate for assuring the expected project benefits.

None of the business cases adequately justified projects' benefits and this makes the absence of any substantive HVHR review of the area a cause for concern.

RandL was not fully captured by the HVHR process because its procurement happened well before the process started. However, the updated late-2011 business case was subject to a HVHR type review, and the project was included in the process from late 2011.

The management and oversight applied to the RandL project was not effective in detecting and controlling risks before they materialised to significantly delay the project and increase the estimated cost of delivery. The single most critical cause of this was the absence of sufficiently skilled resources.

3.3 Business case reviews

We tested the quality of the assurance provided by HVHR reviews against the criteria in the HVHR Business Case Deliverability Assessment Guidance document, which replaced an earlier, less detailed guide that used the same criteria.

We applied the Investment Lifecycle and High Value High Risk Guidelines criteria to assess the quality of assurance. Our assessment is based on the documentation provided by DTF.

Figure 3A describes our assessment of HVHR reviews of the business cases of the different projects against six criteria that include a combined assessment of governance and project management.

DTF applied the greatest amount of scrutiny and achieved the highest level of assurance in the areas of project budgets, time lines and risk management.

In contrast, the level of assurance in relation to project benefits, preferred procurement approach and governance and management was lower. In particular, DTF applied little scrutiny to, and asked few questions about, the reliability of project benefits, even though business cases did not contain sufficient information to verify the scope and quantum of these benefits.

There is considerable room for DTF to improve its performance in this area. Some improvements do not require much change—for example, making sure that business cases are updated where additional significant information is provided. However, other improvements, such as adequately testing the reliability of project benefits, requires a major change in approach and acquisition of skills needed to do this.

Figure 3A

VAGO assessment of assurance against HVHR requirements for project business cases

Deliverability criteria	Mitcham	Eye&Ear	RRS	W Hwy	RandL
Deliver on budget—assurance about costs	✔	?	✔	✔	?
Deliver to planned time lines	✔✔✔	✔✔✔	?	✔	?
Deliver intended benefits:	?	?	?	?	?
Comprehensively manage risks	✔	?	✔	✔	?
Governance and project management	?	?	?	✔	?
Procure for best value for money	✔	✔✔✔	?	✔	NA

Note:

✔✔✔ = Fully assured—There was an updated, reviewed business case which fully addressed the Investment Lifecycle and High Value High Risk Guidelines and provided a high level of assurance.

✔= Improving assurance—There was evidence that HVHR reviews improved the level of assurance while not fully addressing all issues. The significance of these outstanding issues varies from relatively minor, e.g. where business cases were not updated with additional evidence, to more significant, e.g. where DTF has not raised or the agency has not addressed material issues. We provide a commentary on these issues below.

? = Not assured—There was insufficient evidence to be assured about the deliverability of the project.

NA = Not applicable.

Note: 'Mitcham' refers to the Mitcham Road rail grade separation, 'Eye&Ear' refers to the Eye and Ear Hospital redevelopment, 'RRS' refers to the regional rolling stock procurement project, and 'W Hwy' refers to the Western Highway duplication project.

Source: Victorian Auditor-General's Office.

3.3.1 Deliver on budget—assurance about costs

At this HVHR stage departments need to provide a bottom-up, detailed cost estimate, including appropriate and evidence-based allowances for contingencies and risks. We found that DTF applied significant effort in testing cost estimates by questioning their basis and working with departments to address their questions.

Figure 3B shows we rated the RandL and Eye and Ear Hospital projects as 'Not assured' because of significant, unaddressed cost uncertainties. For the remaining projects, we found that DTF's review had clearly improved assurance.

Figure 3B

Findings on costing the recommended solution

Project	Assessment	Reasons
Mitcham Road	Improving assurance	DTF gained greater confidence about the estimated cost of the preferred option by obtaining additional information on the method applied, external review results, cost benchmarking and sensitivity analysis and the rationale for risk and contingency allowances. DTF concluded that the recommended solution could be delivered at this cost. VicRoads did not update the commentary in the business case to include this essential, additional evidence but it was included as an addendum.
Eye and Ear Hospital	Not assured	DTF was not satisfied that the recommended solution could be delivered for the initially estimated cost because the underlying assumptions were unrealistic. The estimates appeared to be at a preliminary stage, applying generic costs to estimated floor areas, and the size of the contingency appeared low for this type of project. DTF endorsed the updated business case cost, which was unchanged from the government's publicly announced cost, but with a higher contingency. While this estimate included updated costings based on more detailed planning and design for early works, critical infrastructure and decanting, the cost estimates for the vast majority of the building works had not gone beyond the 'preliminary stage'—estimates DTF deemed inadequate. The updated business case did not address all material issues.
Regional trains	Improving assurance	DTF identified that the business case contained insufficient detail on the project costs. Public Transport Victoria provided additional information on the purchase and operating costs and included some of this information as an addendum to the business case. DTF concluded that the costs, including risk allowances, were robust but there was still insufficient information in the business case. We note that immediately before contractual close the government increased the number of trains purchased because the per unit price was less than expected and it decided to buy more trains within the budget allocated.
Western Highway	Improving assurance	DTF questioned the adequacy of the cost information in the business case and VicRoads addressed issues about the costing method, risks provisions, external review and benchmarking, sensitivity analysis and the basis for contingencies. The updated business case included additional information, but we are not convinced it adequately addressed all of DTF's material questions. In addition the costs were revised downwards by $67 million after submission of the business case because of a scope reduction but VicRoads adequately explained this.
RandL	Not assured	See commentary under Section 3.5.

Source: Victorian Auditor-General's Office.

3.3.2 Deliver to planned time lines

The Investment Lifecycle and High Value High Risk Guidelines require the business case to include a detailed and realistic project implementation schedule which encompasses key milestones, decision points and delivery events.

Figure 3C shows that we assessed the reviews as fully assuring the time lines for Mitcham Road and the Eye and Ear Hospital, improving assurance for Western Highway but not assuring the time lines for the remaining two projects.

Figure 3C

Findings on business case information on project time lines

Project	Assessment	Reasons
Mitcham Road	Fully assured	The business case included high level information on key milestones, with a full project construction program in an appendix. The construction program was for a completion date one year earlier than the summary presented in the body of the business case. DTF assessed the time lines as reasonable because the information on critical paths and the estimated duration of tasks were transferable to a new completion date.
Eye and Ear Hospital	Fully assured	The business case included adequate information on the project implementation schedule, including key milestones, decision points and delivery events supported by a master program showing detailed activities and time lines for the project.
Regional trains	Not assured	DTF accepted the time lines as reasonable because the business case included an indicative time line for major project deliverables and information on the timing of other planned major changes that would impact on regional rolling stock. However, this was not supported by a detailed program showing activities and time lines for the project, and we do not understand the basis for DTF's acceptance.
Western Highway	Improving assurance	DTF accepted the time lines as reasonable because the business case included a time line showing key tasks and milestone dates for the project. The business case did not include a more detailed program. A key risk to the time lines was the Environmental Effects Statement process and the business case should have included a detailed program incorporating the management of this risk. However, DTF sought and obtained additional information from VicRoads to support its conclusion that the time lines were reasonable and this information was included in the final business case.
RandL	Not assured	See commentary under Section 3.5. The business case information on project time lines and implementation phasing was out of date, as it was based on a phased approach to implementation which did not reflect the fact that by mid-2011 VicRoads was pursuing a single-phase implementation. Prior to the project being paused in early 2014, the implementation approach had been reversed to a multi-phase deployment because of the risks associated with single-phase implementation.

Source: Victorian Auditor-General's Office.

3.3.3 Deliver the intended benefits

Delivering service outcomes to benefit the community is the key reason for spending public money on infrastructure projects. Managing projects to secure, safeguard, maximise and realise the benefits is therefore essential. This requires an evidence‑based understanding of the expected benefits and any risks to them being realised.

It is worrying that such little review effort has been focused on confirming and securing project benefits. None of the business cases we reviewed included sufficient information to understand either the benefits or risks to their realisation. HVHR reviews have focused little or no effort on examining project benefits but instead focus on costs, time lines and cost-related risks. Figure 3D describes the results.

Figure 3D

Findings on business case coverage of benefit definition and delivery

Project	Assessment	Reasons
Mitcham Road	Not assured	The business case did not adequately explain the basis for defining the problems being addressed and estimating the benefits of the recommended solution. There was no evidence that DTF asked any questions about the project benefits, despite the lack of detail on these calculations. The attribution of wider economic benefits to the project without any reference to national and international guidelines on the appropriate use of these techniques went unchallenged.
Eye and Ear Hospital	Not assured	The business case did not adequately quantify the problems that the redevelopment was supposed to address, including inefficient scheduling and resource utilisation, safety and performance risks from ageing infrastructure, and the capacity to meet future demand. None of these problems were adequately quantified and demonstrated to serve as the basis for measuring the benefits of improvement options. Consequently, the benefits of the development are anecdotal, being based on statements about why the hospital needs to be redeveloped. The inability to quantify the problems being addressed and the benefits in the business case should have been raised by DTF.
Regional trains	Not assured	The project's expected benefits are presented with no supporting information on the evidential basis for and source of these benefits. The business case indicates that it is probable that the full benefits would not be achieved without an additional investment in high-capacity Geelong line trains. We found no scrutiny of these benefits by DTF during the HVHR review.
Western Highway	Not assured	The business case provided no information on the benefits of this section of the upgrade. Instead it referred to the cost‑benefit analysis for the whole route, showing a benefit‑cost ratio that was significantly less than one and provided no economic justification for this project.
RandL	Not assured	See commentary under Section 3.5.

Source: Victorian Auditor-General's Office.

3.3.4 Comprehensively manage risks

DTF's Investment Lifecycle and High Value High Risk Guidelines require the business case to:

• clearly identify and assess key risks associated with the proposed investment
• describe the proposed approach to ongoing risk monitoring and management
• document a risk management strategy.

Figure 3E shows that RandL and the Eye and Ear Hospital did not meet the requirements, while HVHR reviews improved assurance for the remaining three projects, although outstanding unaddressed issues prevent us giving these a 'Fully assured' rating.

Figure 3E

Findings on business case coverage of risk

Project	Assessment	Reasons
itcham Road	Improving assurance	The business case identified key project risks, explained how they were identified and included a combined risk register and risk management plan. However, it did not include a description of the proposed approach and arrangements for ongoing risk monitoring and management, and DTF did not raise this during the review.
Eye and Ear Hospital	Not assured	The business case included a brief description of the proposed approach and arrangements for ongoing risk monitoring and management, identified key project risks, explained how they were assessed and included a combined risk register and management plan. DTF raised the lack of a comprehensive risk register in the business case, but the department did not address this in the final version of the business case. This is a concern, given the risks around refurbishing and rebuilding a hospital while still operating from the facility and a relatively small sum allocated to contingency.
Regional trains	Improving assurance	The business case included an appropriately detailed description of the proposed approach and arrangements for ongoing risk monitoring and management, identified key project risks, explained how they were identified and included a combined risk register and management plan. However, the key commercial and legal risk identified in the body of the business case related to an external approval process for the proposed single source procurement approach that was not applicable to the project and so was irrelevant. Also, the risk management section did not highlight the risk to benefit realisation of not securing additional investment in high-capacity Geelong line trains.
Western Highway	Improving assurance	The business case identified key project risks, explained how they were identified and included a risk register. However, it did not adequately describe the proposed approach and arrangements for ongoing risk monitoring and management.
RandL	Not assured	See commentary under Section 3.5. Business case included material on risk analysis and management, implementation risks and an appendix on risk. However, it did not include a risk register, and the material that was included was largely focused on determining the required level of contingency for the project. Subsequent events show the risks were underestimated and not well managed.

Source: Victorian Auditor-General's Office.

3.3.5 Apply appropriate governance and management

DTF's Investment Lifecycle and High Value High Risk Guidelines require the business case to demonstrate sound governance and project management arrangements.

Figure 3F shows that none of the sample business cases we examined included sufficient explanation and evidence to fully assure the proposed governance and management arrangements. While some HVHR reviews identified this issue, the weaknesses were not addressed. For the Western Highway we found that the review had improved assurance about these arrangements.

Figure 3F

Findings on business case coverage of project governance and management

Project	Assessment	Reasons
Mitcham Road	Not assured	The governance material in the business case was not sufficient to demonstrate the soundness of the proposed arrangements. It was high level, did not include decision-making processes and did not provide sufficient detail on how governance would operate under the alliance delivery model.
Eye and Ear Hospital	Not assured	The governance material in the business case was not sufficient to demonstrate the soundness of the proposed arrangements. DTF's HVHR assessment of the December 2011 business case identified a range of weaknesses but these were not addressed. DTF's assessment of the updated August 2012 business case concluded that the governance arrangements were appropriate or standard for a capital project of this type, even though the text in these sections remained unchanged from the initial business case.
Regional trains	Not assured	The governance material in the business case was not sufficient to demonstrate the soundness of the proposed arrangements. It was high level, and did not include decision-making processes.
Western Highway	Improving assurance	The business case did not sufficiently demonstrate the soundness of the proposed governance arrangements. It was high level, consisting of three paragraphs in the body of the business case, did not include a chart explaining decision-making processes and so did not provide sufficient detail on how governance would operate. DTF's HVHR assessment of the business case concluded that the governance structure identified was appropriate for the investment, but could be augmented to include DTF. As part of the HVHR review, DTF obtained additional information from VicRoads to demonstrate the project management arrangements in place to secure required resources, including the required mix of skills, a change management strategy and a plan to move from delivery to asset operation. In summary, we are assured about project management but not project governance.
RandL	Not assured	The governance material in the business case was not sufficient to demonstrate the soundness of the proposed arrangements. A high level governance structure was included, but there was no real detail on roles, responsibilities and decision-making processes. A DTF-led review of the business case in mid-2011 found that it outlined the high level program governance only and would have benefited from more detail as to how lower level governance was intended to work. This review recommended that Victoria Police and Department of Justice representatives be added to the program board.

Source: Victorian Auditor-General's Office.

3.3.6 Best value-for-money procurement

The Investment Lifecycle and High Value High Risk Guidelines require a range of procurement options to be considered and analysed. This analysis should be sufficient to demonstrate the basis for selecting the recommended procurement approach. Figure 3G sets out our findings on the extent to which the five sample business cases met these requirements.

Only one of the business cases was assessed as meeting the requirements. Three business cases did not fully meet the requirements, primarily because they lacked supporting information or analysis to justify the assessments of different procurement approaches against the evaluation criteria.

Figure 3G

Findings on business case analysis of procurement approaches

Project	Assessment	Reasons
Mitcham Road	Improving assurance	The business case did not adequately demonstrate that the recommended alliance contracting approach represented the best value-for-money procurement option. The business case included a single table with tick box assessments for three options across multiple criteria. It did not demonstrate how the preferred alliance approach met the requirements of the national alliancing guidelines. DTF's HVHR assessment identified this weakness and obtained further explanations and evidence from VicRoads. This material was added to the final business case.
Eye and Ear Hospital	Fully assured	The business case included a comprehensive and adequately supported analysis and evaluation of available procurement methods and the basis for selecting the preferred option. DTF noted that the recommended non-fixed fee construction management model probably provides better value for money for this type of project, and that the Department of Health was retaining significant project risks and would need to manage the interfaces between different contractors and the extended staged program of works.
Regional trains	Not assured	The business case did not adequately justify the basis for the selection of the recommended procurement model—a single source approach to an existing supplier. The primary justifications were time constraints, compatibility concerns and an asserted capacity to ensure value for money from the single source model. The agency did not adequately address DTF's concerns about this procurement route. DTF's final assessment supported the procurement approach primarily due to the time constraints and the mitigation of having an 'open book' arrangement with the supplier. However, this assumed that the underlying core contract pricing was still competitive although it had not been market tested for more than 10 years.
Western Highway	Improving assurance	The first business case submitted did not adequately demonstrate that the procurement model—design and construct—would provide greater value for money. DTF raised this, and VicRoads provided further explanations and evidence and incorporated this material in an amended business case. While DTF was satisfied with the updated business case, it did not assess the procurement options against all evaluation criteria.
RandL	Not applicable	The procurement phase was completed well before the updated business case went through a HVHR type process.

Source: Victorian Auditor-General's Office.

3.3.7 Further reviews using an updated template

DTF analysts assessed the business cases for the five projects examined earlier in this section using a HVHR template dating from early 2012. DTF updated this template in November 2012. The scope of the HVHR business case reviews did not change over this time, but the updated template is better structured and provides analysts with more detailed guidance on questions to address when reviewing a business case.

We conducted a rapid assessment of three further business cases to determine whether using the updated template had affected the quality of HVHR reviews for:

• the Blackburn Road grade separation
• the Kilmore–Wallan bypass
• Southland station.

We concluded that the new template did not materially change our findings and conclusions from the earlier analysis of five business cases. Our review of the three business cases that were assessed by DTF using an updated assessment template shows that opportunities remain to improve the depth and rigour of HVHR reviews.

Consistent with our findings on the HVHR reviews of the initial sample of business cases, DTF applied the greatest scrutiny and achieved the highest level of assurance for project budgets, time lines and risk management.

When we reviewed the assessments of these further business cases against each of the deliverability criteria we found:

• Deliver on budget—assurance about costs—DTF continues to mainly focus on scrutinising the cost of the recommended solution.
• Deliver to planned time lines—DTF's level of inquiry is similar to that exercised using the initial template. The business case content was adequate for the Blackburn Road grade separation, but the material provided in the Kilmore Wallan bypass and Southland station business cases did not fully comply with the guidelines.
• Deliver intended benefits—we again found that this is an area subject to minimal, if any, scrutiny by DTF.
• Comprehensively manage risks—while DTF did focus its efforts on reviewing agencies' risk management, we are not fully assured about the adequacy of the risk identification and management material in the business cases for the Blackburn Road grade separation and Southland station projects.
• Governance and project management—use of the updated template has not addressed our concern that DTF's HVHR assessments do not provide sufficient assurance about the adequacy of governance and project management.
• Procure for best value for money—use of the updated template has not fully addressed our concern that DTF does not gain sufficient assurance that agencies adequately demonstrate the basis for selecting a preferred procurement approach. None of the business cases met this standard, and DTF only identified this as an issue for two of the projects—the Blackburn Road grade separation and Southland station.

HVHR review of the Southland station project

Southland station was included in, then withdrawn from, the HVHR process when DTF understood that the major issues affecting its deliverability had been resolved. However, there was no documented confirmation that the substantive deliverability issues raised in DTF's HVHR reviews had been satisfactorily resolved. This should have formed part of the close-out process.

Building Southland station was a November 2010 election commitment and the government allocated $700 000 in the 2011–12 Budget to develop a business case.

While the station cost falls well short of $100 million, the government included the project in the HVHR process because:

• completion required access to land owned by the adjacent shopping centre
• the actual scope and cost depended on negotiations with the shopping centre about how best to link it to the station and the size of the shopping centre's financial contribution to the project.

DTF reviewed versions of the business case in January 2012 and January 2013 advising, on both occasions, that the project should not be considered for funding because of deficiencies in these business cases which included:

• lack of clarity around the shopping centre's financial contribution
• absence of an access agreement to allow construction risk to be effectively managed
• insufficient detail around the costs, time lines, risk management, governance, procurement and project management.

The government decided to include the project in the 2013–14 Budget, allocating funding in the central contingency, with its release subject to:

• reaching agreement on the shopping centre's financial contribution and access
• approval being secured under the HVHR process.

In April 2014, the Treasurer approved release of funding for the project and its removal from the HVHR process based on confirmation that an access agreement had been reached with the owner of the shopping centre and that this did not involve any financial contribution from the owner.

However, the business case had not been updated and subjected to further HVHR review to confirm that deficiencies identified in previous reviews had been satisfactorily addressed following resolution of the agreement with the owner of the shopping centre.

3.4 Tendering and appointment reviews

3.4.1 Introduction

We used the Investment Lifecycle and High Value High Risk Guidelines to inform our review of DTF's HVHR assessments of tender documentation, the preferred bid decision and the final contract prior to sign-off. The assessments are meant to ensure that:

• the preferred procurement approach is effectively implemented and translated into a contractual framework that continues to represent the best value for money
• the risks of diminishing this outcome through unintended scope creep or a contractual form that diminishes the outcomes for the state are managed.

The additional approval requirements for HVHR projects include reviews and Treasurer's approvals at the following stages:

• expression of interest—prior to release
• request for tender/proposal—prior to release
• preferred bid selection—prior to announcement
• final contract—prior to signing and any major variations.

The role of DTF is to analyse the documentation and advise the Treasurer, using high level guidance and a standard briefing format. However, in contrast to its analysis of business cases, DTF has not developed a detailed assessment template to guide reviews at these other stages.

3.4.2 Overall findings

We found evidence that DTF:

• Completed the required reviews for all five projects, as required by the guidelines, except for the approval of the request for proposal document and contract review for regional trains. However, DTF adequately justified this because the Treasurer had approved the invitation of a single bidder through the procurement strategy document and the contract used was identical to that used for previous rolling stock procurements.
• Actively engaged departments for each of these projects and asked a range of reasonable questions to which departments responded.

For two of the five projects the same DTF staff were involved in both advising the Treasurer on HVHR procurement reviews and assessing the bid(s). For:

• regional trains, the DTF analyst sat on the procurement panel and wrote the memos advising the Treasurer—these were reviewed by line management
• the Eye and Ear Hospital, a DTF assistant director sat on the procurement panel and reviewed the memos advising the Treasurer.

We understand that this advice was subject to input and review by DTF staff apart from the person with a potential conflict. However, potential conflicts should be documented, together with an approach to managing them.

We found that in assessing expressions of interest, request for tenders, preferred bids and final contracts, DTF did not apply a consistent approach to examining the underpinning documents, such as procurement, probity, risk management and tender evaluation plans. Figure 3H shows our detailed findings on the procurement reviews.

Figure 3H

Findings on HVHR procurement reviews

Project	Findings
Mitcham Road	All reviews were completed as required and advice was provided to the Treasurer with clear evidence of DTF's input to the development of the procurement documents. DTF raised issues about the practice of allowing contractors to work on preliminary cost estimates and constructability assessments and then compete for the main tender. The Treasurer's approval releasing the tender documents noted that DTF would review the issue. DTF advised that it took action to ensure that all bidders on this project had access to exactly the same information about the tender. DTF has recently released a draft guidance note on early contractor involvement and other collaborative procurement models that clearly indicates that contractors involved in the development of a business case should be precluded from bidding for development or delivery of the project.
Eye and Ear Hospital	All reviews were completed as required and advice was provided to the Treasurer with clear evidence of DTF's input to the development of the procurement documents.
Regional trains	The procurement method approved for this project involved a single source approach to an existing supplier of rolling stock being used on the regional rail network. All except one of the approvals were completed as required and advice was provided to the Treasurer with clear evidence of DTF's input to the final procurement documents. In terms of the exception, the Treasurer approved negotiations with a single bidder but did not approve the request for offer document sent to this bidder in July 2012, as required under HVHR. DTF explained that this involved seeking a proposal from a single supplier, and the Treasurer had approved the procurement strategy document. In October 2012, the Treasurer approved the contract based on advice from DTF that it was identical to previous contracts. However, we have not seen evidence about how DTF determined this. The Treasurer approved the purchase of 34 train carriages, consistent with an earlier funding decision by Cabinet. The cost was less than the funding allocated by the government, and in late November 2011 the government directed V/Line and Rolling Stock Holdings—the state—to amend the contract to purchase 40 carriages. The Treasurer did not formally approve the revised contract as required under HVHR. The decision was made by the relevant ministers and communicated to Public Transport Victoria and DTF. DTF advised that the Treasurer was involved in this final decision‑making process. The speed of the decision meant the agency did not provide advice to government on the implications of this change for incremental operating costs.
Western Highway	All reviews were completed as required and advice was provided to the Treasurer with clear evidence of DTF's input to the development of the procurement documents. During the expressions of interest review DTF raised issues about the procurement approach in the business case and the completion date, in light of a revised project scope and cost. VicRoads secured the Treasurer's approval to release the expressions of interest documents but subsequently decided to abandon the expressions of interest stage and go straight to request for tender. While the decision appears reasonable, VicRoads should have sought the Treasurer's approval to vary the process, but did not. The Treasurer approved the outcome of the procurement process for this project and the awarding of the contract in March 2014.
RandL	The commencement of this project significantly pre-dated the introduction of the HVHR process. As a result, the primary procurement activity did not undergo review and approval under the HVHR process.

Source: Victorian Auditor-General's Office.

3.5 Registration and licensing system project

3.5.1 Introduction

We have dedicated this section to the RandL project because past management actions and oversight efforts have not stopped the realisation of significant delays and cost overruns.

The project is meant to transform registration and licensing processes and the supporting information and communications technology (ICT) systems in order for VicRoads and other agencies that rely on these systems to:

• improve data integrity and accessibility
• improve efficiency, flexibility and responsiveness
• put customers at the centre of the upgraded system
• reduce the potential for fraud.

Figure 3I shows the project's major milestones, describing:

• the approval of a 2009–10 business case for the system costed at $173 million
• how the project was paused in mid-2011 and restarted in late-2011 with a revised, approved cost of $158 million and a scheduled delivery of the entire system through a single phase deployment by mid-2014
• how the project was paused again in early 2014 because the projected costs had increased to $293 million and the time lines extended by 18 months.

3.5.2 Conclusion on RandL's management and oversight

The management and oversight applied to RandL did not effectively foresee and control risks before they materialised to significantly delay the project and increase the estimated cost of delivery. VicRoads has proposed additional expenditure of up to $135 million more than the approved budget of $158 million to finish it. This would mean a total project cost of $293 million—85 per cent more than the approved budget of $158 million.

The single most critical cause of these poor outcomes has been the absence of human resources with the skills and depth of experience needed to manage and oversee a project of this type and complexity.

Critical gaps existed in three key areas—management, oversight and governance, and independent review—which did not detect and respond to the risks inherent in this type of project up to the middle of 2013 because:

• VicRoads did not appoint a project director with sufficient ICT implementation experience until the middle of 2013. This new executive played a key role in understanding and starting to address this shortfall.
• The board did not include sufficient ICT project deployment and business change experience until April 2013.
• The review of the 2011 business case that informed DTF's advice to government on the continuation of the project in late 2011 did not adequately identify and assess the risks.

VAGO's 2008 guide Investing Smarter in Public Sector ICT: Turning principles into practice clearly describes the need to understand the role of business process restructuring and the agency's capability to execute an ICT investment before finalising a business case.

Figure 3I

Key stages in the RandL project

2009–10—business case and funding for initial design and procurement: 2009–10 business case approved with estimated cost of $173 million funding of $38.9 million—$15 million previous funding and new allocation of $23.9 million. July 2010—contract signed for design, implementation and operation: progress beyond initial design subject to government approval and further funding. March to May 2011—design completed and project paused: March 2011—initial core design considered substantially complete May 2011—project put on hold, with DTF review to confirm scope and cost to complete. August 2011—DTF received consultant's review report: DTF commissioned a review based on an updated business case with the same cost consultant concluded the project had been well managed but raised the following concerns: lack of a detailed plan for a single phased, rather than staged, implementation existence of a number of outstanding change requests to the system design lack of a clear data cleansing and migration plan (RandL team, not contractor responsibility) consultant recognised that these concerns generated cost and time line uncertainties and recommended: either pausing and addressing these now, or proceeding with a 30–35 per cent contingency for a total cost of $175 to $182 million. November 2011—funding decision: VicRoads requested total funds of $178.6 million for the project, including the $44 million spent to date, or an additional $89.6 million and $40 million in contingency funds government approved $158.6 million funding, with $20 million contingency, requiring VicRoads to: develop, and the Treasurer to approve, detailed plans for a single phase implementation and data cleansing and migration assess and accurately cost change requests and strengthen its ICT management capability. 2012–13—remobilising the project and addressing the funding conditions: early 2012—project remobilised and work started on meeting the funding conditions 9 July 2012—Treasurer approved the revised schedule for single phase implementation and VicRoads' Data Cleansing and Migration Strategy based on DTF's advice March–April 2013—VicRoads employed an executive director for the Registration and Licensing area with direct experience of this type of transformation and an independent board member with expertise in ICT implementation 25 May 2013—based on DTF's advice the Treasurer approved the release of $6.58 million from the project contingency to fund the project's remobilisation and unavoidable system changes. 2013–14—Events leading to the second project pause: June 2013—VicRoads forecast an 18-month delay in delivery and an emerging, significant funding gap, and DTF red flagged the project and commissioned a review June–September 2013—VicRoads strengthened project governance by appointing a project director with significant ICT deployment and business change experience February 2014—government paused the project based on DTF's advice that it would now, if continued, take at least a further 18 months and cost $293 million ($135 million over budget) because: the 2011 business case and late 2011 Budget and Expenditure Review Committee submission significantly underestimated the time and costs required to complete the project there had been inadequate engagement, especially during the design phase of the project, between the RandL project and other areas of VicRoads and also with external agencies greater than expected design changes affected project complexity and delivery VicRoads and DTF underestimated the skills and resources required to effectively manage and oversee a project of this type through to delivery.

Source: Victorian Auditor-General's Office based on Department of Treasury and Finance and VicRoads documentation.

3.5.3 Underpinning evidence

RandL project complexity and challenges

RandL is an ICT project where VicRoads is responsible for replacing the multiple systems used for road vehicle and marine recreational vessel licensing and registration.

The current systems are old, difficult to use and expensive to maintain and modify, and they do not provide reliable data or adequate protection against fraud. They also constrain the government from making regulatory and policy changes and participating in national improvement initiatives. The Department of Transport, Planning and Local Infrastructure (DTPLI) advised that there have been system enhancements since 2011 to mitigate these issues and risks.

The term 'ICT' can be misleading because this type of project involves much more than specifying and purchasing new technology—it is a business transformation project where new technology and revised policies, practices and behaviours need to fit together if the project is to succeed.

Successfully delivering a project of this type and complexity requires capabilities and experience outside of VicRoads' core competencies around the planning, construction, maintenance and renewal of road-related infrastructure. Specifically, it requires:

• expertise in specifying and procuring ICT hardware and software from major ICT vendors and managing the design and implementation of these systems
• an understanding of current systems and practices and how to cost-effectively achieve intended outcomes through a combination of ICT investments and changes to policy, regulation and work practices
• a successful track record in integrating business needs and constraints in every aspect of project development and delivery, and a good understanding of the planning, consultation and training needed to do this successfully.

Overall finding—management and oversight gaps

Our examination of the evidence shows that VicRoads and DTF put in place the type of management and oversight structures and processes expected for a major government investment. These included:

• appointing an executive director and management team
• regular project reporting
• setting up a project board, with agency, DTF, Department of Premier and Cabinet, and independent representatives, as well as regular meetings to monitor progress
• DTF reviews of the 2009–10 business case and the equivalent of a HVHR review of the updated 2011 business case
• incorporating the project within the HVHR process from late 2011.

However, these structures and processes failed to adequately comprehend and control risks before they materialised to significantly delay completion and increase the estimated cost well beyond the approved project budget.

DTF independent review, September 2013

DTF commissioned a further review of the project, based on advice from VicRoads in June 2013 forecasting an 18 month delay in project delivery and an emerging, significant funding gap.

The September 2013 independent review confirmed the serious weaknesses in the project's management and oversight up to this point by concluding that:

• The approved budget and scope did not deliver a working solution because VicRoads estimated it needed $61 million more than the approved budget to cover the change requests, and this excluded a range of other cost risks.
• The project was not as advanced as the project team though it was. When testing, deployment and change requests were considered, the project was only 40percent complete, compared to the team's estimate that 85 per cent of the core solution had been built.
• The poor definition of the project scope had led to change requests as more detailed analysis uncovered new requirements. The project was set up to replace legacy systems and streamline processes, but did not have a mandate to transform business operations. Without this mandate, change requests to replicate current processes had all been viewed as 'critical'.
• There had been a lack of representation of VicRoads Information Management and Technology division within the governance structure, leading to a misalignment of the project to broader VicRoads strategies, limited expert scrutiny of contractors and limited consideration of the operational impacts of decisions and change requests.
• There had been significant shortfalls in the project's management since remobilisation, particularly around system design validation, vendor management and project financial oversight.
• There was a core skills gap with respect to vendor management and independent quality assurance.
• There had been a lack of focus on strategic risk management. Sufficiently detailed information on risks, finances and schedule had not been provided to the project board, including information about the risks of a single stage deployment strategy.

The review included recommendations that:

• VicRoads analyse staged deployment options because a single phase deployment is too risky
• VicRoads improve the clarity and transparency of the information provided to the project board about costs, risks, mitigations and vendor management issues
• VicRoads engage an appropriately experienced project director and source an enterprise architect to advise on the deployment options
• VicRoads, with the project board, revisit the project's governance regarding regular independent quality assurance reporting and the integration of VicRoads ICT functions and chief information officer into the project governance.

The September 2013 review contrasts with the August 2011 review, because the earlier review did not highlight issues with the team's capability, the design process or the choice of a single phase implementation.

3.5.4 Lessons from the RandL experience

The particular lessons from this project are not new but are worth reaffirming.

Major ICT projects need to be approached as business transformation projects, not as siloed technology upgrades.

This involves an end-to-end, holistic consideration of how to combine upgraded technology, revised legislation and policies and modified business processes to best meet the intended outcomes. The process must involve—from start to finish—all those connected to the changes, including partner agencies, employees and customers.

Doing this effectively requires people who can bring this set of skills to manage, oversee and independently quality-assure these types of projects. This should affect the appointment of project directors, the composition of project boards and the appointment of consultants to do independent reviews.

Ideally, each line of management and oversight should include people with a track record of delivering similar projects.

3.5.5 Current status of RandL

By the end of 2013 VicRoads had:

• strengthened the project team and board by appointing appropriately experienced personnel
• commissioned reviews to re-evaluate the project costs and risks and underpin these estimates for a more holistic approach to business transformation
• worked up phased development options to mitigate the risks of a single phase development and to secure major beneficial changes as soon as possible.

The DTPLI verified the options and estimated costs and time lines before submitting this material to government.

The project is currently paused for one year from February 2014 to allow VicRoads to update the business case to further explore the options for delivering the project and working with DTPLI to investigate the potential for legislative reform to streamline service delivery.

To the end of February 2014, a total of $102.4 million had been spent on RandL.

Recommendations

That the Department of Treasury and Finance:

1. provides greater assurance that High Value High Risk reviews comprehensively test compliance with its Investment Lifecycle and High Value High Risk Guidelines in areas critical for project deliverability
2. checks that for complex, risky projects—particularly those involving information and communications technology transformations—the specialist skills needed to successfully manage, oversee and quality assure these projects have been assessed and acquired.

RESPONSE provided by the Secretary, Department of Treasury and Finance

 

RESPONSE provided by the Secretary, Department of Transport, Planning and Local Infrastructure