Biotechnology in Victoria: the Public Sector's Investment

Conclusions

In its role as the public sector agency responsible for administering the bulk of public investment in biotechnology, DBI has delivered a range of world class facilities and technology platforms for the biotechnology and broader life sciences sector.

The eight biotechnology projects examined have directly supported life sciences researchers and practitioners and have the potential to:

• stimulate scientific research and innovation
• assist pharmaceutical and medical discoveries
• attract more private investment
• develop highly skilled people
• extend existing scientific knowledge.

DBI has publicly reported that of the 16 high-level strategic goals and targets that the government set for the biotechnology sector over the last decade, ten have been met, and adequate progress has been made for targets that have not been fully met.

Notwithstanding the claimed achievement of high-level sector goals and targets, there are shortcomings in DBI's approach to investment targeting, and gaps in its benefits measurement and monitoring approaches. In particular, DBI does not use recognised or accepted methodologies to systematically determine portfolio risk and inform investment allocations.

DBI's administrative focus for the sector is also mainly on grant and contract compliance. Though useful in its own right, this focus does not produce performance data about whether the strategic aims of the public sector's investment in the biotechnology sector are being delivered. Instead, the primary focus of DBI's performance measurement is monitoring the impact of short-term stimulatory activity in the sector, without evident attention to the delivery of longer-term and strategic benefits.

As a result, after more than ten years of active investment and 'market intervention' in the biotechnology sector, DBI is not yet able to objectively demonstrate the development of a cause and effect relationship between its investments in the biotechnology sector and the results achieved in that sector. It is, however, reasonable to attribute at least part of the achievement of BSDP targets to investments made by the public sector.

Against a background of typically long lead times for the full benefits from such investments to become evident, it is now timely for DBI to address these issues so that progressive assurance can be provided that its portfolio of investments in biotechnology is both targeted to where it is needed and is achieving its intended purpose.

Further, because there are no measures or targets that could be used as a benchmark against other leading international biotechnology locations, it is not possible to conclude on the aspirational goal that Victoria is, or will be, 'one of the world's top five biotechnology locations'.

Findings

Based on our review of eight representative investments in the biotechnology sector, there are consistent systemic issues in relation to measuring benefits. These issues can be summarised as follows:

• monitoring and evaluation of benefits is limited
• benefits from investments are not always sustainable
• DBI can not systematically demonstrate whether benefits have been realised.

Monitoring and evaluation of benefits

Because DBI's benefits measurement and management framework is not comprehensive or robust, DBI is not able to objectively demonstrate whether benefits have been achieved from public sector investments in biotechnology.

Sustainability of realised benefits

Notwithstanding the achievements recorded, DBI's preferred funding models do not explicitly require the identification and pursuit of sustainable future benefits.

Achievement of benefits

In terms of benefits measurement, we observed:

• reporting is at a very high level and often based on grantee data that has not been validated
• a baseline of a current or 'before' state is not set so that the 'after' state can be accurately measured

• data collected by DBI has been focused on sector-level metrics that do not measure the efficacy of individual grants
• retrospective studies commissioned by DBI have methodological weaknesses in that they use general economic 'activity' as a proxy for specific public sector investment 'effectiveness'.

In consequence, reliable conclusions cannot be drawn regarding the extent to which intended objectives are being met in a timely and cost-effective manner.

Recommendations

1. The Department of Business and Innovation should develop an overarching biotechnology strategy to:

   • guide whole-of-government activities and investments in the sector
   • link the various funding programs to optimise investments' contribution to the agreed goals and targets for the sector
   • set cause and effect measurement criteria that will allow for progressively more objective assessment of the achievement of goals and strategies.

2. The Department of Business and Innovation should adapt an investment portfolio allocation model to determine whether there is alignment of investments in the biotechnology sector with the Department of Business and Innovation's desired risk and return profile.

3. The Department of Business and Innovation should further develop its grant management processes for biotechnology investments by:

   • setting, for each of the investments that it funds, performance targets that relate to strategic objectives
   • validating information provided by grant recipients
   • embedding benefits planning in the funding approval process.

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report was provided to the Australian Synchrotron Company Limited, Australian Synchrotron Holding Company Limited, Department of Business and Innovation, Melbourne Health, and the University of Melbourne with a request for submissions or comments.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix C.

Biotechnology comprises the technologies, techniques and tools that use living organisms and their elements to make products and develop new industrial processes. It is a 'life science'-the study of the structure, behaviour and life processes of organisms such as plants, animals and human beings.

Biotechnology can be used to:

• develop new diagnostics, vaccines, and treatments to improve health
• lift agricultural productivity
• manage the environment through pollution and pest control and the use of organic materials to generate energy
• assist with industrial innovations.

1.1.1 The value of biotechnology

In addition to benefits from its social and environmental applications, biotechnology has the potential to generate great economic value. There are several steps to gaining these benefits-often described as a 'value chain'-shown by Figure 1A.

Figure 1A
The biotechnology 'value chain'

1.2 Biotechnology policy and strategy

1.2.1 Encouraging innovation

Since 1999, DBI has funded $3.442 billion on programs and investments in science, technology and innovation (STI) to boost Victoria's science base. Many of these activities overlap, so DBI does not try to determine exactly how much it has spent specifically on biotechnology.

DBI estimates that, since 1999, of the $3.442 billion spent on STI initiatives, $722 million (21 per cent) has been directly spent on biotechnology, and another $610 million (18 per cent) has been indirectly spent in areas that have some relationship to biotechnology as shown in Figure 1B.

Figure 1B
Victoria's $3.442 billion innovation spending 1999-2009

Source: The Department of Business and Innovation.

Since 1999 there have been various innovation policies, strategies and activities, as shown in Figure 1C.

Figure 1C Biotechnology and innovation strategies

Source: Victorian Auditor-General's Office.

1.2.2 The biotechnology strategic development plans

The 2001 Biotechnology Strategic Development Plan for Victoria (BSDP) envisioned that 'by 2010 Victoria is recognised as one of the world's top five biotechnology locations for the vibrancy of its industry and quality of research'.

It aimed to strengthen the state's research and development base by investing significantly in building infrastructure and scientific capability. Two more BSDPs were published in 2004 and 2007.

The 2004 BSDP focused on 'co-investing' in facilities that would help the industry develop products to the pre-market stage. It aimed to attract investment capital, facilitate international partnerships and build capability in pilot-scale manufacturing and early product testing.

The 2007 BSDP proposed a framework to build a mature and commercially sustainable sector by 2010 by using a partnership model between government, industry and research organisations to attract philanthropic and Commonwealth co-funding.

The most recent, the Victorian Biotechnology Action Plan 2011 was released just before the state election in November 2010. However, at the time of this audit the new government had yet to release any specific biotechnology policy or confirm whether it would support the action plan.

DBI is responsible for producing the BSDPs and the strategic coordination of the state's biotechnology. Between 2001 and 2007, the government allocated $49.6 million to DBI to develop and implement the BSDPs.

2.1 Introduction

In 2001, the first Biotechnology Strategic Development Plan for Victoria (BSDP) was launched, with a vision that 'by 2010 Victoria is recognised as one of the world's top five biotechnology locations for the vibrancy of its industry and quality of research'.

The former Department of Innovation, Industry and Regional Development, now known as the Department of Business and Innovation (DBI), developed three BSDPs. It received $49.6 million between 2001 and 2007 to coordinate this process as well as to monitor and report on whole-of-government biotechnology activities. This monitoring role applies to about $722 million in direct public sector investment in biotechnology.

The 2001 BSDP aimed to strengthen the biotechnology research and development base by investing in scientific infrastructure and capability. Subsequent BSDPs were developed in 2004 and 2007. This audit does not review the most recent plan, published in late 2010. DBI reports against BSDPs to government, the biotechnology sector and the public on whether it has met the goals and targets.

The specific targets and goals in these BSDPs have changed over time, however, the core aspirational goal for Victoria to be one of the world's top five biotechnology locations has been constant.

This Part of the report examines whether the BSDPs have been effective in:

• guiding public sector investment in biotechnology projects
• developing successful biotechnology-based industries in Victoria.

2.2 Conclusions

There is no overarching biotechnology strategy to guide all publicly funded biotechnology activities and investments. Although BSDPs have high-level goals, such as Victoria's place in biotechnology in the international context, they are a small part of overall biotechnology expenditure. Therefore, their ability to impact on the performance of overall sector investment is likely to be limited.

The various biotechnology projects, initiatives and funded activities are not coordinated or reviewed for potential synergies. This increases the likelihood of inappropriate prioritisation and funding, and could lead to duplication, inefficiency and waste. Such outcomes would reduce the state's ability to gain optimum value-for-money from its investments.

DBI has reported that 10 of the 16 high-level strategic goals and targets set in the three BSDPs for biotechnology over the last decade have been met. However, DBI cannot demonstrate that the funded programs and activities achieved those results. This is because the measures adopted by DBI do not clearly demonstrate how the $722 million contributes to meeting the strategic goals and targets for biotechnology.

Because there are no measures or targets for benchmarks, it is not possible to determine whether Victoria is on the way to, or perhaps has already achieved, its goal of being one of the world's top five biotechnology locations.

2.3 Coordinating public sector investments

The five main policies and programs that fund biotechnology are:

• Science Technology and Innovation Initiative: First and Second Generation-since 1999 $638 million of funding through competitive and non-competitive rounds to 136 investments, most of it for infrastructure.
• Healthy Futures-building infrastructure and skills in medical research. It was launched in 2006 with $230 million to support growth in medical research and the life sciences.
• Operational Infrastructure Support Program-annual performance-based funding to independent medical research institutes to support indirect costs associated with carrying out research which are not specifically funded by government research grants.
• Innovation: Victoria's Future-commits a further $300 million to innovation initiatives including: $145 million for Victoria's Science Agenda, $50 million for the Victorian Life Sciences Computation Initiative and $20 million for 'Biotechnology Bridges'.
• Biotechnology Strategy-small grants, usually a few hundred thousand dollars, to a range of activities, the majority of which are related to marketing.

These policies and programs account for over $1.2 billion of the state's investment in biotechnology and related projects.

Figure 2A outlines the share of the programs spent directly on biotechnology projects and initiatives.

Most of the funding in Figure 2A has been for infrastructure, skills and basic research and development, with BSDP funded programs contributing only 7 per cent of the total.

Source: Victorian Auditor-General's Office.

Criteria for allocating funding under the programs do not specifically assess whether the projects align with the strategic goals and targets in the BSDPs.

For example, they do not require an assessment of whether the project being considered will help Victoria achieve the goal of being one of the world's top five biotechnology locations.

2.4 Biotechnology strategic development plans

3.1 Introduction

Biotechnology is an inherently risky venture due to the high cost of commercialisation of new products and the high chance of once-promising discoveries being abandoned after further testing.

Given that there are usually many projects competing for limited public funding, if the Department of Business and Innovation (DBI) wants to be a strategic investor it needs to reliably determine which projects are more likely to realise the greatest benefits to the state.

To maximise the impact of public funds, project investments should therefore have a high likelihood of generating benefits. Thus, it is critical that inherent risks and expected benefits from proposed biotechnology projects are well understood by DBI. Measuring and reporting benefits from expenditure are key components in the chain of public sector accountability. These expectations are in line with global better practice for innovation programs, including the OECD's latest guidance, which advocates a mixture of high level performance measures, linked to attributable data from individual investments.

3.1.1 Scope of the audit and assessment criteria

This Part of the report examines DBI's approach to managing biotechnology investments. It relies upon detailed case study analysis to assess how DBI:

• makes funding decisions and manages risk
• implements and monitors the progress of funded projects
• measures benefits from biotechnology investments.

The case studies shown in Figure 3A were selected to represent the broad range of public sector funded activities and investments in the biotechnology sector. They were all funded in the period 2000-06 under various programs and were expected to deliver a range of outputs including physical infrastructure, technology platforms and research capability.

They account for 11 per cent of the public sector money identified as being spent on biotechnology since 2000.

To assess whether benefits from these investments were realised we examined whether:

• expected benefits were clearly articulated, realistic and measurable
• actual benefits were achieved
• benefit management plans were documented and included measures to monitor, analyse and report progress
• post-implementation reviews were conducted to assess if the business cases were realistic and to what extent the expected benefits have been achieved.

3.2 Conclusions

Since 1999, DBI funding has been effective in assisting the delivery of a range of world class facilities and technology platforms for the biotechnology and broader life sciences sector. This was confirmed through our examination of a range of investments in the sector.

However this examination also identified consistent and systemic issues in relation to planning for and measuring of benefits.

These issues can be summarised as follows:

• Portfolio risk analysis is not used to inform investment allocations. While selected investments considered in this audit were all assessed as high return, more than 82 per cent were also assessed as high risk.
• DBI's monitoring and evaluation of benefits is not sufficient.
• Benefits from investments are not always sustainable.
• DBI doesn't consistently know whether benefits have been achieved.

3.2.1 Portfolio risk analysis to inform investment choices

DBI does not use recognised and accepted methodologies to determine portfolio risk and inform investment allocations. This could lead to ad hoc investment planning and decision-making processes and limited strategic option analysis for potential investments.

3.2.2 Monitoring and evaluation of benefits

Because DBI's benefits measurement and management framework is not comprehensive or robust, DBI is not able to consistently demonstrate whether benefits have been achieved from public sector investments in biotechnology.

In particular the following weaknesses with benefits planning were evident:

• consistent criteria are not used for the selection of public investments due to the mix of funding types
• benefits planning is not consistently embedded into grants allocation or decision-making processes
• benefits are often poorly specified, with many difficult to measure or unrealistic benefits specified, and with some benefits set at too high a level and therefore unable to be convincingly attributed to the grant or project.

3.2.3 Sustainability of realised benefits

Notwithstanding the benefits which DBI report they have achieved, DBI's preferred funding models do not encourage identification and pursuit of sustainable future benefits.

This is because:

• most grants are one-off or too short-term in nature for benefits to have emerged during the grant period
• grantees are not required to develop an effective third party access or ownership strategy for department-funded scientific infrastructure after the grant term expires
• some of the funding models that DBI actively encourages are based on an overly optimistic assumption that the grantee will be able to generate sustainable fee-for-service revenue after the grant expires.

3.2.4 Achievement of benefits

In terms of benefits measurement, the following weaknesses were evident:

• reporting is at a very high level and is often based on grantee data that has not been validated
• a baseline of a current or 'before' state is not set, so that the 'after' state can be accurately measured
• data collected by DBI has been focused on sector-level metrics that do not measure the efficacy of individual grants
• retrospective studies commissioned by DBI have methodological weaknesses in that they use general economic activity as a proxy for specific public sector investment effectiveness.

3.3 Funding decisions and investment risks

The biotechnology industry sector is diverse and complex. This diversity and complexity increases the potential for public money to be wasted if the relationship between a proposed project's inherent risks, and the expected benefits, aren't appropriately defined and adequately considered when funding decisions are made.

3.3.1 Portfolio allocation and risk models

Portfolio allocation models are a common tool used by investors to understand the weighting of aspects of an investment portfolio. Qualitative and quantitative factors are used to assess the risk and return of proposed investments up front. This allows an investor to assess the investment options in the context of their risk appetite and likely return on their investment.

Portfolio management goals can be represented graphically, with the risk profile and potential value of investments assessed and plotted on a portfolio allocation graph. Public sector investors could adapt this model to suit their investment context, as shown in Figure 3B.

Figure 3B
A generic portfolio allocation model for public sector investors

Source: Victorian Auditor-General's Office.

Biotechnology projects are inherently risky and costly and can take some time to show a return on investment, if at all. They may not achieve expected benefits despite a high level of investment. Consequently, private sector investors may be unwilling to invest in such projects. Economists describe this problem as 'market failure'.

Policies and programs administered by DBI can provide stimulus to the biotechnology industry market by providing grant funding, which eliminates the need for project proponents to raise costly private sector capital. Economists broadly describe this activity as 'market intervention'.

Unlike a profit-seeking investor, decision-makers in the public sector are not usually driven by a primary desire to receive a financial return on outlays. For example, in the case of biotechnology, development of the sector, and generation of 'public value' may be the key desired returns on public sector investment.

Notwithstanding that DBI may be less focused on making profits or monetary returns, it has a general duty to invest wisely and prudently so that value-for-money is optimised for the Victorian community. To do this, it needs to be particularly conscious and careful in its investment choices, as applicants for its grant funding are likely, to be largely from the high risk and high return quadrant.

This is because the logic of profit seeking private sector investors would lead them to focus the bulk of their investments in the low risk and high return quadrant, with fewer in the high risk and high return quadrant. The remaining two quadrants, which offer only low returns, would typically be avoided by investors from both sectors.

In order to illustrate how the portfolio allocation model could be used by DBI, Figure 3C shows our assessment of the risk-return ratio and relative cost of the eight representative investments we examined for this audit.

The size of each 'bubble' is proportional to the amount of investment, and the position of the bubble in a particular quadrant indicates our assessed level of risk and return for that investment.

Figure 3C
Risk and return ratio of selected biotechnology investments

Source: Victorian Auditor-General's Office.

Figure 3C shows that the bulk of the value of the selected investments, $233.1 million (82 per cent), is in the high risk and high return quadrant. Although this allocation pattern shows a potential over-weighting of high risk investments, it also shows that all of the eight investments are expected to generate a high return from public investment.

Figures 3B and 3C highlight the potential usefulness of this approach for DBI, as the key public investor in the biotechnology sector, as it clearly maps the:

• relative cost of each investment in the portfolio
• assessed risk of the investment
• expected return from the investment.

3.3.2 Risk analysis for investment choices

It was not evident that an investment portfolio allocation model, or a reasonable alternative, is in place within DBI to inform its funding decisions.

DBI should consider adapting such a model to help structure its understanding of its historical and future investment portfolio. DBI should also use the model to determine whether there is alignment of investments in the biotechnology sector with DBI's desired risk and return profile.

3.4 Implementing and monitoring investments

Over the past decade, DBI has used both recurrent and capital asset funding of $722 million to support the biotechnology sector. The recurrent funding has mainly been in the form of grants which are managed according to DBI's internal financial and administrative processes.

Due to the mix of funding types used by DBI to support the biotechnology sector, there is a blurring of the concept of an 'investment' into the sector, as most of the funds have been disbursed via 'grants'.

However, some of DBI's grants to the biotechnology sector are more akin to an asset investment, due to the expectation that certain tangible assets will be acquired by the grantee, such as a specialised laboratory or a scientific instrument.

Because the state does not necessarily own the asset at the end of the grant period, DBI's grant funding to the biotechnology sector does not meet the technical definition of asset funding. Consequently this does not attract the typical central agency scrutiny or mandatory processes that apply to asset funding, including benefits measurement.

This means that DBI does not consistently use investment management tools. It should maximise the use of tools such as the Department of Treasury and Finance's (DTF) Investment Lifecycle Guidance , to assist with the development of clear investment logic maps, business cases and benefits plans for candidate investments.

Figure 3D
Business cases for the Science Technology Innovation grants

Source: Victorian Auditor-General's Office.

The DTF investment management approach emphasises the importance of a well-constructed investment case and shifts the focus from monitoring project metrics to strategically managing the long-term realisation of benefits.

Much of DBI's investment (or grant giving) activity is characterised by being one-off and short term, and could be described as 'donor' rather than 'investor' behaviour. This is because funds are disbursed to generate activity in the sector, without necessarily proactively managing a return (be it economic, social, or environmental).

Although good project and contract management processes are important, DBI needs to display more investment management behaviours to assure the government that its grants are delivering expected returns for the public's investment, and are thus value-for-money.

3.4.1 Contract and grants management

Milestones contained within funding agreements are used by DBI to monitor project performance and trigger stage payments. These milestones do not always correlate with project KPIs and are more an indicator of project processes, rather than the likely success of benefits delivery.

Milestones are project management metrics and represent a project view rather than a strategic investment view. However, we also noted examples where payments were made, even when grantees had not achieved expected milestones.

Figure 3E
Monitoring performance of the BioGrid investment

Source: Victorian Auditor-General's Office.

3.4.2 Measurement and reporting on effectiveness

DBI does not consistently analyse the outcomes and benefits of individual projects against their original business cases. This is a missed opportunity to capture lessons learned and to inform future investment decisions.

STI grant recipients complete a self-assessment survey on an annual basis after the grant period has ended. These surveys are generic and DBI does not have a formal process to independently validate the information provided nor does it systematically collate responses into a 'lessons learned' document for grant recipients.

Consultants commissioned by DBI have also recognised this issue, with their reports including a disclaimer to the effect that the information they have relied upon has not been validated.

DBI needs to improve its grant management processes by developing, for each of the biotechnology projects that it funds, performance targets that relate to relevant program objectives, as well as validating information from grant recipients.

DBI has taken some actions to improve its grant management process under the more recent Victoria's Science Agenda (VSA) program. This includes amending grant guidelines, redesigning the outcome monitoring survey, and revising the grant database and grant funding agreement template.

3.4.3 Conclusion on effectiveness of processes

DBI's administrative structures are mainly focused on grant and contract compliance processes and do not provide useful performance data about whether the strategic (policy) intent of government is being delivered.

The focus of DBI's grant giving processes appears to be on short-term stimulatory activity in the sector, rather than on delivery of long-term strategic benefits.

3.5 Case studies: benefits realisation

3.5.1 Summary of findings

In summary, the benefits realisation results for the investments we reviewed have been mixed. Figure 3F provides our assessment against the assessment criteria for all the reviewed investments.

3.5.2 Detailed findings from our review

Appendices A and B provide the detail to support each assessment.

Figure 3F
Summary of findings for benefits realisation of selected biotechnology investments

Source: Victorian Auditor-General's Office.

3.6 Systemic issues from case study observations

Based on our review of eight representative investments in the biotechnology sector, there are a number of consistent thematic issues in relation to monitoring and evaluation and benefits realisation.

These issues can be summarised as follows:

• benefits realisation measurement is underdeveloped
• benefits from investments are not always sustainable
• DBI cannot systematically demonstrate whether benefits have been realised.

3.6.1 Benefits realisation measurement

Generally, DBI's measurement of activity in the biotechnology sector focuses on private investment activity. This is not directly attributable to DBI's own performance in the sector and is useful primarily as a way to understand the 'state of health' of biotechnology in Victoria.

One of these tools, the Victorian Life Sciences 2009 Sector Report, reports on:

• total market capitalisation of the Victorian listed life sciences sector
• number of life science companies in Victoria
• number of drug candidates in submission phase involving Victorian companies
• Victorian life science company listings
• sales of products by Victorian life science companies
• profit generated by the listed Victorian life science companies
• estimated employment created by the Victorian life science companies
• funding raised by listed Victorian life science companies
• employment and exports created by international life science companies in Victoria.

DBI has also developed the Outcome Monitoring Tool - STI monitoring survey (OMT) which uses standard questions to collect information on the achievements of STI projects in relation to the five core intended outcomes of the program.

The monitoring survey captures only limited information about the measured projects' performance and is primarily intended to provide information to reflect high level program performance in a relatively cost-efficient way.

The OMT does not distinguish outcomes from benefits, and confuses some outcomes with benefits. Since the OMT is a generic survey, the questions are not always relevant to a particular project, and therefore do not always link back to the expected benefits articulated in the project's business case. There may also be additional benefits identified in a project's business case that are not captured in the survey.

No accompanying documents substantiate the data reported in the monitoring survey. In theory, DBI requires its client managers to validate the data and to follow up with grant recipients if any discrepancies are noted. However, DBI could not provide any documentation of this validation process in relation to the selected STI projects reviewed in this audit. This means the accuracy and quality of the information collected through the monitoring survey is not assured.

None of the various reports or surveys we reviewed provide DBI with adequate information on progress made towards achieving the claimed benefits. As a result, DBI cannot be assured that project benefits either have been or will be delivered as proposed.

3.6.2 Benefits are not always sustainable

Since 1999, DBI funding has been effective in assisting the delivery of a range of world class facilities and technology platforms for the biotechnology and broader life sciences sector. This was confirmed through our examination of a range of investments in the sector.

These facilities have directly supported life sciences researchers and practitioners and have the potential to:

• stimulate scientific research and innovation
• assist pharmaceutical and medical discoveries
• attract more private investment
• develop highly skilled people
• extend existing scientific knowledge.

Many of the investments that DBI makes in the biotechnology sector are in the form of grants, which by their nature are a limited contractual relationship between DBI and the entity it funds.

DBI focuses extensive administrative effort on setting up and administering 'arms length' panels and advisors to help choose projects for funding. It also deploys numerous procedures and systems to manage contractual terms and milestones of funding agreements, and to acquit grant expenditure.

However, very little of DBI's up-front administrative effort is focused on making sure that the initiative that was funded-and any related benefits arising-are able to be sustained after the grant funding ceases.

This means that:

• most grants are one-off or too short-term in nature for benefits to emerge
• grantees are not required to develop an effective third-party access or ownership strategy for DBI-funded scientific infrastructure after the grant term expires
• some of the governance models that DBI actively encourages are based on an overly optimistic assumption that the grantee will be able to generate sustainable fee-for-service revenue after the grant expires.

The outcomes of DBI's biotechnology investments could be improved by testing some of the underlying assumptions used in grant decisions against known actual performance of similar investments over the past 10 years and applying these lessons to future investments.

3.6.3 Benefits realisation

Expected benefits (if defined) for DBI's investments in the biotechnology sector are usually documented in business plans written as a requirement of funding agreements. Our review of the selected investments showed that many purported benefits are more correctly described as outputs or activities.

Improved benefits planning and measurement by DBI would allow it to:

• embed benefits planning into grants allocation decision-making processes
• better specify benefits and map inter-relationships with other investments
• specify benefits more accurately, rather than being too high level, not measurable, or not realistic.

In terms of benefits measurement, we observed the following weaknesses:

• reporting is at a very high level and is based on data that has not been validated
• a baseline of a current or 'before' state is not set which means that the 'after' state cannot be accurately measured
• the data that is collected by DBI is focused on sector-level metrics which can't directly measure the efficacy of the grants
• retrospective studies commissioned by DBI have methodological weaknesses in that they use economic activity as a proxy for investment effectiveness.

Recommendations

2. The Department of Business and Innovation should adapt an investment portfolio allocation model to determine whether there is alignment of investments in the biotechnology sector with the Department of Business and Innovation's desired risk and return profile.

3. The Department of Business and Innovation should further develop its grant management processes for biotechnology investments by:

   • setting, for each of the investments that it funds, performance targets that relate to strategic objectives
   • validating information provided by grant recipients
   • embedding benefits planning in the funding approval process.

This Appendix shows the detailed results of our reviews of the extent to which benefits have been realised for a number of key public sector investments in the biotechnology sector.

We examined:

• Australian Synchrotron-state funding of $207.2 million
• Bio21 Molecular Science and Biotechnology Institute-asset funding and Crown land contribution amounting to $30million.

We also examined Department of Business and Innovation (DBI) documents and processes related to the following investments:

• Neurosciences Victoria -grant funding of $13.34 million
• BioGrid (formerly Molecular Medicine Informatics Model)-grant funding of $12.6million
• Victorian Tissue Bank Initiative/Victorian Cancer Biobank -grant funding of $7million
• Australian Tissue Engineering Centre -grant funding of $5.2 million
• Victorian Microarray Technology Consortium -grant funding of $4.42 million
• Centre for Drug Candidate Optimisation -grant funding of $4 million.

To assess whether benefits from the above investments have been realised we examined whether:

• expected benefits were clearly articulated, realistic and measurable
• actual benefits were achieved
• benefit management plans were documented and included measures to monitor, analyse and report progress
• post-implementation reviews were conducted to assess if the business case was realistic and to what extent the expected benefits have been achieved.

The review relied upon the objectives set by government or grant funding agreements and are not an assessment of the overall performance of the recipient entity.

The Australian Synchrotron

Conclusion

The Australian Synchrotron has achieved benefits in line with the original expectations of the government when it agreed to invest in the facility.

However, DBI and the Australian Synchrotron should make an attempt to better articulate and measure these benefits, so that future funding after 2012 can be linked to more measurable and attributable outcomes.

Overview of the investment

A synchrotron is a powerful source of light that scientists can use to assess the structure of materials at a molecular level.

A 1999 study conducted for the government by a strategic economic centre at a Victorian university noted that 'The synchrotron will act as a catalyst for development: enabling technology transfer by bringing together researchers from industry and academia. It will be the focal point for the development of a high-technology industry cluster.'

In June 2001 the then government announced that the state would contribute up to $100 million towards the cost of building a synchrotron.

A further funding submission in January 2003 set out an enhanced design and additional funding requirement of up to $57.2 million. Construction commenced in September 2003 for a total supported cost of $157.2 million for the building and machine (excluding beam lines).

An additional $50 million was subsequently provided by the state to partially cover costs for the first five years of operation.

The Australian Synchrotron was officially opened in July 2007-on time and on budget-and has been operating as a functioning synchrotron since that date. Since its opening, the Australian Synchrotron has established itself as a world-class facility and one that should deliver long-term scientific, economic and social benefits to both Victoria and Australia.

The Australian Synchrotron has developed an investment case to seek future funding from the Australian and Victorian Governments beyond the current funding agreement which concludes at the end of the 2011-12 financial year. This is currently under consideration by the Victorian Government.

Expected benefits

While the estimated economic and social benefits contained in early studies were not subject to independent scrutiny, this would not have necessarily affected the underlying primary case for the investment as identified to the government.

The expectations contained in the 2003 funding submission to government provide a set of suitable key performance indicators (KPIs) that could be used by DBI or a third party for benefits measurement. Figure A1 assesses these KPIs against the Department of Treasury and Finance's (DTF) better practice guidance.

Source: Victorian Auditor-General's Office analysis of Department of Business and Innovation documents.

Identified benefits

A Gate 6 - Benefits Evaluation Gateway Review was conducted by DTF in September 2009. Its report stated that the Australian Synchrotron project objectives were:

• to be a national icon of leading-edge science, of pivotal importance within the research structure of Australia
• to boost economic development by increasing opportunities for innovation and commercialisation, and encouraging the clustering of research-based industries
• to be designed and maintained to satisfy at least 95 per cent of Australian synchrotron science requirements
• to be recognised within the international community of users as an advanced and well-managed facility that meets or exceeds national and international standards
• to be accessible and user-friendly to a wide array of users around Australia and the region, and to facilitate collaboration and multi-disciplinary research partnerships
• to be operated in a commercially responsible manner to ensure the facility can continue to grow and develop to meet the changing needs of the Australian research community through reinvestment.

Although these project objectives are not equivalent to specific benefits, they capture the general project benefits at a macro level.

The Gateway Review Report arrived at the same conclusion by noting that 'While the Review considered the initial objectives (as outlined in this report) to have been broadly met, there are no specific metrics attached to these objectives that would have allowed for a definitive assessment of the benefits achieved to date.'

Plans for benefits management

No benefits management plans were developed by DBI or the Australian Synchrotron to monitor, analyse and report the achievement of the project's expected scientific, economic and social benefits.

The benefits described in the 2003 funding submission were developed many years before the Department of Treasury published the Investment Management Standard-Benefit Definition. As such, there was no specific policy requirement at the time for DBI to develop a benefits management plan or a clearly articulated set of benefits or supporting KPIs.

However, the lack of a current Australian Synchrotron benefits management plan does not preclude DBI from determining at some future point the actual benefits delivered by the state's investment in the project.

An interior view of the Australian Synchrotron facility. Photo provided courtesy of the Department of Business and Innovation.

Post-implementation reviews

The Gateway Review Report made a range of favourable findings about the Australian Synchrotron investment including:

• 'The Australian Synchrotron [...] is operating at world-class standards for availability, beam line stability, safety and other key matters...'
• 'There was consistent feedback from national and international experts that the Australian Synchrotron represented a world-class outcome achieved at a 'bargain basement price', i.e., has provided good value for money.'
• 'The Review Team finds that the Australian Synchrotron was well conceived, effectively and efficiently delivered, has been successfully commissioned and is operating as a world-class facility.'

Based on our own observations and fieldwork, we agree with the observations of the Gateway Review team.

In terms of the sustainability of benefits, the Gateway Review Report noted a number of issues that could affect the ongoing sustainability of benefits. In particular, the report noted tensions between:

• staff of the Australian Synchrotron and its Board, arising from a culture clash between science and commercial governance
• the Australian Synchrotron and its members
• various governments with an interest in the facility.

However since these issues came to light, a new senior management team and new members of the Scientific Advisory Committee have been appointed. The Australian Synchrotron is also operating at full capacity.

Bio21 Molecular Science and Biotechnology Institute(Bio21 Institute)

Conclusion

The state's investment in the Bio21 Institute, through the University of Melbourne, has assisted the university in delivering and developing a high-quality facility that currently houses a number of collaborative research groups, major technology platforms and industry tenants.

Given the substantial investment of state funds in the Bio21 Institute, DBI should attempt to determine what benefits the Bio21 Institute has delivered.

Overview of the investment

The origin of the Bio21 Institute lies in a 1999 joint initiative between the University of Melbourne, the Walter and Eliza Hall Institute of Medical Research and the Royal Melbourne Hospital.

At that time, these three organisations secured offers of funding from the University of Melbourne ($50 million) and an anonymous donor ($30 million) whose offer was conditional on additional funding being obtained from the state government. The state government provided $50 million to the Bio21 Project, of which $30 million was invested directly in the Bio21 Institute facility (including a $15 million transfer in kind of Crown land).

The Bio21 Institute is an institute that forms one entity of the Bio21 Cluster and was a recipient of funds from the Bio21 Project for facility and capital investments.

The Bio21 Cluster is a loose grouping of biotechnology entities consisting of organisations and entities from academic institutions, private institutes and private sector companies.

The Bio21 Project was the vehicle used by the state and Bio21 Australia (originally Bio21 Corporate) to create the original Bio21 Cluster and manage approved Bio21-related investments in capital facilities. The majority of the state's investment in the Bio21 Project was allocated to the Bio21 Institute.

Given the absence of benefits directly attributable to the Bio21 Institute in the early stages of the project, we assessed the economic and social benefits of the broader Bio21 Project so that the Bio21 Institute's contribution to project benefits could be determined.

The benefits for the Bio21 Project were first articulated publicly as state government policy in the Biotechnology Strategic Development Plan (BSDP) published in 2001.

The benefits included:

• creating up to 100 new biotechnology companies over the next ten years
• attracting new investment directly into the industry at a rate of at least $30 million per year
• generating thousands of new jobs each year, both directly at the Bio21 facilities and through companies feeding off its activities
• creating millions of dollars in high value added export income.

The Bio21 Institute building was originally intended to accommodate over 1 000 people in order to meet its twin objectives of collocation and commercialisation. However, the building plans were scaled back and around half of this number currently work in the Institute. The Bio21 Institute building opened in 2005.

The internal atrium of the Bio21 Institute building. Photo provided courtesy of the Department of Business and Innovation.

Expected benefits

The economic and social benefits contained in the BSDP 2001 provide a set of suitable measures which could be used to measure the benefits of the Bio21 Project. These benefits cannot be achieved by the Bio21 Institute in isolation.

Source: Victorian Auditor-General's Office.

The Bio21 Project Business Plan does not contain any clearly articulated benefits statements.

It does, however, contain 13 objectives, some of which could be used to develop benefits statements and KPIs. The Bio21 Institute Business Plan 2005-2014 does contain a well developed set of KPIs and these have been used by Bio21 Institute to report performance internally within the University of Melbourne.

A range of these KPIs could be used to develop an agreed set of benefits applicable to the Bio21 Institute.

Actual benefits

The Bio21 Institute has delivered a range of positive outcomes to Victoria. These include:

• world-class infrastructure that houses strategically important biotechnology platform technologies that can be readily accessed by academia, government research organisations and industry within the Parkville biotechnology precinct
• a new and innovative organisational research model based around multi-disciplinary research with a focus on innovative science discoveries in the areas of structural biology, chemical biology and nano-biotechnology
• the co-location of academics, a cooperative research centre and industry promotes multi-disciplinary research and collaborative activity.

The achievement of these outcomes and the resulting benefits has been made possible through the establishment of the Bio21 Institute.

Plans for benefits management

Commercialisation was one of the original key objectives of the Bio21 Project, however, the underlying financial analysis of commercial revenues was unrealistic and overoptimistic at the time. In 2002, the department agreed that the focus of the Bio21 Project should be on collaborative research.

However, DBI did not fully assess the implications of agreeing to change the focus from commercialisation to collaborative research, particularly given that commercialisation was the fundamental driving factor behind the state's decision to fund the Bio21 Project.

As a result, DBI did not provide comprehensive advice to the Minister prior to the approval of the Bio21 Project Business Plan. DBI could have exercised the state's rights in the funding agreement to require more detailed performance data against the measures contained in the business plan. Thus, an opportunity was lost to provide a suitable vehicle to report benefits-related data once the project had closed.

DBI has not taken any action to obtain more detailed benefits reporting on the Bio21 Institute and did not assess whether its investment in the Bio21 Institute would produce any more or fewer benefits than would have otherwise been achieved by undertaking an alternative investment.

Any such assessment should clearly differentiate between the benefits arising from the investments in the Bio21 Institute and other entities involved in the Bio21 Project.

Post-implementation reviews

Although there has been a broad review of the Bio21 concept, no independent review has been conducted to specifically assess achievements of the four key benefits espoused by the government (refer to Figure A2) when it funded the facility (i.e. hundreds of companies, millions in new investment, thousands of jobs, millions from exports).

DBI should sponsor reviews by appropriately qualified independent experts to confirm the economic and social benefits arising from the state's investment in the Bio21 Institute.