Enhancing Food and Fibre Productivity

Conclusions

Measuring and attributing the impacts of the department's RD&E investments is an inherently complex task because of the various external factors that influence agricultural production, the highly collaborative nature of agricultural RD&E in Australia, and the long time needed for RD&E activities to produce the desired benefits. However, there is evidence that the state's investment in RD&E contributes to productivity growth in Victoria's priority agriculture industries.

Many aspects of the department's approach to agricultural RD&E reflect good practice. The department has a robust investment framework that enables evidence-based priority-setting and investment decision-making. It also has a strong commitment to RD&E collaboration nationally. It has carried out many program- and industry-level evaluations of its RD&E activities.

However, the application of the department's investment and monitoring, evaluation and reporting frameworks has some gaps. These have limited its ability to show the basis for its investment decisions, to demonstrate their impact and to show how it has acted on the results of past RD&E programs. Also, the absence of a clearly documented overarching strategic direction for RD&E has resulted in a fragmented planning approach that does not clearly identify or communicate to staff and stakeholders how the department plans to attract and grow investment, maintain adequate RD&E capability or foster innovation.

The capacity for RD&E to drive future productivity growth and to achieve long-term goals continues to be challenged by a range of factors, including a lack of industry investment in key areas and constrained government resourcing. Future performance also depends on RD&E activities leading to change in practices and farmers adopting new technologies. Although many evaluations show where the department's R&D results have been effectively disseminated to end users and other stakeholders, the department has not followed through on plans to assess the overall effectiveness of its approaches to delivering services and changing practices.

Findings

Research priorities and funding allocations

Investment framework well designed but inconsistently followed

Every year, the department uses its agriculture investment framework to set RD&E priorities—which are detailed in investment plans for each of its four priority industries—and to select RD&E projects for investment based on agreed priorities. This investment framework and its selection criteria are well designed and enable a logical, evidence‑based approach to prioritising and choosing projects.

At an industry level, the department's RD&E priorities and projects are in line with state and national frameworks. However, inconsistent application of the key steps in the department's investment framework and a lack of evidence showing how specific decisions have been made have limited the reliability and transparency of the department's detailed decisions about RD&E priorities and projects.

The department has recognised the need to improve its approach to setting RD&E priorities for cross-industry issues and has expanded the scope of its industry leadership groups—which comprise departmental staff and provide advice to senior management on priorities, investments and stakeholders—to support this.

Need for an overarching strategic direction for investment in RD&E

Although the department's investment framework is well designed, its strategic direction for agricultural RD&E is not clearly documented. The department has many elements of a strategic direction detailed in a range of documents so it is in a good position to more clearly and cohesively communicate these to staff, to clarify its aims and directions for RD&E and to more closely guide investment decisions.

Industry co-investment in RD&E lacking in key areas

Bilateral agreements have been central to the department's capacity to get industry co-investment for RD&E in the dairy and grains industries. The absence of such agreements in the sheep and beef, horticulture, and soil and water sectors means that industry co-investment is unbalanced and not in line with the department's criteria for choosing projects.

Planning significant changes to national RD&E roles

In early 2016, the department reviewed its nominated priority roles under the national framework and it now plans significant changes. These proposed changes still need to be negotiated with other partners in the national framework.

If implemented, the proposed changes will mean more targeted research in priority industries and less productivity-focused research in industries where there is inadequate co-investment, with realignment into research areas that have greater public benefit such as biosecurity, traceability, animal welfare and value-chain efficiency. The department's rationale for these planned changes reflects:

• the evolving national model for RD&E, where more non-government bodies are carrying out research and government bodies are placing more emphasis on more focused, strategic research
• the need to realign the department's national RD&E roles with its existing capability and capacity.

Delivering research and development outputs to end users

The 'route to market', or 'path to impact', refers to the way the results of research and development (R&D) are further developed and delivered to the target audience—the next users or end users of the results. These results commonly include information, products and services. In agriculture, users are most commonly farmers, service providers such as agricultural consultants, industry bodies, other researchers, policy analysts and other government agencies.

The department has made the route to market a critical component of its investment decisions and project delivery. Although it has had many successes in delivering its R&D results to farmers, there are several aspects it can improve.

Good model for engaging service providers but not fully delivered

The department's 2009 Better Services to Farmers strategy improved its model for delivering R&D results to end users and involved a greater focus on working in partnership with private sector providers to deliver extension services. Evaluations of some larger programs show that the department has had success in delivering this partnership approach, particularly in the beef and sheep industries. Similar success with this model has been shown by the department's performance against performance indicators that it recently established.

However, as a result of incomplete implementation and monitoring of the model, the department is not yet engaging farmers and service providers as early, consistently and effectively as it needs to. It has also not monitored how well it has managed the risks associated with the model.

Selection of audiences and delivery methods needs to improve

The department can do more to make sure its R&D results and information are effectively tailored, accessible and targeted at the right audience. For many projects, it identifies only generic audiences and routes to market rather than selecting more specific approaches based on detailed considerations and analysis.

This undermines its investments in R&D and constrains innovation—it limits the department's ability to maximise the number of farmers it reaches with its information, products and services and the rate at which farmers adopt new practices.

Successful delivery can be further enhanced

The department has had many successes in delivering its R&D results to farmers, through both public and commercial routes to market. It has used a range of good practices to do this. However, across its more than 300 projects, it does not have sufficient oversight of whether it is systematically engaging the right audiences in the right way. Nor does it have sufficient assurance that it is using routes to market to increase the reach and accelerate the adoption of its information, products and services as much as possible.

Monitoring, evaluating and reporting on agricultural RD&E

Sound monitoring, evaluation and reporting framework followed inconsistently

The department's latest monitoring, evaluation and reporting (MER) framework for agricultural RD&E remains in draft format. Its documented practices are sound, but its performance measures, key performance indicators and targets do not focus enough on the impact of changes in agricultural practices and productivity on the long-term sustainability of natural resources.

The department regularly monitors the delivery of its RD&E projects and has carried out or contributed to more than 100 RD&E evaluations in the past five years. Over the past decade, it has also met most of its Budget Paper 3 measures that relate to agricultural RD&E.

There are gaps in the department's application of its documented MER practices which limit its understanding of the impacts of its RD&E activities and how its evaluations have been used. Weaknesses in its information systems further undermine its MER capacity.

'Mega' evaluations show significant RD&E impacts but have variable rigour and relevance

The department has a key role in measuring the impacts of its RD&E activities, which are difficult to assess. Its industry-level evaluations of RD&E are valuable and point to evidence of long-term improvements in the productivity and profitability of Victoria's agricultural industries which have been partly driven by RD&E investment. However, the rigour and relevance of these evaluations to RD&E varies. The department has room to improve how it prioritises, prepares for and coordinates these evaluations.

RD&E important in push to double food and fibre production by 2030

The department's RD&E activities will be a major contributing factor to Victoria's capacity to achieve the 2012 Growing Food and Fibre initiative's stated target of doubling food and fibre production by 2030. However, achieving this goal also depends on favourable economic and environmental conditions.

It remains unclear how achievable the target is, four years after it was set.

Recommendations

That the Department of Economic Development, Jobs, Transport & Resources:

1. consistently apply its agriculture investment framework to set research, development and extension priorities, make investment decisions, and plan for and use evaluations.
2. prepare an overarching investment strategy for agricultural research, development and extension that guides investment decision‑making by clarifying the role of government and describing its approach to:

• fulfilling its major, supporting and linking roles under the National Primary Industries Research, Development and Extension Framework
• promoting and encouraging innovation
• choosing and preparing suitably co-invested, collaborative research, development and extension projects
• aligning investment decisions with its risk appetite
• developing and maintaining the capability of staff and facilities.

3. clearly and consistently document its research, development and extension priority-setting and investment decisions so that it can demonstrate:
   • its rationale for agreed annual changes in priorities
   • how priorities have been shaped by industry engagement and past reviews and evaluations of research, development and extension programs
   • that it has considered the role and merits of government investment in setting priorities
   • how project investment decisions are in line with the agriculture investment framework's project selection criteria
   • how project investment decisions are in line with the objectives, themes, priorities and changes in the corresponding four-year industry investment plans
   • how investment decisions are informed by its investment strategy.

4. improve how it delivers research and development results to farmers and other users by:

    

5. more explicitly identifying and analysing potential audiences, delivery channels and partnership arrangements when planning the route to market
6. consistently involving next users and end users early in project development and throughout delivery
7. identifying and managing risks to effectively delivering development and extension activities
8. overseeing and measuring its overall success in optimising the access, reach and impact of its research and development information, products and services, including by evaluating the delivery and success of its service delivery model.

5. improve its monitoring, evaluation and reporting framework and practices by:
   • preparing evaluation plans for all key projects and actively using these to monitor, evaluate and report on industry-level performance.
   • developing or utilising external performance measures to provide assurance that changes in agricultural practices and productivity are not affecting the long-term sustainability of the natural resource base
   • addressing identified weaknesses in its information systems supporting research, development and extension project development, monitoring, evaluation and reporting
   • incorporating a documented approach to choosing, planning for and carrying out industry-level evaluations of the long-term impacts and benefits achieved by its research, development and extension investments

Submissions and comments received

We have professionally engaged with the Department of Economic Development, Jobs, Transport & Resources throughout the course of the audit. In accordance with section 16(3) of the Audit Act 1994 we provided a copy of this report to the Department of Economic Development, Jobs, Transport & Resources and the Department of Premier & Cabinet and requested their submissions or comments.

We have considered those views in reaching our audit conclusions and have represented them to the extent relevant and warranted. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

Victoria's agricultural sector comprises around 30 000 businesses and employs more than 190 000 Victorians, mostly in regional Victoria. The sector makes a large contribution to the state's economy. In 2014–15, Victorian food and fibre exports were worth about $11.6 billion—about 27 per cent of Australia's total food and fibre exports. Victoria's main export industries during 2014–15 were meat (earning $3.1 billion) and dairy (earning $2 billion), which collectively accounted for 44 per cent of the total value of the state's food and fibre exports.

Figure 1A

Victorian food and fibre exports by commodity group, 2014–15

Note: Fibre products comprise animal fibre such as wool, skins and hides, and forest products.

Note: Figures may not total 100 per cent due to rounding.

Source: Victorian Auditor-General's Office.

Australia's agricultural productivity continues to grow, but the rate of growth has slowed over the past two decades. The Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES) says that agricultural total factor productivity—a widely used index for measuring productivity in Australia—grew at an average annual rate of 2 per cent between 1948–49 and 2013–14. However, since the late 1990s, growth in agricultural total factor productivity has slowed to an average annual rate of 0.9 per cent.

ABARES attributes the slowing of productivity growth in Australian agriculture mainly to severe drought, but also to reduced public investment in agricultural research, development and extension (RD&E) since the 1970s.

Recent national and state initiatives and reviews have identified food and fibre as one of only a few sectors with the potential to boost economic development and innovation.

The federal government's National Innovation and Science Agenda identified food and fibre as a focus area and introduced a new innovation hub to encourage new technology and the use of big data in agriculture. The Committee for Economic Development of Australia's 2015 report Big Issues ranked agriculture as the second‑most important sector in providing future growth for the economy.

Food and fibre is one of the Victorian Government's six priority growth sectors. The Department of Economic Development, Jobs, Transport & Resources (the department) has estimated that farmers producing food and fibre manage around 60 per cent of the state's available land and use up to 70 per cent of available water.

Reviews of the food and fibre sector have identified a range of environmental constraints that limit farmers' capacity to increase productivity while sustaining the natural resource base on which the sector relies. These include:

• land degradation, such as soil erosion, salinity and waterlogging
• pests and diseases, and their resistance to management efforts
• water shortages, such as reduced allocations and poor water quality
• climate change, such as extreme weather events and wildfires.

1.2 Agricultural RD&E

RD&E underpins the productivity and competitiveness of agricultural industries. Agricultural RD&E can lead to improved farming systems and practices that increase the value and volume of production and reduce input costs.

Research typically includes:

• basic research—experimental or theoretical work to acquire new knowledge of the underlying foundation of phenomena and observable facts, without looking at any particular application of that knowledge
• applied research—investigating to acquire new knowledge, but focusing on a specific practical aim or objective.

Development is systematic work that focuses on producing or improving products or services, by drawing on existing knowledge gained from multiple sources including research and/or practical experience. It includes applying, adapting and validating known technologies and information to suit specific environments and practices.

Extension draws on research and development (R&D) and involves the public and private sectors working to transfer knowledge about ways to improve productivity and sustainability to and between farmers. Knowledge transfer may be direct or indirect through service providers.

Agricultural RD&E and productivity are directly linked. The 2011 ABARES report Public investment in R&D and extension and productivity in Australian broadacre agriculture, found that:

• public R&D investment had 'significantly promoted productivity growth in Australia's broadacre sector'—covering grain, beef and sheep production—between 1953 and 2007
• the relative contributions of domestic and foreign R&D had been 'roughly equal, accounting for 0.6 per cent and 0.63 per cent of annual total factor productivity growth in the broadacre sector, respectively'
• internal rates of return for research and extension were 15.4 to 38.2 per cent and 32.6 to 57.1 per cent a year, respectively.

The 2011 ABARES report also highlighted a decline in agricultural RD&E investment in Australia since the 1970s. Research intensity—the ratio of public spending on RD&E to agricultural gross domestic product—peaked at 5 per cent in 1978, before declining to 3 per cent in 2007. The annual rate of growth in public spending on agricultural R&D declined from about 7 per cent a year between 1953 and 1978 to about 0.6 per cent a year from 1978 to 2007.

State investment in agricultural RD&E has also been in long-term decline. Data from the department shows that:

• state government investment in agricultural research has declined by about 45 per cent since 1998, when adjusted for inflation
• state government investment in agricultural development and extension services has declined by 53 per cent since 2009, when adjusted for inflation.
• Victorian agricultural RD&E intensity declined from about 1.35 per cent in 1998 to nearly 0.6 per cent in 2015.

1.3 National Primary Industries Research, Development and Extension Framework

Victoria's agricultural RD&E is part of a national agricultural innovation system that is based on collaboration. Governments, researchers, farm businesses and suppliers work together to generate ideas and knowledge, to turn these into new products, services and processes, and then to support the agricultural sector to adopt them, with the aim of increasing productivity, profitability and sustainability.

A key element of this is the National Primary Industries Research, Development and Extension Framework (the national framework), which the Commonwealth Government began in 2009 to:

• increase collaboration and coordination between key stakeholders involved in RD&E—the Commonwealth Scientific and Industrial Research Organisation (CSIRO), Rural Research and Development Corporations, state government departments, industry and universities
• focus research capability to better address problems in and across industries
• focus RD&E resources so that they are used more effectively and efficiently.

The framework is based on the premise that basic research can be carried out nationally and the outputs can be adapted to meet regional and local needs. Under this model, each state can focus on its RD&E strengths and access additional information from the states that have strengths in other aspects.

Under the national framework, 14 national RD&E strategies have been prepared for specific agricultural industries and seven to address cross-industry issues. States decide what their research role is in specific industries by assigning a:

• 'major priority' role—the state will carry out a lead national role in R&D
• 'support' role—the state carries out some R&D, but others will provide the major effort
• 'link' role—the state will carry out little to no research in the industry, but will access information and resources from other states and agencies.

Figure 1B lists Victoria's 'major priority' roles under the national framework relevant to agriculture.

Figure 1B

Victoria's 'major priority' roles relevant to agriculture under the National Primary Industries Research, Development and Extension Framework

Source: Victorian Auditor-General's Office.

1.4 The department's role in RD&E

The department's Agriculture and Resources Group is responsible for delivering state‑funded agricultural RD&E in Victoria. It aims to sustainably increase productivity and competitiveness through RD&E. Its priorities for RD&E are the dairy, sheep, beef, grains and selected horticulture industries. These are the focus of the previous government's Growing Food and Fibre and Food to Asia Action Plan initiatives.

Figure 1C shows the department's role within the broad continuum for agricultural RD&E.

Figure 1C

The department's RD&E continuum

Note: CRCs are cooperative research centres; RDCs are rural research and development corporations; CMAs are catchment management authorities.

Source: Department of Economic Development, Jobs, Transport & Resources.

As shown in Figure 1D, the department received over $108 million in 2015–16 for agricultural RD&E, which includes external industry and other government sources. It receives matching funding for RD&E mainly through industry co-investors—primarily research and development corporations and the Commonwealth Government.

Figure 1D

Funding for agricultural RD&E in Victoria, 2015–16

(a) Covers areas not attributable to a specific industry.

Note: Figures may not total 100 per cent due to rounding.

Source: Victorian Auditor-General's Office, using data provided by the Department of Economic Development, Jobs, Transport & Resources.

To deliver its RD&E priorities, the department's Agriculture and Resources Group works with a range of public and private sector stakeholders, through various arrangements including:

• collaborations, such as AgriBio, a $288 million joint venture with La Trobe University that provides an agricultural biosciences R&D facility
• engaging external providers, including universities, R&D bodies and private providers that are contracted to provide research
• partnerships with organisations such as CSIRO, rural research and development corporations and cooperative research centres.

Measuring water movement in soils to improve pasture management practices. Photo supplied by the Department of Economic Development, Jobs, Transport & Resources.

Figure 1E details the various RD&E roles of each branch within the department's Agriculture and Resources Group.

Figure 1E

Branches in the Agriculture and Resources Group involved in RD&E

Source: Victorian Auditor-General's Office.

1.5 Victorian agricultural RD&E policies

The 2012 Growing Food and Fibre initiative aimed to double food and fibre production by 2030. The initiative included an investment of $61 million for targeted R&D over four years and ongoing annual funding of $15.7 million in subsequent years to boost productivity and profitability in Victoria's dairy, grains, red meat and horticultural industries.

The 2014 Food to Asia Action Plan aimed to work with industry in seven areas to grow exports of premium, safe and reliable food and beverage products to Asia and included investment of about $35 million. One of these seven areas included better-targeted RD&E. Actions under this theme covered:

• increasing support for RD&E services to improve natural resource management (including soils, water and climate adaptation)
• building understanding of Asian consumer preferences
• supporting small- to medium-sized enterprises to collaborate and innovate through strategic partnerships with innovation hubs
• exploring the next generation of milk components that will help to convert milk from a commodity to higher-value products for Asian markets.

The current government is investing $200 million to establish a Future Industries Fund that will support businesses in six high-growth sectors, including food and fibre. This has included the release of the Food and Fibre Sector Strategy in March 2016, which sets priorities for the sector such as helping businesses to innovate and adopt technology.

1.6 Audit objective and scope

The audit objective was to assess the extent to which agricultural RD&E is used to drive innovation, productivity and practice change.

To address this objective, the audit assessed whether the department's:

• research priorities and funding allocations are based on evidence and aligned with relevant national and state frameworks
• R&D outputs have been effectively disseminated to end users and other relevant stakeholders to drive improvements in innovation, productivity and practice
• RD&E activities are effectively monitored, evaluated and reported on to demonstrate the achievement of intended outcomes and drive continual improvement.

Machinery-of-government changes in recent years have resulted in changes to agency responsibilities for agricultural RD&E in Victoria. Specifically, agricultural RD&E had previously been managed by:

• the former Department of Environment and Primary Industries between April2013 and January 2015
• the former Department of Primary Industries until April 2013.

For convenience, this report attributes the RD&E activities of these former departments to the Department of Economic Development, Jobs, Transport & Resources.

1.7 Audit method and cost

The audit examined agricultural RD&E activities through document reviews and interviews with the department.

The audit was carried out under sector 15 of the Audit Act 1994, in keeping with the Australian Auditing and Assurance Standards. Pursuant to section 20(3) of the Audit Act 1994, unless otherwise indicated, any persons named in this report are not the subject of adverse comment or opinion.

Total cost of the audit was $420 000.

1.8 Structure of the report

The report is structured as follows:

• Part 2 looks at the prioritisation of and investment in RD&E
• Part 3 looks at how R&D outputs are disseminated
• Part 4 looks at the department's RD&E monitoring, evaluation and reporting framework and the benefits and impacts of agricultural RD&E.

2.1 Introduction

The National Primary Industries Research, Development and Extension Framework (the national framework) is highly collaborative, with many participants. For the national framework to be effective, each state and territory must prioritise and invest in research, development and extension (RD&E) activities that are in line with its nominated roles.

Therefore, it is important that the Department of Economic Development, Jobs, Transport & Resources' (the department) agricultural RD&E priorities and investments are consistent with the national framework. It is also important that priorities and investments are:

• consistent with state policies, strategies and plans
• supportedby evidence-based decision-making to provide assurance that resources are being targeted to the areas of greatest need.

Selecting better crop varieties using robotic imaging technologies. Photo supplied by the Department of Economic Development, Jobs, Transport & Resources.

2.2 Conclusion

The department's investment framework for agricultural RD&E allows for an evidence‑based approach to setting priorities and making decisions about investments. The department's industry-level priorities and investment decisions are in line with national and state frameworks. It is taking steps to improve its (previously lacking) focus on prioritising RD&E in cross-industry areas.

Although the investment framework is well designed, the department has not applied it consistently and this has resulted in a lack of assurance that detailed changes in priorities and investment decisions have been well informed. The absence of a clear overarching strategic direction document for RD&E investment in Victoria further limits the investment framework's potential effectiveness. The department is in a strong position to address this latter issue, as it already has aspects of a strategic approach outlined across various documents.

In response to a lack of industry co-investment for RD&E in priority sectors—and a range of other issues—the department is planning significant changes to its nominated roles under the national framework. If put into practice, this will lead to a more targeted approach to agricultural RD&E in Victoria.

2.3 The investment framework for agricultural RD&E

Every year, the department's Agriculture and Resources Group uses its agriculture investment framework to:

• set RD&E Priorities and detail these in the investment plans for each of its four priority industries—dairy, grains, horticulture and sheep and beef
• select RD&E Projects for investment based on agreed priorities.

Figure 2A outlines the investment framework's cycle.

Figure 2A

The agriculture investment framework

Source: Department of Economic Development, Jobs, Transport & Resources.

2.3.1 Well-designed investment framework

The department's agricultural investment framework enables effective, evidence-based RD&E priority-setting and project selection in line with the national framework. This is achieved through:

• quarterly industry fora that give the department the opportunity to discuss existing and potential future industry priorities with key stakeholders in its four priority industries—dairy, grains, horticulture and sheep and beef
• regular meetings of industry leadership groups(ILG), made up of departmental staff, that provide advice to senior management on priorities, investment allocation and stakeholder management in its four priority industries
• an annual planning conference where draft four-year industry investment plans—which are produced by ILGs and specify RD&E Priorities and proposed investment shifts—are discussed by senior management before being finalised
• an annual investment conference where current and proposed agricultural RD&E projects are presented to senior staff for comments and discussion
• providing opportunities for evaluations of past projects to inform future investment decisions.

The department has also regularly reviewed its investment framework with a view to making ongoing improvements. A 2012 review commissioned by the department found that the framework was transparent, promoted strategic focus, facilitated consultation and had positive impacts for RD&E.

2.3.2 Adequate project selection criteria

The investment framework's selection criteria for agricultural projects provide a logical basis on which to make decisions about RD&E investments. These criteria are summarised in Figure 2B.

Figure 2B

Selection criteria for agricultural projects

(a) ‘Route to market’ refers to the way the results of research and development are further developed and delivered to the target audience—the next users or end users of the results.

(b) Reviewed after all project proposals are assessed individually.

Source: Victorian Auditor-General's Office.

2.4 Aligning priorities and investments

There is alignment between the department's industry-level RD&E priorities and projects and the state and national frameworks.

2.4.1 Industry-level priorities in line with national framework

Victoria's industry-focused agricultural RD&E priorities are in line with the national framework.

The areas for which the department has a lead role under the national framework—dairy, grains, horticulture and sheep meat—strongly influence prioritisation. The department's focus areas, such as almonds (horticulture) and pulses (grains), are also in keeping with the national framework.

Since 2007, the Victorian Government and the department have had a central role in administering the national framework. In a 2009 joint statement of intent with the other states and territories, the government committed to 'work collectively and collaboratively to implement the Framework and underpinning industry and cross sector strategies' and to put into practice the framework's nine principles. The department has largely put these principles into practice. The principles required that the states endeavour to at least maintain funding levels, but funding across Australia has been declining since the 1970s. Victoria's funding has declined by about $6 million, or 10 per cent, since 2013.

2.4.2 Industry-level priorities in line with policies and strategies

The department's agricultural RD&E priorities for each of its four priority industries are also in line with the relevant state policies and strategies, including the 2012 Growing Food and Fibre initiative and the 2014 Food to Asia Action Plan. The main objective of the department's Agriculture, Energy and Resources strategic plan—'increasing productivity'—is clearly in line with these and sets the objective for RD&E investment planning in all four industries. This broad objective is also in line with the industries' objectives and the aims of the national RD&E strategies.

The national RD&E strategies strongly influence the selection of investment themes in the four industries. These themes are wide ranging and change little from year to year. They include but are not limited to:

• dairy—flexible dairy feeding systems
• grains—more efficient grain production practices, more profitable farming systems, and better management of risk while maintaining or improving grain quality and the natural resource base
• horticulture—innovative production systems—advanced, efficient production for the 21st century
• sheep and beef—flexible animal production systems.

The strong influence of the national framework appropriately reflects its purpose, which was to harmonise RD&E efforts so that each state focused on areas that were strategically important and reduced duplication with other states.

2.4.3 Investments in line with industry-level priorities

The department's RD&E project investments over the past three financial years have been made in line with its four priority industries. Figure 2C details the department's key projects, which comprise strategic groupings of all its individual RD&E projects.

Figure 2C

The department's key projects for agricultural RD&E

Source: Victorian Auditor-General's Office.

The alignment between the department's project selections and the detailed priorities set in its four-year industry investment plans is discussed below.

2.5 Gaps in applying the investment framework

Inconsistent application of the key steps in the department's investment framework and a lack of evidence showing how detailed decisions have been made have limited the reliability and transparency of the department's decisions regarding RD&E priorities and projects. The department has recognised the need to improve its approach to setting RD&E priorities for cross-industry matters and has revised the structure of its ILGs to support this.

2.5.1 Key framework steps followed inconsistently

Over the past five years, the department has not consistently applied all key steps of the investment framework's annual cycle as intended.

No planning conference was held in 2015, 2013 or 2011, limiting the reliability of RD&E priority-setting in these years. No investment conference was held in 2014, limiting the reliability of detailed RD&E investment decisions for that year and assurance that they were in line with agreed priorities.

Quarterly industry stakeholder fora—which started in 2014—have not taken place since the second quarter of 2015. The department noted that the limited availability of participating industry bodies makes it difficult to schedule these fora every three months.

The department told us that machinery-of-government changes in 2013 and 2015 contributed to it not following its investment framework completely.

2.5.2 Gaps in transparency of specific decisions

Lack of evidence to support specific priority-setting decisions

The department's detailed RD&E priorities have not been set transparently. Specifically, the department is lacking documents that show:

• how the department's ILGs have gone about filtering potential RD&E priorities, including assessing the merits and impacts of investing or divesting in different priority areas
• the department's reason for annual changes in the RD&E priorities
• the extent to which RD&E priorities have been shaped by industry engagement and past reviews and evaluations of RD&E programs
• how the department has considered the role and merits of government investment compared to private investment
• senior managers having agreed to RD&E priorities before industry investment plans are finalised.

Likely project benefits assessed but adherence to selection criteria not always clear

Under the investment framework, individual branches in the department's Agriculture and Resources Group have the flexibility to prepare and assess RD&E project concepts and proposals in a way that is fit for purpose. This flexibility has led to the Agricultural Research and Farm Services branch and the Biosciences Research branch each adapting the investment framework's selection criteria to meet their respective needs. In principle, this is reasonable—so long as the criteria remain meaningful and aligned. It reflects differences in the nature and scale of RD&E activities between branches.

Both branches estimate net benefits for their project proposals, albeit in different ways:

• The Agricultural Research and Farm Services branch assesses each proposal using a 'rapid net benefits assessment'—a tool that allows scoring and ranking of proposals against its adapted criteria. The rigour of these assessments is adequate considering that its proposals are typically more numerous and smaller in scale compared to biosciences research proposals.
• In contrast, the nature of biosciences research means that the Biosciences Research branch typically invests in fewer RD&e projects but its projects are larger and longer term, developed as national or international programs. This results in bioscience project investments that are supported by more detailed, collaborative planning and assessments of likely benefits.

The Biosciences Research branch's project planning is extensive, and there is evidence that its investment decisions align with its adapted selection criteria. However, this alignment needs to be more clearly and consistently documented before final decisions about investments are made.

Figure 2D

Case study of bioscience project development: DairyBio

Source: Victorian Auditor-General's Office.

Aligning key project plans more clearly with industry investment plans

The department can improve how its key project plans demonstrate alignment with the objectives, themes, priorities and changes in its four-year industry investment plans. Although these key project plans provide commentary describing the proposed changes in investment for the upcoming financial year, it remains difficult to reconcile this with the corresponding investment plan.

2.5.3 Focusing better on cross-industry RD&E

The department has major RD&E roles under the national framework for six cross‑industry areas. In fulfilling these roles, it has chosen to identify cross-industry priorities and deliver corresponding RD&E projects through its four industry-themed investment plans, rather than through dedicated cross-industry investment plans. However, the extent to which the four industry investment plans address cross-industry issues varies.

As part of the department's 2014 planning conference, it recognised the need to develop a better approach to addressing cross-industry RD&E priorities identified in the industry investment plans. This has led to the department introducing two new cross‑industry ILGs in 2016 focused on:

• soils, water and climate change
• adding value to Victorian agriculture.

Measuring grain growth and quality in response to elevated levels of carbon dioxide in experimental field trials. Photograph supplied by the Department of Economic Development, Jobs, Transport & Resources.

The department's potential to identify cross-industry issues has also been enhanced by recent changes to its industry-focused ILGs. For example, the scope of its sheep and beef ILG has been broadened to include all livestock industries, which allows it to better focus on animal welfare issues.

It is too early to assess the effectiveness of the new ILG structure in addressing the department's previous lack of focus on cross-industry RD&E priorities.

2.6 Strategic approach to investment

The department has carried out a range of strategic work that has increased the efficiency and effectiveness of its agricultural RD&E. Its current strategic directions are articulated in a wide range of documents, but there is no overarching strategy or document that brings these together clearly.

2.6.1 Benefits of past strategic planning

The strengths of the department's current approach stem from earlier strategic planning. These strengths include:

• significant investments in RD&E facilities and capability, most notably AgriBio, a $288 million joint venture with La Trobe University that provides an agricultural biosciences R&D facility
• signing longer-term bilateral investment agreements with several key stakeholders
• introducing ILGs to provide industry-specific advice to support setting priorities and making decisions about investments
• its contribution to establishing and administering the national framework.

2.6.2 No overarching strategic direction documented

The current investment framework and other planning documents do not clearly identify and communicate the general principles that underpin the department's approaches to attracting and growing investment, or its risk appetite for doing this. This includes the amount and types of project investments it is prepared to make, given the level of benefits it seeks from these investments, and the amount and sources of co‑investment for these, both existing and new. The department also does not clearly identify what new approaches and incentives it will promote to encourage innovation or what capability mix it needs across the agriculture portfolio to deliver its forward plan.

Some aspects of the department's strategic direction are fragmented across a range of separate documents or are known by staff but not recorded. These include its principles and models for considering the role of government investment and how the role is evolving in agricultural RD&E and innovation.

Some of the strategic direction for agricultural RD&E in these areas was included in the former Department of Environment and Primary Industry's 2014 Science Strategy, but the current department does not have an equivalent strategy.

A range of the department's documents already detail many elements of an investment strategy. This places it in a good position to more clearly and cohesively communicate the key elements to staff, to clarify its aims and directions for RD&E, and to more closely guide investment decisions. It could also develop a 'road map' to link this overarching strategy to the relevant strategies and plans that collectively provide the detail of its approach.

2.7 Industry co-investment for RD&E projects

The amount of co-investment in a proposed RD&E project and the relative role of government investment are key elements of the department's project selection criteria. In practice, this means that projects should only proceed when:

• a market failure exists
• the balance of co-investment aligns with the potential beneficiaries.

Figure 2E shows varying levels of co-investment for RD&E projects in the department's priority industries since 2013–14.

Figure 2E

Investment in agricultural RD&E, 2013–14 to 2015–16

Source: Victorian Auditor-General's Office, using data provided by the Department of Economic Development, Jobs, Transport & Resources.

External partnership agreements have been important to the department's ability to secure co-investment for RD&E projects in Victoria's dairy and grains industries. Conversely, the absence of similar funding agreements focused on the sheep and beef, horticulture, and soil and water sectors means that the co-investment for RD&E in these areas is small by comparison. The department acknowledges that the co‑investment for RD&E projects in these areas is unbalanced and not in line with its project selection criteria.

2.8 Reviewing roles under the national framework

In early 2016, the committee responsible for overseeing the national framework requested that all states and research organisations review their nominated major priority, support and link roles. After its review, the department plans significant changes to its national RD&E roles. These changes have been approved by the department but still need to be negotiated with other partners in the national framework.

2.8.1 Planned changes reflect a more targeted approach to RD&E investment

If implemented, the department's proposed changes will mean:

• more targeted research in priority industries
• less productivity-focused research in industries where there is inadequate co‑investment
• are aligned focus on research areas that have greater public benefit such as biosecurity, traceability, animal welfare and value chain efficiency.

Using smart phone apps to rapidly identify plant pests and diseases.Photo supplied by the Department of Economic Development, Jobs, Transport & Resources.

2.8.2 Planned changes informed by evolving RD&E system and capacity challenges

The department's internal briefings about its planned changes note that there 'has been significant change in the research and funding environment' since national roles were agreed in 2009, where the national model for RD&E now has:

• more industry, private-sector bodies and universities doing research
• more emphasis on governments doing focused research that is often more strategic
• more need for research to be translated and packaged to improve access for a wide range of organisations that provide extension services
• both industry and the private sector seeking to accelerate R&D outcomes and economies of scale through regional hubs and precincts.

There are also indications that the department needs to realign its national roles with its existing capability and capacity. Specifically, the briefings note that:

• its planned changes reflect the 'significant fiscal pressures and the competition for government funding from across the economy'
• its 2009 commitment to be a major research provider across 10 areas—while also having a central role in administering the national framework—'was an ambitious commitment, and to an extent an over-statement of the department's research role in some areas'.

Recommendations

That the Department of Economic Development, Jobs, Transport & Resources:

1. consistently apply its agriculture investment framework to set research, development and extension priorities, make investment decisions, and plan for and use evaluations.
2. prepare an overarching investment strategy for agricultural research, development and extension that guides investment decision-making by clarifying the role of government and describing its approach to:

• fulfilling its major, supporting and linking roles under the National Primary Industries Research, Development and Extension Framework
• promoting and encouraging innovation
• choosing and preparing suitably co-invested, collaborative research, development and extension projects
• aligning investment decisions with its risk appetite
• developing and maintaining the capability of staff and facilities.

3. clearly and consistently document its research, development and extension priority-setting and investment decisions so that it can demonstrate:

• its rationale for agreed annual changes in priorities
• how priorities have been shaped by industry engagement and past reviews and evaluations of research, development and extension programs
• that it has considered the role and merits of government investment in setting priorities
• how project investment decisions are in line with the agriculture investment framework's project selection criteria
• how project investment decisions are in line with the objectives, themes, priorities and changes in the corresponding four-year industry investment plans
• how investment decisions are informed by its investment strategy.

3.1 Introduction

'Route to market', or 'path to impact', refers to the way results of research and development (R&D) are further developed and delivered to the target audience—the next users or end users of the results. These results commonly include information, products or services. In agriculture, the users are most commonly farmers, service providers such as agricultural consultants, industry bodies, other researchers, policy analysts and government agencies.

The Productivity Commission's 2011 report Rural Research and Development Corporations identified that accelerating dissemination and adoption locally was one of the biggest challenges facing Australia's agricultural research bodies.

For most R&D activities, the selected route to market is through public dissemination. Examples include making results available online, publishing them in peer-reviewed journal papers, mobile applications, in online farmer networks or in agricultural media, and disseminating them via field demonstrations and workshops. The less common route to market is through commercialisation, where new technology or information is licensed or sold on a commercial or cost-recovery basis that generates income. Commercial routes include licensing or selling a patent and plant breeding rights.

Identifying and protecting intellectual property created as a result of R&D activities can help to maximise the benefits of the work, for both public and private routes to market.

3.2 Conclusion

The Department of Economic Development, Jobs, Transport & Resources (the department) has made route to market a critical component of its investment decisions and project delivery. Although it has had many successes in delivering its R&D results to farmers, it can improve the way it targets the proposed audience and identifies preferred channels for delivering the results, to help it deliver greater innovation and changes to current agricultural practices.

A more tailored approach that extends beyond the department's traditional audiences and delivery channels is likely to help deliver the innovation and practice change needed to significantly boost productivity.

The department needs to focus specifically on overseeing the implementation of its engagement model and its effectiveness in using the route to market to maximise the reach, accessibility and impact of its work, across more than 300 projects.

3.3 Engaging with farmers and service providers

The department's approach to ensuring that its research, development and extension (RD&E) work results in practice change and improved productivity remains heavily influenced by its 2009 Better Services to Farmers strategy. This strategy was aimed at improving the targeting, accessibility and relevance of the department's development and extension services.

To achieve this, the department changed its service delivery model in two ways:

• exiting some service areas where consultants and other organisations were providing services
• focusing on working cooperatively with private service providers in other areas, to jointly deliver extension services.

This change to the delivery model followed a period of growth in the number of private providers delivering extension services and coincided with other state governments changing the way they deliver agricultural extension activities.

Under the new model, the department identifies target end users—specific groups of farmers—and then works in partnership with private service providers to:

• shape information, products and programs for the service providers to deliver to those farmer groups
• build the service providers' capability to deliver the information, products and programs.

This model is illustrated in Figure 3A. Its direct work with farmers is more focused on aspects of farm management that deliver 'public good', such as environmental management and biosecurity.

Figure 3A

Components of the Better Services to Farmers service delivery model

Source: Department of Economic Development, Jobs, Transport & Resources.

The department's Horticulture Industry Network program began in 2009 and was designed to follow this model—see Figure 3B.

Figure 3B

Case study of the new service delivery model —the Horticulture Industry Network program

Source: Victorian Auditor-General's Office.

The Better Services to Farmers model is well designed and has improved the way the department works with the users of its RD&E results. However, the department is not engaging farmers and service providers as early, consistently and effectively as it could.

3.3.1 Good engagement model but not fully delivered

A significant change resulting from the new service delivery model was the decrease in departmental extension staff that deliver services to farmers from at least 360 in 2009−10 to less than 120 in 2015−16. The department's data indicate that, since 2009, the state government's investment in development and extension decreased by approximately 53 per cent when adjusted for inflation.

A number of examples show that the department has worked successfully with private sector service providers and has avoided competing against them—for example, the department withdrew from dairy productivity extension when Dairy Australia increased its focus on this area. In its commercial activities, the department has shifted its focus, too—for example, as the private plant breeding sector has grown, the department has shifted its focus from breeding to pre-breeding for some plant varieties.

Although the model has been in place for seven years, the department has not assessed the extent to which the model has improved the targeting, accessibility and relevance of its services. The department does not know whether it achieved the initial targets of reducing time to adoption by 25 per cent and increasing audience reach by 50 per cent.

The department had a four-year plan to monitor and evaluate its success in delivering the model, but it did not follow through with many planned activities such as the final evaluation, as shown in Figure 3C.

Figure 3C

Progress monitoring and evaluation of the new development and extension model

Note: ✔ = successfully completed elements, ~ = partially completed, ✘ = not completed.

Source: Victorian Auditor-General's Office.

The interim reports, surveys and mid-term evaluation identified some successes flowing from the model but also some poor progress.

By 2012, many farmers and service providers considered that the department had increased the relevance and accessibility of its services, that it had increased collaboration with service providers and that it was better targeting the audience for its work. However, improvements had not been achieved consistently in all agricultural RD&E activities, and poor performance indicators and data collection made it difficult to measure progress. The department had not engaged strategically with service providers, and the department's staff lacked a shared understanding of how to partner with them.

The department identified some reasons for this, including a lack of accountability for delivering the strategy, an absence of leadership for the changes required and inadequate resourcing.

The department responded to some issues, such as insufficient data collection, and reissued the strategy in 2013—but it did not change its overall approach to putting the strategy into practice. From late 2013 to 2015, progress on implementing the strategy was affected by staff reductions to meet government efficiency targets and the transfer of the agriculture group between departments as a result of machinery-of-government changes.

The department also identified new performance indicators in 2013. It has measured its progress against some of these indicators since 2014–15, meeting the targets set. Recent evaluations of its larger programs show many successes in delivering this partnership approach, particularly in the beef and sheep industries.

3.3.2 Engagement risks not monitored as planned

The Better Services to Farmers strategy had also identified several risks that were common in the experience of other international jurisdictions where they had moved from working directly with farmers to working only with private service providers. The strategy was designed to mitigate these risks, particularly those associated with the new model of partnering with private providers to develop and deliver services.

However, the department did not put additional risk management measures into practice and has not formally or systematically assessed the extent to which it has managed these risks. It gains some assurance from measuring client satisfaction and monitoring the development of partnerships with the private service provider sector. However, information from its reviews—or, in some instances, a lack of information—suggests that it is likely that the department remains exposed to several risks, including:

• lack of integration of productivity and environmental or social goals
• marginalisation and reduced credibility of the government department, particularly with farmers
• impact on small and/or remote landholders
• loss of skills, knowledge and staff motivation in the public sector.

3.4 Selecting the audience and delivery methods

The department can do more to ensure that its R&D results and information are effectively tailored, accessible and targeted to the right audience. For many projects, the department identifies only generic audiences and pathways rather than selecting more specific approaches based on detailed considerations and analysis.

This undermines its investments in R&D and constrains innovation—it limits the department's ability to maximise the number of farmers it reaches with its information, products and services and the rate at which farmers adopt new practices.

3.4.1 Clear, largely sound requirements

The department's investment framework and project monitoring, evaluation and reporting processes require end users and routes to market to be considered early in the development of projects and during delivery. They do not specify how this should be done, although the National Primary Industries Research, Development and Extension Framework's extension principles and the department's own agriculture intellectual property framework provide more detailed guidance.

There is a hierarchy of state, departmental and agriculture-specific intellectual property policies and guidelines that thoroughly detail the principles and steps that the department needs to follow to protect and commercialise its intellectual property. The department has considered these policies and guidelines in preparing its investment framework and its project development and management processes.

3.4.2 Some good practices

The department carries out general market research in all agriculture industries to understand its farmer and service provider audiences, and carries out specific research for some individual projects. It also explores the potential of new channels and tools to disseminate the output of its R&D work.

The department also aims to use knowledge about how people learn and what makes people change practices to work out the level and format of information it delivers and to work out which users and farmers to target, as shown in Figure 3D. This approach has been used well on some projects but not used at all on others.

Figure 3D

How R&D can lead to practice change

Source: Victorian Auditor-General's Office.

The department uses a range of other good practices on some projects to identify the target audiences and routes to market. These include:

• clearly aligning the selection of delivery channels/tools and the associated communications strategy for the project with knowledge of the issues, barriers to adoption and information needs of the target audience
• involving users in designing the project
• documenting risks to the project and intellectual property considerations.

3.4.3 Planning often too generic to be useful

Although the department routinely includes route-to-market information in its project plans and has examples of good planning that meet the requirements, its information is more often generic and does not:

• usefully identify the range of audiences it needs to engage with and deliver to—particularly the next users such as service providers
• critically assess which delivery pathways give it the best chance of getting the R&D results adopted—including which pathways best meet users' preferences for receiving information and what the barriers to adoption are
• consistently apply its requirements and good practices to projects.

In general, projects do not describe strategies for setting up and maintaining collaborations and partnerships with service providers.

Route-to-market planning could also be improved by consistently including success measures and targets—such as for anticipated audience reach and access, and for uptake and practice change by farmers. Consistently including baseline information to help measure future performance—such as farmers' existing awareness and practices—would help this.

Some factors make it difficult to be specific about route to market on some projects. These factors include:

• the experimental nature of research
• uncertainty about the final outputs
• the potential for user needs to change quickly, such as with changing seasons and market conditions, and new technology.

However, the generic information included in project plans does not reflect the importance the department places on route to market as one of its key project investment criteria. Poorly developed routes to market can undermine the effort invested in the research—for example, if the research results do not lead to sufficient changes in farming practice. This also makes it difficult to accurately budget for delivering R&D results to farmers.

3.5 Delivering R&D results to farmers

The department has had many successes delivering its R&D results to farmers. However, it does not have good oversight of whether it is systematically engaging the right audiences in the right way. Nor does it have good assurance that it is using routes to market that will amplify the reach and accelerate the adoption of its information, products and services as much as possible.

3.5.1 Good delivery practices but used inconsistently

In disseminating R&D results and engaging and influencing its target audiences for different projects, the department has used a range of good practices, including:

• making the most of its expertise and connections to build pathways with providers and producers
• engaging target audiences early when delivering projects
• adapting delivery channels as the project progresses, based on further assessment of the audience types and their preferred pathways or through feedback and observation of the project's progress.

The department also uses a comprehensive and contemporary range of media and formats to ensure its R&D outputs are effectively tailored, targeted and accessible to end users and other relevant stakeholders. Channels used to disseminate outputs include scientific publications, reports, fact sheets, direct interactions, tools, virtual networks, and online and mobile technologies. In using these channels, the department has applied successful formats from one area to other areas and adapted approaches from other agencies and jurisdictions. In turn, other organisations have adapted the department's work.

The department's intellectual property protection and commercialisation activities have generally followed the pathways recommended by the hierarchy of intellectual property policies and guidelines. For its commercial projects, requiring maximum adoption plans and minimum performance standards are an additional and important focus of tendering processes for licence agreements.

However, many of these good practices have not been used consistently in all the department's RD&E projects.

3.5.2 Many examples of successful delivery

The department's project reporting and its evaluations show many examples where individual RD&E projects have successfully delivered information, products and services to farmers and where this has led to farmers adopting new practices. The department measures project achievements in a range of ways—such as numbers of service providers and farmers engaged with, their satisfaction with the engagement and their intentions for, or actual, practice changes.

In 2014–15, the department introduced new measures of how well it engaged farmers and service providers and satisfaction with its services. It met these targets.

The extent to which delivery of R&D outputs leads to farmers adopting new practices is not measured as often, because of the time lag between when the project is delivered and when maximum adoption is expected—which also makes it difficult to access the funding needed to do the evaluation.

Other aspects of route to market that are not often measured include:

• the department's success in targeting the right audiences and choosing the best routes to market, and how this has helped it maximise access and adoption
• how readily the public can discover and use the information and data that is suitable for open access, as required by the department's intellectual property framework.

Given the difficulty in measuring practice change, improving the department's assessment of how well it manages its route-to-market activities would help assure that it is maximising its chances of success.

Demonstrating improved farm management practices to farmers and agricultural consultants. Photo supplied by the Department of Economic Development, Jobs, Transport & Resources.

Public delivery success

As about 95 per cent of the department's projects use public routes to market, getting these routes right is an important step in getting farmers to adopt its products and services. The department uses a range of measures to assess its success in this area, as shown in Figure 3E.

Figure 3E

Examples of program success in public routes to market

Source: Victorian Auditor-General's Office.

It can be challenging to demonstrate success from public routes to market, as it can be difficult to identify how much the department contributed to a change in practice compared with all the other factors that may be affecting the industry.

The change to the service delivery model has made this more difficult, with the department working less directly with farmers and more through private service providers.

The department's success disseminating productivity-related R&D results to farmers relies not just on the department working effectively with private service providers, but also on the private service providers working effectively with farmers. Where activities are delivered in partnership with private providers, the department evaluates effectiveness in delivering farmer-level impacts, providing several examples of success. Where programs are not jointly delivered, the department does not actively monitor how effectively service providers work with farmers. Instead, it gets some information from the rural research and development corporations, which measure this in some industries.

The department could strengthen its understanding of success in this area by periodically assessing the combined results of its evaluations and those of the rural research and development corporations, as well as using selected evaluations to examine its engagement processes.

The department's data indicates that it has strengthened its partnerships over time by signing more formal agreements with service providers, as recommended by Better Services to Farmers and subsequent reviews.

Commercial delivery success

In some instances, adoption of the department's inventions by industry will be faster and more widespread if the information is delivered through a commercial entity, rather than being disseminated directly to the public. In such instances, the department uses intellectual property rights and a commercial route to market to maximise productivity from its inventions. Typically, this involves the department licensing rights to its intellectual property to private commercial entities, which gives them market advantages in developing new products and services for farmers.

Although this only applies to about 5 per cent of the department's RD&E projects and 11 per cent of its total investment, the department's monitoring of the success of its commercial routes to market shows these commercial routes to market can have significant widespread and rapid impacts, as shown in Figure 3F.

Figure 3F

Examples of success in commercial routes to market

Source: Victorian Auditor-General's Office.

There are also some cases where the department has developed technology, protected the intellectual property and offered licence rights to the private market to commercialise it, but the market has not been willing to invest in the technology until issues and risks such as regulatory requirements, product segregation, product affordability or consumer perception requirements were fully addressed.

3.5.3 Sound arrangements for commercialising intellectual property

Where a commercial route to market is likely, the department works with Agriculture Victoria Services (AVS) to identify routes to market and commercialise the products. AVS is the department's commercial arm, created to protect intellectual property, to commercialise R&D outputs and to provide an interface between agriculture R&D and the private sector. The department is responsible for identifying the intellectual property and the routes to market, while AVS assumes responsibility for protecting and commercialising the intellectual property. The department pays $2 million a year for these services through a formal five-year service agreement.

The number of intellectual property protections held by AVS has been relatively stable over time—see Figure 3G. In any one year, there is a small annual turnover of the intellectual property portfolio.

Figure 3G

Intellectual property protections, 2011–15

Source: Department of Economic Development, Jobs, Transport & Resources.

In 2014–15, the commercial routes to market stemming from this intellectual property protection generated $7.74 million in royalties and licence fees, particularly from licensing new varieties of barley, pulses and canola. This revenue is paid into a fund managed by the department's Agriculture and Resources Group and used to fund the generation, protection and commercialisation of new intellectual property assets and provide for additional science capability.

Recommendation

1. That the Department of Economic Development, Jobs, Transport & Resources improve how it delivers research and development results to farmers and other users by:

• more explicitly identifying and analysing potential audiences, delivery channels and partnership arrangements when planning the route to market
• consistently involving next users and end users early in project development and throughout delivery
• identifying and managing risks to effectively delivering development and extension activities
• overseeing and measuring its overall success in optimising the access, reach and impact of its research and development information, products and services, including by evaluating the delivery and success of its service delivery model.

4.1 Introduction

Measuring and attributing the impact of the Department of Economic Development, Jobs, Transport & Resources' (the department) research, development and extension (RD&E) activities is challenging because of economic and environmental factors that influence agricultural production, the highly collaborative nature of agricultural RD&E and the typically long time between research and development (R&D) activities and impacts.

However, the department has a key role in monitoring, evaluating and reporting (MER) on the delivery and outcomes of its RD&E investment in Victoria's agricultural sector.

4.2 Conclusion

Although it is difficult to assess, there is evidence that the department's RD&E investments contribute to productivity growth in Victoria's priority agricultural industries.

The department's MER framework is mostly sound and has supported a wide range of RD&E-related evaluations, project monitoring and reporting. However, gaps in the application of its MER framework, a lack of guidance driving its industry-level 'mega' evaluations and inadequate information systems underpinning its MER activities have limited the extent to which the department can demonstrate RD&E impacts.

Victoria's capacity to double food and fibre production by 2030—the overall target of the 2012 Growing Food and Fibre initiative—remains unclear, although RD&E will play a significant role along with environmental and economic conditions. This uncertainty is heightened by the retrospective and inconclusive nature of the modelling used to show how the target could be met.

4.3 The MER framework

The department's latest MER framework for agricultural RD&E remains in draft form. The framework's documented practices are sound, but its performance measures, key performance indicators and targets do not focus enough on how changes in agricultural practices and productivity affect the long-term sustainability of natural resources.

The department regularly monitors the delivery of its RD&E projects and has carried out or contributed to more than 100 RD&E evaluations in the past five years. Over the past decade, it has also met most of its Budget Paper 3 measures that relate to agricultural RD&E.

However, gaps in the department's application of its MER practices limit its understanding of the industry-level impacts of its RD&E activities and how its evaluations have been used. Weaknesses in its information systems further undermine its MER capacity.

4.3.1 MER arrangements

The department's Agriculture and Resources Group has drafted a replacement for its 2010 MER framework for agriculture and fisheries, which is now out of date. Although the framework remains in draft form, it provides an up-to-date outline of MER processes for RD&E.

Monitoring and reporting arrangements

The framework details monitoring and reporting arrangements at multiple levels:

• Four-year industry investment plans prescribe monitoring by industry leadership groups against the plans' performance indicators and targets.
• Budget Paper 3 measures prescribe annual reporting against departmental output measures that relate to agricultural RD&E.
• Key projects(thematic groupings of individual projects) require reporting on key project deliverables every six months or more, often to branch executive directors.
• Individual projects require reporting on the delivery of individual RD&E projects at every major milestone, every six months, and when the project is completed. The framework also highlights the need for reporting to external partners and co‑investors on project delivery where required under any agreements or contracts.

Evaluation arrangements

The department's MER framework for agricultural RD&E prescribes five types of evaluation:

• Strategy evaluations look at the four-year industry investment plans to make them clearer, more specific, better aligned and more relevant.
• Investment processes evaluations look at processes to allocate investments to RD&E projects, programs and initiatives.
• Impact evaluations look at the short-, medium- and long-term economic, social and environmental (net) benefits of an investment.
• Design and delivery evaluations look at the efficiency of project and program management and delivery to identify aspects that might be improved.
• Capability evaluations look at the areas of capability that might need to be established, increased, consolidated or reduced to adapt to changing priorities, scientific and technological advances, or evolving governance arrangements.

The framework sets out sound arrangements for planning, scheduling and using evaluations. Specifically:

• Evaluation plans are required for all key projects and other major programs (such as cooperative research centres). These plans set high-level evaluation questions that are supported by key performance indicators, data collection and analysis methodologies, as well as budgets, milestones, deliverables and time lines.
• Every RD&E project must be evaluated at least every five years, typically as part of a broader key project, program or thematic evaluation.
• The framework prescribes that the department's Agriculture and Resources Group maintains a rolling four-year schedule of evaluations that is reviewed every year at an evaluation conference. The conference also provides the opportunity to review responses to past evaluations and inform decisions about future investments.

Lack of MER focus on long-term sustainability of natural resource base

The department's high-level objectives and priorities for agriculture focus on the need for increased productivity to be accompanied by sustainable natural resource management. Although its performance measures, key performance indicators and targets have a strong productivity focus, they provide no assurance that changes in agricultural practices and productivity are not affecting the long-term sustainability of the natural resource base.

The department works in partnership with bodies that use performance measures that could be used to address this gap—for example, the Department of Environment, Land, Water & Planning and catchment management authorities measure and report on the condition of the state's natural resources.

4.3.2 The extent of MER activities

The department carries out a wide range of MER activities for its agricultural RD&E projects.

It has regularly monitored and reported on individual RD&E project deliverables, typically every six months and when project milestones have been met (including when projects are completed). The form and content of project reports vary depending on the extent of external project delivery roles, the type of research, development and/or extension being carried out, and the timing of the report.

The department has carried out or contributed to more than 100 RD&E evaluations in the past five years at varying levels and in the five MER framework categories. It has scheduled more than 30 evaluations between now and 2021.

The department has reported at a divisional level on its progress in meeting RD&E deliverables, including those under the 2012 Growing Food and Fibre initiative. The form and content of these reports have varied significantly over the past three years because of machinery-of-government changes.

The department also reports every year against 14 Budget Paper 3 measures that relate to agricultural RD&E. These are shown in Figure 4A.

Figure 4A

The department's Budget Paper 3 measures relating to agricultural RD&E

Source: Victorian Auditor-General's Office.

A number of new measures were added in 2014–15 and 2015–16 that improved the department's capacity to show the extent of its extension and practice change service delivery.

Analysis of the department's performance against its RD&E-related Budget Paper 3 measures since 2005–06 shows that it has met 80 out of 85 set targets. It did not meet its Budget Paper 3 target for:

• the percentage of research project milestones and reports completed on time in 2005–06, 2006–07 and 2008–09
• the number of applications for intellectual property protection in 2013–14
• the number of commercial technology licence agreements finalised in 2010–11 and 2012–13.

4.3.3 Aspects of MER framework not applied as intended

Although the department has actively monitored its projects, carried out wide-ranging evaluations and achieved most of its Budget Paper 3 targets, in some areas it has not stuck to its MER framework. These gaps limit the department's ability to show the benefits of its RD&E activities at an industry level and how results have been used.

The department does not directly monitor against the indicators in its four-year industry investment plans. These indicators are important because they outline the production needed for each industry to meet the government's objective in the Growing Food and Fibre initiative of doubling food and fibre production by 2030.

The department prepared key project evaluation plans for only four out of 14 key projects that were active during 2014–15. This gap is compounded by the absence of any direct performance MER against the four key project evaluation plans that had been prepared.

The department has held no annual evaluation conference since September 2012. This has weakened its ability to show that its evaluation schedules are well informed and that past evaluations have guided priority-setting, program development and continuous improvement. The department has recognised this and told us that it plans to restore evaluation conferences to its annual investment cycle.

4.3.4 Inadequate information systems to support MER

Weaknesses in the department's information systems are further limiting its capacity to effectively record, track and evaluate RD&E projects. Our 2008 performance audit Agricultural Research Investment, Monitoring and Review found that 'data collection systems do not readily support research investment program monitoring, evaluation and continuous improvement'. This weakness persists eight years later.

The department's Agriculture and Resources Group uses three information systems to support its RD&E project MER. These systems are not linked with each other or with department-wide systems. The systems also lack capability in key areas—for example, the department has only recently rectified its systems' inability to cost projects.

The department told us that much manual intervention is required to overcome these weaknesses. This increases the risk of unreliable MER activities and limits assurance that investments are being administered effectively and efficiently.

Previous plans to improve the former Department of Environment & Primary Industries' project management systems—which were intended to address the weakness at a department-wide scale—were not delivered after machinery-of-government changes on 1 January 2015. These system improvements are being delivered in the Department of Environment, Land, Water & Planning only.

In February 2016, the department's Agriculture and Resources Group assessed these weaknesses as high risk in its risk register. This highlights the need to prioritise improving its information systems for RD&E.

4.4 Impacts and benefits of RD&E

The department has a key role in measuring the impact of its RD&E activities. Its industry-level evaluations of RD&E are valuable and point to some key long-term improvements in the productivity and profitability of Victoria's agricultural industries that have been partly driven by RD&E investment. However, the rigour and relevance of these evaluations to RD&E vary. The department has room to improve how it prioritises, prepares for and coordinates these evaluations.

The department's RD&E activities will be a major contributing factor to Victoria's capacity to achieve the 2030 target. However, achieving this goal also depends on favourable economic and environmental conditions.

It remains unclear how achievable the 2030 target is four years after it was set.

4.4.1 Measuring impacts and benefits of RD&E activities

Measuring the impact of the department's RD&E activities is inherently complex for three reasons:

• Various external factors influence the performance of agricultural industries, including climate variability, drought and R&D outside Victoria that leads to changes in practice.
• The collaborative nature of agricultural RD&E means that state and federal governments, industry bodies and/or universities jointly fund many projects.
• Typically, RD&E activity takes place long before the corresponding benefits are realised—as shown in Figure 4B.

Figure 4B

Lag time for benefits of agricultural research and development to be realised

Source: Source: Alston et al. 2010, Persistence Pays: U.S. Agricultural Productivity Growth and the Benefits from Public R&D Spending, Springer, New York.

4.4.2 'Mega' evaluations—significant impacts but variable rigour and relevance

Since 2007, the department has either commissioned or carried out four industry-level 'mega' evaluations that have assessed the impact of RD&E activities in some capacity. These are summarised in Figure 4C.

Figure 4C

Industry-level 'mega' evaluations of RD&E

(a) Commissioned or carried out by the department.

(b) As the reviewers were not asked to test the conclusions and assumptions outside of the department, the results should be treated with caution.

Source: Victorian Auditor-General's Office.

These evaluations point to major long-term improvements in the productivity and profitability of Victoria's agricultural industries, with RD&E being a major contributor. However, the rigour and relevance of these evaluations to RD&E vary:

• The 2011 dairy mega evaluation provided the most reliable estimate of a minimum return of $3.30 for every dollar spent on R&D in the Australian dairy industry between 1980 and 2010.
• The 2015 grains mega evaluation estimated a return of $5.70 for every dollar spent on RD&E over 20 years, but this estimate was considered to be preliminary only.
• The 2015 soil mega evaluation did not conclude on the long-term outcomes achieved by soil RD&E, citing the review's limited scope and the lack of both a statewide strategy for soil RD&E and a monitoring and evaluation framework as reasons for this.
• The 2007 lamb mega evaluation estimated that the volume of lamb production had increased by 47 per cent over 25 years, but the evaluation's broad scope meant that it did not quantify the impact of RD&E. However, Meat and Livestock Australia (MLA) subsequently used this evaluation as an input to its economic assessment of lamb production RD&E investment between 1990−91 and 2007−08. This assessment reported that the combined investment of MLA, state governments and others over the 18-year review period had resulted in a cost−benefit ratio of 6:1.

The department notes that several factors make industry-level mega evaluations particularly difficult and complex to deliver:

• Certain industries are more difficult to evaluate than others—for example, the fragmented nature of Australia's horticulture sector makes it more difficult to evaluate compared to the dairy or sheep meat industries.
• The value proposition of a mega evaluation relies on the willingness of industry and other external partners to co-invest.
• The department might not be the most suitable body to carry out an industry-level evaluation, which typically looks at nationwide impacts. It told us that it is more likely to resource these evaluations where Victoria accounts for most of a specific industry, such as dairy.

The inconsistencies in past mega evaluations—and the complexities associated with the department carrying them out—reinforce the need for dedicated guidance material to support the selection, planning and delivery of industry-level evaluations of RD&E impacts. The department's existing MER guidelines and procedures regarding impact evaluations of RD&E activities are too general to meet this need.

Improving milk production through better pastures. Photo supplied by the Department of Economic Development, Jobs, Transport & Resources.

Figure 4D

Dairy 2011 mega evaluation case study: Impact of innovation on the dairy industry over the past 30 years

In 2011, the former Department of Primary Industries and the Geoffrey Gardiner Dairy Foundation jointly commissioned an evaluation of the impact of government and industry investment in pre‑farm-gate RD&E in the Australian dairy industry between 1980 and 2010. The evaluation estimated that: increases in on-farm production raised farmers' returns by $10 billion over the three decades in present-value terms almost half (46 per cent) of this increase is attributable to governments' and industry's combined investment in RD&E—about $2 billion over 30 years. This translates to an estimated economic benefit of $3.30 for every dollar invested. The graph below compares actual productivity growth over the 30-year period with a hypothetical scenario in which no government or industry intervention was made. Source: Centre for International Economics, 2011, The impact of innovation on the dairy industry over the last 30 years. The evaluation identified specific areas where RD&E had made major contributions, including increased pasture production and utilisation, better use of supplements and genetically improved cows that are more efficient at converting feed to milk. Other contributing factors included bigger farms, improved animal health and better management of natural resources.

4.4.3 Likelihood of doubling food and fibre production by 2030 unclear

In 2014, the department commissioned a study of the state government's commitment to achieve the 2012 Growing Food and Fibre initiative's target of doubling food and fibre production in Victoria by 2030. Importantly, the above modelling was performed two years after the 2030 target was set in 2012. This raises doubts as to whether the target was sufficiently evidence-based.

The objectives of this study were to:

• provide scenarios for how Victorian agriculture could increase production and profitability in support of the 2030 target
• quantify the benefits of each scenario for Victorian agriculture and the wider economy.

Three scenarios were developed to show the possible outcomes by 2030 across Victoria's four high-priority industries—dairy, meat, horticulture, and sheep and beef. The study did not compare the probability of these scenarios, which included:

• a 'low' scenario that reflects the recent experience of industry through the 2000s—significant climate variability, where farmers generally became more conservative in their attempts to maintain productivity and profitability by reducing inputs
• a 'medium' scenario that features a return to 'normal' market and seasonal conditions of the 1980s and 1990s and results in higher levels of productivity growth and uptake of existing technology
• a 'high' scenario where economic conditions are 'buoyant' as a result of strong export demands from Asia and the Middle East, leading to higher prices and production levels—the scenario also includes mild climate change impacts, shifts in land use to higher-value agricultural uses, widespread adoption of productivity improvements, and more intensified production through the use of inputs and adoption of known and yet-to-be developed technologies.

The study found that Victoria can increase its food and fibre production by more than 160 per cent by 2030—exceeding the target—'if the drivers and assumptions in the most favourable high scenario prevail'. It found that Victoria will not achieve the 2030 target under the low and medium scenarios:

• the medium scenario was estimated to result in a production gain of 64 per cent by 2030
• the low scenario was estimated to result in a decrease in production of 10 per cent by 2030.

These three scenarios are illustrated in Figure 4E.

Figure 4E

Modelled scenarios for potential agricultural productivity growth to 2030

Source: The Boston Consulting Group via the Department of Economic Development, Jobs, Transport & Resources.

The study found that RD&E would need to make a significant contribution if the 2030 target were to be met. It concluded that, of all the contributing drivers within the department's control:

• the largest potential contribution would come from the adoption of 'known new technologies'
• the second biggest contribution would come from the 'continued adoption of best practice management and intensification'.

This study reinforces the critical role that RD&E plays in enhancing food and fibre productivity in Victoria. It also highlights the favourable economic and environmental conditions needed to achieve the government's goals of doubling food and fibre production by 2030.

It remains unclear which of the above three scenarios is most likely to take place.

Recommendation

1. That the Department of Economic Development, Jobs, Transport & Resources improve its monitoring, evaluation and reporting framework and practices by:

• incorporating a documented approach to choosing, planning for and carrying out industry-level evaluations of the long-term impacts and benefits achieved by its research, development and extension investments
• addressing identified weaknesses in its information systems supporting research, development and extension project development, monitoring, evaluation and reporting
• developing or utilising external performance measures to provide assurance that changes in agricultural practices and productivity are not affecting the long-term sustainability of the natural resource base
• preparing evaluation plans for all key projects and actively using these to monitor, evaluate and report on industry-level performance.

RESPONSE provided by the Secretary, Department of Economic Development, Jobs, Transport & Resources

RESPONSE provided by the Secretary, Department of Premier & Cabinet