Department of Education and Training: Strategic Planning

Conclusions

DET has a good framework to guide its strategic and operational planning. It has produced a range of plans that detail what it intends to achieve and how it will achieve those aims. However, despite this planning framework being in place, the achievement of its outcomes is significantly undermined by ineffective department-wide governance and the poor implementation of its plans.

Longstanding and unaddressed governance weaknesses have resulted in systemic failures across the organisation. Changes in recent years have had little effect in improving governance, planning and implementation. Until governance weaknesses are addressed, DET is unlikely to make significant progress in achieving its outcomes. It assigned a 10-year time frame to achieve its objectives, but three years in there is little sign that DET is on track to achieve them. Considerable effort will be required to address the known weaknesses to ensure that DET is in the best position to achieve its goals and objectives.

Findings

Governance

DET's governance framework is deficient and consequently undermines the application of its planning framework. DET lacks strong leadership, has a poor culture of communication and does not work cohesively. This severely weakens DET's capacity to make informed strategic decisions, implement work plan actions successfully and monitor performance.

DET's reliance on committees makes it difficult to identify responsible executive officers and diminishes their accountability. It is impossible to know whether these committees are operating effectively as it is often unclear who oversees their performance, how decisions are made, who is attending, and how often they meet.

DET has known about these governance weaknesses since 2011, however, has only taken limited action to address them until recently. In 2015, DET commenced a thorough review of its governance, including committee structures and arrangements, leadership and culture. While it is too soon to examine the sufficiency and appropriateness of DET's actions, early indications do show promise.

In addition to weak governance, DET's oversight and assessment of executive officer performance is flawed. An examination of the performance and development plans of 100 of DET's executive officers and senior technical specialist officers found the comprehensiveness of these plans differs significantly across the department. There were also many instances of missing plans. Given this, it is not clear how DET ensures individual executive officers are being held accountable for delivering the work plan actions they are responsible for.

Strategic planning and implementation

DET's strategic planning framework is generally sound. It attempts to link strategic planning with resource management, operational planning, implementation, monitoring, reporting and evaluation. However, in practice it is unclear whether decisions about priorities, strategies and work plan actions are based on clear, sufficient and appropriate evidence, because there is a lack of documentation around these processes.

DET's implementation plans are often missing key elements of good implementation planning. DET does not have a consistent format for its plans—some are comprehensive and others are not.

DET also does not have a consistent project management framework to ensure its work plan actions are adequately implemented and overseen. Although DET reports quarterly to the executive board on all work plan actions, the information contained in these reports is not always reliable. This impedes DET's ability to adequately monitor whether its policies, programs and services are delivered on time, on budget and are achieving the desired outcomes.

DET's Integrated Resourcing Strategy seeks to bring together strategic planning, evaluation and department-wide consultation to inform future investment and funding recommendations. Despite having this strategy, there appears to be a limited focus on resources across DET to fulfil strategic objectives from year to year. DET's 2015 Review of Departmental Governance found that despite this framework being in place, the resources and budgets of divisional and group plans are only partially aligned.

DET has a thorough monitoring and review process for departmental priorities. However, while DET was able to demonstrate that it modifies its work plan and subsequent work plan action documents in response to changes in government commitments, it could not provide evidence that these are modified for any other reason.

Performance and evaluation

DET evaluates its identified strategies and programs to establish whether the objectives have been achieved. However, DET does not aggregate the results of these evaluations which means its overall performance is not assessed. Divisions and groups can, but are not required to, use a centrally coordinated evaluation panel. This is a missed opportunity for DET to gather and evaluate crucial evidence about its performance.

DET's Education and Training Outcomes Framework (Outcomes Framework) includes a group of outcomes and measures that give information on DET's performance. While the Outcomes Framework contains the information needed to inform DET's strategic planning, there are no specific targets included. The Outcomes Framework would provide greater accountability if it included specific targets against which performance could be assessed.

DET's external targets as stated in Budget Paper No. 3 Service Delivery (BP3 measures) are weak in parts. For 2014–15, DET had 86 BP3 measures covering six outputs, which account for $12.065 billion in funding. DET had the highest budgetary output-to-measure ratio of any government organisation—one measure for every $140 million of output, compared to a Victorian average of one measure for every $40 million. Having such a high ratio reduces the level of scrutiny of DET's performance.

In 2014, a DET-commissioned review of its BP3 measures identified measurement gaps and recommended better targets, additional measures across most output areas and the removal of outdated and redundant measures. While DET has worked to improve its BP3 measures since the review, most of the review's recommendations have not been addressed.

Recommendations

That the Department of Education & Training:

1. reduces or streamlines its committee structure, assessing the number, role and performance of each committee
2. modifies the executive officer performance and development process to ensure sufficient performance oversight, including the development of an underperformance policy
3. introduces greater transparency around decision-making processes into its corporate management framework, including better documenting new project proposals and resource allocation
4. introduces a project management framework that reflects better practice and includes guidelines and templates for project management, implementation, monitoring and oversight
5. develops targets for the Education and Training Outcomes Framework, where applicable, so that progress and performance can be properly assessed
6. establishes a central repository for strategic program evaluations and applies the lessons learned from evaluations to current and future programs.

Submissions and comments received

We have professionally engaged with the Department of Education & Training throughout the course of the audit. In accordance with section 16(3) of the Audit Act 1994 we provided a copy of this report to the Department of Education & Training and requested their submissions or comments.

We have considered those views in reaching our audit conclusions and have represented them to the extent relevant and warranted. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Department of Education & Training performance

1.1.1 Long-term outcome measures

DET's 2014–18 Strategic Plan shows that its performance for 15 of its 26 (57.7 per cent) long-term outcome indicators have either deteriorated or shown no significant change over recent years. This is consistent across all four of DET's long-term outcome areas of achievement, engagement, wellbeing and productivity, and across both government and non-government schools.

Figure 1A

Trends in Department of Education & Training outcomes indicators

(a) These indicators refer to government and non-government schools.

(b) This indicator refers to government schools for the national assessments and both government and non-government schools for the international assessments.

(c) These indicators refer to government schools.

(d) In the future, the term 'or equivalent' will be removed from this indicator and replaced with Year 12 or equivalent vocational qualification completion rates of young people.

# The trend for this indicator is down for women, marginally up for men; down for all age groups.

Source: Department of Education & Training.

Victorian governments have continually emphasised a devolved education governance model. This model requires DET to play a strong central role to ensure that sufficient support, guidance and oversight of education providers is in place.

We analysed 27 of VAGO's audits from 2009 to 2014 related to DET. The analysis identified five areas in which DET has continually underperformed:

• Information management—DET does not consistently collect and analyse data to determine which areas require performance improvement.
• Guidelines, standards and performance indicators—governance arrangements, guidelines, standards and performance indicators are not always clearly articulated to service providers (schools, teachers, principals, or contractors).
• Integrated services—the absence of an overarching framework to support and govern particular program and policy areas has led to less effective service provision. There is a lack of streamlining and linking-up of services, programs and objectives across service providers. Developing new programs that overlap and mismatch with programs already in place can cause confusion, and programs do not operate as intended.
• Oversight and monitoring—DET has not consistently held education providers to account for poor performance. It does not adequately measure the effectiveness of its own programs, or monitor the performance of service providers to whom it delegates responsibilities.
• Consistent support to education providers—DET's standard of advice and guidance to principals and teachers is inconsistent.

Figure 1B shows how prevalent these themes were across the 27 audits.

Figure 1B

Percentage of audits in which each identified theme was prevalent

Source: Victorian Auditor-General's Office.

Taken together, DET's performance against its own outcomes measures and VAGO's audits indicate that its strategic planning and work plan action implementation is not as effective as it could be in driving improved outcomes.

1.2 Audit objective and scope

The audit objective was to examine how effectively DET plans to achieve its objectives. The audit assessed whether:

• an effective planning framework supports the development of strategic and operational plans
• plans are effectively implemented
• planning is effectively monitored to assess the achievement of outcomes and objectives.

The focus of the audit was on DET's strategic planning, and the operational plans and other work plan actions that support this. This includes business plans within various DET business units. The strategic planning of DET's portfolio agencies, schools, early childhood education providers, and vocational education training providers was not directly examined, but considered in terms of how DET uses these plans to inform its own strategic planning.

1.3 Audit method and cost

The audit involved desktop research, reviews of documents, and interviews with DET staff.

The audit was conducted in accordance with the Australian Auditing and Assurance Standards. Pursuant to section 20(3) of the Audit Act 1994, unless otherwise indicated any persons named in this report are not the subject of adverse comment or opinion.

The cost of the audit was $315 000.

1.4 Structure of the report

The report is structured as follows:

• Part 2 examines the adequacy of DET's governance framework
• Part 3 examines the sufficiency of DET's planning framework and implementation of work plan actions
• Part 4 assesses the appropriateness of DET's performance and evaluation framework.

2.1 Introduction

Good governance is a cornerstone of effective planning. Governance establishes the processes through which an agency directs, controls and holds itself to account. Good governance also enables an agency to make the right decisions, achieve value for money and recognise and adjust to emerging risks.

2.2 Conclusion

The Department of Education & Training's (DET) planning is not supported by effective governance. It lacks strong leadership, has a poor culture of communication and is siloed. These deficiencies weaken DET's capacity to make informed strategic decisions, implement work plan actions successfully and monitor performance. Ultimately, this impedes DET's ability to effectively and efficiently deliver essential educational services to the Victorian community.

2.3 Governance and planning

To achieve good governance, an agency must have strong leadership, a culture of communication and must work cohesively. It must also have clear accountability mechanisms, sound risk management and compliance systems, and be able to plan effectively and evaluate its performance.

DET is governed by its executive board (EB), which is made up of the Secretary, six deputy secretaries and the Chief Executive Officer of the Victorian Curriculum Assessment Authority. The EB is accountable for:

• strategic direction and leadership
• management
• decision-making
• risk management
• monitoring and evaluating the department's activities
• compliance
• stakeholder management.

The EB is also accountable for ensuring that a strategic plan is developed and endorsed, and for ensuring the plan is implemented and monitored.

As Figure 2A shows, DET is a large department. It is responsible for covering all facets of education from early childhood through to adult learning. Consequently, it is critical that its governance works cohesively in order to achieve DET's collective goals.

Figure 2A

Department of Education & Training organisational structure as at 1 May 2015

Source: Department of Education & Training.

However, DET's governance structure does not easily align with its organisational structure. Figure 2B outlines its governance structure.

Figure 2B

Department of Education & Training governance structure as at 25 May 2015

Source: Department of Education & Training.

DET's governance framework is deficient and does not adequately support its planning. There are silos across DET, weaknesses with its committee structures and its executive leadership does not sufficiently demonstrate the behaviours needed to support DET's objectives.

The number of committees and subcommittees DET has, and whether they support planning, is not widely understood across the department. There is no central repository for this type of information, and knowledge is dispersed across the department. This issue is partly caused by DET's failure to formally dissolve committees when they are no longer required and document dissolution.

These deficiencies severely weaken DET's capacity to make informed strategic decisions, implement work plan actions successfully and monitor performance.

2.3.1 DET's committees do not effectively support planning and governance

Committees are an efficient way to bring expertise together for the purpose of making decisions, providing advice and monitoring and evaluating performance. When run well, they pool knowledge, improve communication and break down organisational silos.

Although DET is partly structured around its functions, with responsible executive officers accountable for their own areas, the structure also contains standing committees and subcommittees. These committees should be an essential part of DET's governance framework, bringing together knowledge across program areas. However, there are significant weaknesses with DET's committee framework.

In 2015, DET commissioned a Review of Departmental Governance (2015 governance review) which is underway but not complete. Consistent with this audit's findings, it identified key weaknesses with DET's governance committee structures and arrangements—lack of role clarity, poor understanding of mandate, poor communication, ineffectiveness, and inconsistent governance processes.

DET's standing committees have unclear roles and limited accountability

DET's EB is supported by 10 standing committees each responsible for oversight, leadership and planning for specific objectives. Figure 2C outlines the responsibilities of these standing committees.

Figure 2C

Committees supporting the Department of Education & Training Executive Board

Source: Victorian Auditor-General's Office.

Standing committees have decision-making powers and are therefore accountable for the consequences of those decisions. However, given decision-making is devolved across committee members it is difficult to determine individual responsibilities. This diminishes the accountability of responsible executive officers.

The functions and remit of standing committees are unclear. There is also limited shared understanding of which matters should be taken to a standing committee, and which committee should consider them.

A lack of clarity around when a committee, or which committee, should receive work impedes the effective development of major policies and projects across DET. The 2015 governance review's survey of DET staff found that 87 per cent of respondents strongly disagree, disagree or were unsure whether relationships between governance committees are clear. It also identified:

• overlap in the remit of standing committees in the areas of policy and strategy, service delivery, evaluation and procurement
• confusion around the role of the Early Childhood and School Reform Board and how it interacts with the EB.

To work effectively and efficiently, DET needs clarity around the roles of its standing committees. Each committee's terms of reference should be clear and roles and responsibilities between committees should be unambiguous.

Some standing committees have the power to make decisions while others provide an advisory role. Given this difference, it is important that each standing committee's mandate is clear.

DET's standing committees' terms of reference do not specify whether the role of committees is to make a decision as a delegate of the EB, or to provide advice to the EB. This lack of clarity contributes to uncertainty about whether committees' decisions are binding or subject to approval by the EB.

There is also confusion among committee members about their role. The 2015 governance review found that members of the People and Culture Standing Committee held differing views on whether it could make decisions. The 2015 governance review also found that regular reviews of the specific purpose, role, capability and membership of the committees are not consistently performed and that committee terms of reference do not consistently include behavioural rules, induction procedures and conflict of interest clauses.

Key governance and planning committees are not effective

The key committees for planning are the PPSC and the SPDSC.

PPSC is made up of nine representatives including four deputy secretaries and the Chief Executive Officer of the Victorian Curriculum and Assessment Authority. PPSC has oversight and approval responsibilities for:

• major portfolio-wide strategy and sector-specific policy statements
• research and evaluation strategy
• performance monitoring and reporting
• significant policy, strategy and performance issues, risks and developments
• minutes and recommendations of relevant subcommittees and working groups.

SPDSC is made up of 11 members including five deputy secretaries and five executive directors.

However, it is impossible to know whether these committees are operating effectively, because their performance is not evaluated, their decision-making rationale is not clear and minutes of meetings are not disseminated across DET's groups. These findings are replicated in DET's 2015 governance review.

Decisions are poorly communicated

Decisions made by DET committees are not clearly and consistently communicated and are often not acted upon or implemented. This inconsistency has affected staff efficiency and effectiveness and has reduced the capacity to hold staff accountable for implementation. More specifically, the 2015 governance review found:

• communication channels for recommendations, actions or decisions following committee meetings are inconsistent and not well understood by staff
• stakeholders are sometimes unclear about what decisions and actions follow from committee meetings
• EB decisions are not consistently referred to or communicated back to standing committees.

The 2015 governance review recommended DET introduce a consistent and disciplined approach to the communication of decisions made by the EB to executives, by enhancing processes to distribute and discuss minutes.

In August 2015 DET approved new governance arrangements including protocols for the consistent communication of decisions made by the EB and standing committees.

Governance issues are longstanding and unaddressed

Issues around DET's governance are not new. In 2011, DET commissioned a review of its governance framework (the 2011 review) which specifically looked at the effectiveness of DET's governance and accountability arrangements, and identified key elements of good governance. However the review's recommendations were not adequately addressed by DET and many of the issues remain, and were identified again by the 2015 governance review.

The 2011 review found significant gaps in key governance processes that contributed to DETs challenges in delivering responsive, efficient and integrated services. It also found that there was:

• siloed decision-making
• ineffective use of evidence
• avoidance of making hard decisions
• a lack of understanding about how DET is governed
• a perception that the governance committee structure was ineffective.

The 2011 review concluded that DET's senior leadership group needed to strengthen its decision-making and prioritisation of key activities. The 2011 review recommended that DET needed:

• unified commitment to more structured collaboration
• effective planning, performance monitoring and evaluation
• greater clarity between central office, regional offices and education providers
• accountability for leaders and staff to be communicated and enforced
• better decision-making and prioritisation
• clarification around the roles and responsibilities of governance committees
• an EB—to be reviewed after one year
• to establish four advisory committees
• an improved culture, using performance management processes so that leaders modelled good governance and collaborative behaviours
• reporting guidelines for the EB
• role clarity for individual positions
• to initiate a capability framework.

Despite identifying significant weakness in its governance model, DET did little to address the review's recommendations. DET introduced the four advisory committees, but none of the remaining 11 recommendations were addressed.

DET's EB was told of the 2011 review's findings and it endorsed the establishment of a project team to implement the recommendations. However, due to staff changes and the Sustainable Government Initiative—a government policy to reduce the public service workforce by 3 600—these recommendations were not implemented.

The 2011 review's findings raised serious issues around DET's ability to perform its functions. There should have been a standing item on the EB meeting agenda until all the recommendations were addressed, and the responsible deputy secretaries should have been held to account. This failure to address the reviews findings is a consequence of weak governance and the absence of accountability in leadership.

DET is now acting to address governance weaknesses

DET's EB approved a new governance committee structure, including roles, responsibilities, and supporting processes in August 2015. However, given the review is still underway, it is not possible to assess the effectiveness of its recommended improvement actions. VAGO will revisit this issue as part of its follow-up audit program.

The 2015 governance review classified DET's governance committee structures and arrangements as 'developing' in May 2015. This was on a scale of weak, developing, integrated, mature and advanced. It listed governance committee structures and arrangements as a key priority to address in the immediate future.

The 2015 review recommended that DET:

• abolish redundant committees
• develop and communicate refreshed committee charters, including for the EB
• develop a register of all departmental committees that includes terms of reference and membership
• induct all committee members by clearly identifying the purpose and role of the committee and of individual members
• provide coaching for committee chairs.

The second stage of the 2015 governance review will focus on designing improvements to committee structures, and developing or refining supports to enable DET to deliver its new Education State vision. Getting this right will be critical for DET to achieve its desired outcomes.

Based on consultation with the EB, the 2015 governance review lists the end of 2017 as the target for reaching a 'mature' rating for DET governance committee structures and arrangements. To reach this target, the 2015 governance review proposes improvements to the overarching governance arrangements that include the establishment of committees, including roles and term of reference. VAGO will assess DET's progress on this issue as part of its follow-up audit program.

2.3.2 DET's executive leadership lacks accountability

At the time of this report, DET's leadership team consists of a Secretary and 77 executive officers. The combined salary of these officers is $16.8 million.

Each officer's terms of employment are outlined in a standard contract of employment which requires that each executive officer be subjected to two formal performance assessments a year.

These biannual assessments are carried out through performance and development plans (PDP).Through the PDP process, each executive officer is required to detail the actions they are responsible for achieving, including work plan actions. This element of the strategic planning framework is crucial for DET to ensure that its work plan actions are implemented, are done so efficiently and effectively, and that someone is accountable for the success or failure of each action.

However, DET's oversight and assessment of executive officer performance is deficient. An examination of all DET's executive officer PDPs for 2013–14 showed that just four of 73 PDPs were fully completed. Despite this, 70 per cent of executive officers received a performance bonus. Furthermore, 47 executive officers received bonuses totalling $458 412, or an average of $9 753.4 each, despite not having completed performance management assessments.

Figure 2D highlights the outcomes of the performance management process, including assessment against the existence of the PDPs. It shows that 17 executive officers received a bonus for 2013–14, despite DET being unable to locate their performance plans. It is unclear how those assessments were made.

Ultimately, the absence of PDPs, but positive assessments of performance, is a failing of DET's systems and processes, and also of its senior leadership. DET is accountable for ensuring its staff have clear expectations around performance and they are appropriately assessed against those expectations.

Figure 2D

Executive officer performance and development plans for 2013–14

Source: Victorian Auditor-General's Office.

Figure 2E breaks this data down by level of seniority, with Executive Officer 1 being the most senior in the public service.

Figure 2E

Executive officer levels and performance pay for 2013–14

Source: Victorian Auditor-General's Office.

DET has no underperformance policy for executive officers, instead this is dealt with on a case-by-case basis. This is a significant gap in its performance management framework and can lead to inconsistent approaches to the underperformance of its highest paid staff. DET has such policies for teachers and Victorian public service staff which could be easily adopted for executive officers, bringing accountability to an acceptable level.

Recommendations

That the Department of Education & Training:

1. reduces or streamlines its committee structure, assessing the number, role and performance of each committee
2. modifies the executive officer performance and development process to ensure sufficient performance oversight, including the development of an underperformance policy.

3.1 Introduction

A strategic planning framework is integral to an agency's ability to make sound business decisions. It is the mechanism through which an agency identifies priorities, develops work plan actions and allocates resources, responsibilities and accountabilities. An effective planning framework enables the effective implementation of work plan actions and evaluation of their impact.

3.2 Conclusion

The Department of Education & Training's (DET) strategic planning framework is generally sound. It links strategic planning with resource management, operational planning, implementation, monitoring, reporting and evaluation. However, weaknesses in its application have resulted in decreased accountability.

Implementation lacks key elements of better practice such as monitoring, clear time frames and accountabilities. This causes unreasonable delays, increased costs and failure to deliver products and services.

3.3 Application of the planning framework is weak

Key strategic and operational plans need to outline an agency's corporate objectives and set goals that will achieve those objectives. In order to achieve corporate objectives, it is important that there is a clear line of sight from the high-level corporate plan down to the operational plans.

DET's documented planning framework brings together key decision-makers for the purpose of identifying objectives and strategic priorities. It then requires the production of cascading plans, starting at the strategic, department-wide level and ending with accountability for operational plans, managed through executive performance and development plans.

The framework broadly adheres to better practice, however, there are gaps in its application including a lack of transparency in decision-making, poor resource management and missing business plans.

Figure 3A shows DET's various plans, and their relationships.

Figure 3A

Department of Education & Training plans

Source: Department of Education & Training.

DET's corporate plan consists of two parts—the public strategic plan and the 'internal‑to‑government' resource plan.

3.3.1 DET's strategic plan is sound but lacks clear time frames for achieving outcomes

DET's strategic plan is its key long-term plan, outlining its overarching goal, outcomes and indicators over both a four-year and 10-year period. The strategic plan is designed to set directions for DET's groups, divisions and regions. DET's 2014–18 Strategic Plan sets out the following priorities, strategies and outcomes.

Figure 3B

Department of Education & Training 2014–18 Strategic Plan

Source: Department of Education & Training.

The strategic plan is sound and reflects better practice. From an accountability perspective, the plan's key weakness is that it does not clearly articulate due dates for the achievement of its 10-year goal and outcomes. DET advised that the 10‑year period started in 2012 with the publication of the strategic plan. However, this is not evident from reading the plan, and DET should improve its transparency so that Parliament and the community know when they should expect to see the intended improvements. Based on DET's advice, it is over one-third of the way to achieving its goal, although what progress should have occurred by this stage is unclear.

3.3.2 DET's annual work plan provides clear information on reform priorities

DET's annual work plan outlines the key reform actions the department will introduce in the short-term to deliver on the priorities and strategies outlined in the strategic plan. It is updated annually, and alongside the strategic plan it guides the annual business planning process for DET's groups, divisions and regions.

The current work plan clearly lists actions and allocates responsibilities. It also aligns with the strategies listed in the strategic plan. However, the plan does not consider the available resources against the resources, so it is unclear how DET ensures its groups, divisions and regions have the capacity to implement all the actions they have been allocated.

3.3.3 Group plans provide the necessary information to guide work over the medium term

Group plans are four-year plans that are completed by each of DET's six groups on an annual basis. They are intended to implement the four-year priorities and 12-month reform actions in the strategic plan, and to provide direction for regional and divisional plans.

DET's group plans are adequate, in that they generally align with the overarching priorities listed in DET's work plan. However, not all group plans clearly allocate resourcing to reform actions, and in the absence of other routine resource planning, this is a weakness.

3.3.4 Divisional and regional plans are not consistently developed

These plans are annual plans completed by each of DET's divisions and regions. They are intended to align with group plans, and to detail projects and activities, including both reform and core business activities.

The comprehensiveness of business plans differs across the department. Not only was DET missing over 6 per cent of its business plans for 2010–11 to 2014–15, but there were often key elements missing from those that were developed, such as resource allocation and analysis of implementation risks.

DET has improved its development of divisional and regional plans in recent years. However, it is unclear how divisions and regions were able to effectively deliver programs and services in the absence of plans detailing what needed to be done and by when.

3.4 Supporting frameworks are not effectively used

In addition to its planning framework, DET has supporting frameworks. Broadly, these are intended to action DET's planning framework by setting the direction for planning activities and outlining key milestones. The Corporate Management Framework is the mechanism for developing and executing work plan actions across DET. It attempts to link strategic planning with resource management, operational planning, implementation, monitoring, reporting and evaluation. It does this through:

• setting directions—sets the long-term departmental direction
• planning—involves developing business and implementation plans that outline how strategic directions will be delivered
• monitoring—assesses risks, issues and challenges, and monitors the budget and progress in meeting outcomes using performance indicators
• evaluation—focuses on evaluating performance, policies and programs to determine what changes are required; informs planning and resource allocation
• governance and communication—structures and processes for decision-making, accountability, control and behaviour at the top of organisations, and communicating strategic directions.

However, the Corporate Management Framework is not routinely used as part of DET's planning process, particularly in the areas of planning, monitoring, evaluation and governance—for example, the annual investment conference has not been held since 2013.

DET's annual Integrated Corporate Management Calendar (ICMC) provides an outline of the time frames in which key planning, evaluation and reporting tasks should be completed. It includes actions and milestones relating to high-level direction setting, planning, monitoring and evaluation.

ICMC is governed and operated by a steering group with 13 representatives, including eight executive directors and two directors. The steering group's role is to:

• consider the outcomes arising from key planning events and allocate work plan actions as required
• provide updates on where different areas responsible for developing plans are in the planning cycle
• provide updates to the executive board (EB) on any work plan actions progressing, evaluations occurring, and activities that are upcoming.

ICMC is sound in theory, however, in practice there are weaknesses in its application that have led to a lack of transparency around how strategic priorities are determined, as well as inconsistent implementation processes and a poor evaluation framework.

3.5 DET's planning activities are evidence-based

DET's strategic planning activities use appropriate evidence and research, including both internal and external inputs. While these planning activities are appropriate, it is difficult to examine whether the information relied upon to make decisions about priorities is sufficient.

DET's Strategic Policy and Planning Division leads departmental strategic planning, and monitoring and reviews processes. It develops DET's strategic priorities in consultation with the senior executive team through planning the following events:

• Annual retreat for its EB—held at the start of the calendar year. At this event the EB reflects on performance data, current issues and drivers that may affect priorities and strategies for the next 12 months. The EB is also required to confirm DET's priorities and strategies for the next 12 months.
• Annual planning conference—held mid-year for DET's senior leadership team, participants reflect on performance and determine actions for the year ahead. Outcomes from this event form inputs into strategic and work plans.

Groups, divisions and regions may also hold planning discussions prior to leadership planning events to develop ideas to input.

To help inform decision-making across the department, DET's Performance and Evaluation Division (PED) collects, analyses and disseminates evidence. PED developed DET's Education and Training Outcomes Framework across 2014 and 2015.

Work plan actions are primarily developed at the EB. A range of performance data is provided to inform this event. In relation to proposed new actions, information from divisions or regions and from Deputy Secretaries is provided verbally. This precludes any examination of whether decisions around new actions are based on sufficient and appropriate evidence.

DET's current planning processes are a significant improvement on previous practices. Regardless, there is still much to do to ensure that planning processes, particularly the recording of decisions, reaches an acceptable standard and achieves transparency.

3.6 Resource management is not effective

Resource management is the efficient and effective deployment and allocation of all types of resources when and where they are needed. DET's current approach to resource planning is piecemeal, inadequate and lacks transparency.

3.6.1 Internal-to-Government Resource Plan

DET's Internal-to-Government Resource Plan is its key resource plan, and is produced as part of DET's annual corporate plan. It is provided to the Department of Treasury and Finance and outlines DET's asset and workforce needs, financial outlook, risks and challenges and policy reform proposals aimed at improving the delivery of outputs.

While this plan serves its broad purpose in assisting the Department of Treasury and Finance with statewide service demand and asset planning, it does not outline resource needs in enough detail. Resource allocation is considered only across DET's three key learning areas—early childhood development, school education, and higher education and skills. While there is a need for resource planning at that level, there is also a need for documented and centralised resource planning at a group and divisional level. This does not routinely occur.

3.6.2 Integrated Resourcing Strategy

DET's Integrated Resourcing Strategy, implemented in 2011, seeks to bring together strategic planning, evaluation and department-wide consultation to inform future investment and funding recommendations. Its aim is to maximise the utilisation of resources to achieve its objectives.

However, the Integrated Resourcing Strategy is not being adhered to by DET. A key component of this strategy is an annual investment conference through which DET develops proposed funding bids for the annual State Budget processes. However, this conference has not been held since 2013 and it therefore cannot be considered an integral part of DET's resource planning. DET's Review of Departmental Governance (2015 governance review) found that there is inadequate alignment of funding with resources:

• The resources and budgets of divisional and group plans are only partially aligned, with salary and operating funding differing between the two.
• In practice there appears to be a limited focus and limited resources across DET for fulfilling strategic objectives from year to year.

DET's 2015 governance review also found that there are strong silos across DET, limiting resource sharing and ultimately effecting service delivery. It found that cross-departmental collaboration, particularly to inform evidence-based policy and service delivery reform, needs to be stronger and more effective.

3.7 Program implementation is not effective

Strategic implementation is the process through which strategies and plans are turned into actions and requires effective planning, monitoring and evaluation. DET's implementation framework is weak, which has resulted in inconsistent practices, poor oversight and delays.

3.7.1 Implementation plans lack the necessary information

Effective implementation requires planning that considers all the elements essential to the successful delivery of strategies and actions, and enables an agency to monitor whether its policies, programs and services are delivered on time, on budget and are achieving the desired outcomes. Key elements to consider include:

• responsibilities
• accountabilities
• time frames
• priorities
• resourcing
• implementation risks
• implementation monitoring
• alignment with relevant policy.

DET's implementation plans are often missing key elements of good implementation planning. In some cases these elements were included in separate documents, however, this compromises their usability, including the ability to implement actions and monitor progress.

As a result, these actions face a greater chance of being impacted by unforeseen risks, missed opportunities, and insufficient resourcing. A lack of implementation monitoring, clear time frames and resources causes unreasonable delays, increased costs, and failure to deliver products and services.

Implementation plans varied unnecessarily

The audit examined 12 actions from DET's 2014–15 work plan and found that DET generally develops implementation plans for its actions, although the comprehensiveness of these plans differs. There is no consistent format for these plans, and the level of detail varies.

Some plans are comprehensive and others are not, which creates risks for their successful implementation. These include risks around missed opportunities, delays, increased costs, and the failure to deliver products and services. The 2015 governance review identified similar issues, noting that when it comes to the transition from policy development to implementation, DET needs to provide clearer direction on objectives and time frames, and needs logical follow-up plans and appropriate tools and templates for measuring success.

Of the 12 work plan actions assessed:

• two used a project charter
• one used a project charter alongside action summaries consisting of goals, actions, resourcing, time lines and allocation of responsibilities
• one used a project charter alongside a number of other documents:
  • a spreadsheet with goals, strategies and reform actions
  • a spreadsheet of outcomes, performance measures and actions
  • a risk register
  • a table outlining resourcing, action outline and key messages
• one used an implementation plan
• one used a project plan and communications strategy
• two used spreadsheets
• one used separate documents consisting of a program description, resourcing information, time lines and milestones, an evaluation strategy, an exit strategy and key performance indicators
• one used a Gantt chart progress report, project update form and work in progress document
• one used an implementation schedule, communications plan, assessment project plan, rolling actions list, roll out schedule, calendar of events, table of coaching sessions, program overview, project showcase agenda and module workshop time line.

Some plans contain all key elements of good implementation planning and showed evidence of review and modification in response to changing circumstances. On the other hand, one work plan action did not have a plan at all.

Given the disparity in practices, it is clear that DET lacks thorough planning and implementation processes. In order for DET's programs and initiatives to meet their desired outcomes, DET needs to implement a clear and comprehensive project management framework. This framework should require implementation plans for each work plan action that clearly identify responsibilities and accountabilities and incorporate goals, time frames, resourcing, implementation risk management plans, strategies for coordination across agencies and sectors, and include processes for regular reviews and updates, as required.

3.7.2 Modifying plans occurs inconsistently

Monitoring and reviewing the implementation of work plan actions is critical for their effective, efficient and economical delivery. It requires robust management and oversight of progress, and remedial action where issues such as delays and budgetary blow-outs are identified.

Implementation plans were modified in response to identified risks at the operational level, however, at the strategic level there was little evidence that modifications were made for reasons other than responding to changes in government commitments. Where changes did occur, these were not typically reflected in modifications to group, divisional or regional plans. Not modifying all plans following a change in policy, or due to other factors, has the potential to diminish the effectiveness of any actions underway. This impacts the efficient and economical use of resources.

3.7.3 Committee oversight of work plan actions is inadequate

DET oversees work plan actions in two ways—oversight committees and a quarterly status report provided to DET's EB.

DET allocates an oversight committee to each action listed in its work plan. The committee is required to oversee progress of the action, including considering and endorsing project planning documentation. However, the frequency of oversight committee meetings differs across work plan actions and the amount and type of information provided to these committees varies. For instance, in 2014–15, 65 work plan actions were reported to the EB as having sufficient and appropriate planning documentation in place. However, 35 of these (53 per cent) had never had these documents considered by the relevant oversight committee.

DET's quarterly status report provides its EB with a detailed overview of the progress of each work plan action. The quarterly reports show that 10 out of 40 (25 per cent) of DET's non-election commitment work plan actions for 2014–15 had not been considered at a meeting of the relevant oversight committee at any point in 2014–15. Of these 10:

• two were reported to be complete
• three were reported to be between 51–75 per cent complete
• two were reported to be between 26–50 per cent complete.

VAGO has also identified instances where data reported by the project team in the quarterly report was incorrect. This brings into question the reliability of some of the information provided by program areas to inform the preparation of these reports. For example, the 2014–15 work plan action that had no implementation plan in place was reported in the June 2015 report to EB as having sufficient and appropriate planning documentation in place. It was also reported to have had this planning documentation considered and endorsed by the relevant oversight committee in March 2015. This demonstrates a fault in reporting.

3.7.4 Projects are frequently delayed

The completion of DET work plan actions is frequently delayed. Of the 26 actions in the 2013–14 work plan that were due to be completed by June 2014:

• nine (34.6 per cent) were completed on time
• 17 (65.4 per cent) were not completed on time.

Between July 2013 and June 2015 on average one in five work plan actions (19.4 per cent) were behind schedule at some point. The worst performance was in the January to March 2014 quarter when 32.7 per cent of actions in the 2013–14 work plan were behind schedule.

Recommendations

That the Department of Education & Training:

3. introduces greater transparency around decision-making processes into its corporate management framework, including better documenting new project proposals and resource allocation
4. introduces a project management framework that reflects better practice and includes guidelines and templates for project management, implementation, monitoring and oversight.

4.1 Introduction

Monitoring and evaluating performance is a key element of effective planning. It tells an agency whether it has used its resources effectively, efficiently and economically and whether intended objectives have been achieved. Effective monitoring and reporting also strengthens an agency's ability to identify and address underperformance.

4.2 Conclusion

The Department of Education & Training's (DET) evaluation governance structure is unnecessarily complex, repetitious and incoherent in parts. Overlapping roles and responsibilities contribute to this confusion and diffuse accountability.

While its Education and Training Outcomes Framework (Outcomes Framework) provides sound performance measures, it has no targets. DET's Budget Paper No. 3 Service Delivery (BP3) external performance targets are weak. A lack of internal targets and weak external targets impedes DET's ability to assess whether its performance is effective, efficient and economical. It also inhibits external scrutiny as to whether DET is achieving its funded objectives. Ultimately, these weaknesses impede DET's ability to plan to meet its strategic priorities and efficiently allocate resources.

4.3 DET's evaluation governance structure is not effective

An evaluation governance structure sets out the processes through which judgements can be made about the effectiveness, efficiency and appropriateness of an agency's activities. This can include guidance on oversight models, data sources and reporting processes. Evaluation provides important insight into how an agency is performing, whether it is achieving its objectives and whether those objectives are still appropriate.

DET has an evaluation policy that sets out its protocols and approach to evaluation, which are designed to improve outcomes. The evaluation governance structure incorporates the Victorian Government's evaluation principles, which include:

• evaluations shall be fit for purpose
• evaluations will meet minimum standards and be subject to appropriate quality assurance
• the value of evaluations is in their use.

The policy also includes governance arrangements—committee-based oversight, and roles and responsibilities.

However, there are weaknesses in the structure including convoluted governance, lax committee oversight and poor central coordination of evaluation outcomes. Together, these undermine its effectiveness.

4.3.1 Evaluation governance arrangements are convoluted

DET's evaluation governance arrangements are complex. There is an overlap of roles and convoluted processes. This has resulted in no-one bringing together all the evaluations and using them to improve performance.

Figure 4A outlines DET's complex evaluation governance structure.

Figure 4A

Department of Education & Training evaluation governance structure

Source: Victorian Auditor-General's Office.

There are weaknesses with the evaluation governance:

• Both the Evaluation Board and the Performance and Evaluation Division (PED) are responsible for communicating evaluation findings. However, it is unclear whether both do it, whether it is coordinated, whether it occurs routinely and whether it is intended to occur for all evaluations or just those on the evaluation schedule.
• There are multiple bodies responsible for identifying evaluation activities. The Policy and Performance Standing Committee (PPSC) is responsible for determining strategic and lapsing program evaluation activities. Groups and divisions are responsible for identifying evaluations through their planning activities. However, it is not clear how DET assures itself that this process identifies the most essential programs for evaluation.
• There is a lack of continuity and coordination for how DET deals with its evaluation activities. For example, its evaluation policy is dealt with as follows:
  • PPSC—responsible for the policy
  • Evaluation Board—responsible for implementing the policy
  • PED—facilitate the sharing and incorporation of evaluation findings into policy
  • Strategic Evaluation and Research—maintains and reviews the policy
  • PED authorised the latest version of the evaluation policy in September 2013.

Figure 4B outlines the roles and responsibilities of DET's evaluation bodies.

Figure 4B

Evaluation governance roles and responsibilities

Source: Victorian Auditor-General's Office.

4.3.2 Evaluations are undertaken as planned

Understanding how well specific programs and activities are performing is an important part of an effective performance framework. Proper evaluation of programs provides a qualitative perspective on performance.

DET's program of evaluation involves assessing significant strategies and lapsing programs as well as group and divisional programs to establish whether objectives have been achieved. A program of evaluation is contained in each strategy's charter.

Each strategy's plan must quarantine funds for evaluation. DET's evaluation policy recommends the following amounts for sufficient evaluations:

• $100 000 for programs costing less than $5 million
• $150 000 for programs costing between $5 million and $10 million
• 1.5 per cent of program cost for programs costing over $10 million.

In 2014–15, there were three strategies on the schedule of evaluation. An examination of these evaluations demonstrated that two were appropriately carried out. The third item on the schedule was not an evaluation but rather a project plan for a new strategy that included an outline of how the project will be evaluated. This demonstrates good practice for significant strategies as it provides DET's leadership with an opportunity to comment on the adequacy of the proposed evaluation plan.

DET has not adequately learnt from its evaluations

However, DET does not aggregate what it learns from the evaluations it undertakes to inform an assessment of its overall performance. This is despite this aggregation being the responsibility of two bodies within the governance structure.

For strategies that are not included in the evaluation schedule, DET has an evaluation panel—made up of consultants—that business areas may choose to use. However, business areas are not required to use the evaluation panel when procuring an evaluation. Not using the panel means that PED has no oversight of evaluations being conducted independently. This means that of the 78 projects listed in DET's work plan, there are potentially 75 evaluations that have no central coordination or oversight.

DET has a Research and Evaluation Register that is used as a central repository of information on evaluations and research conducted in the department. Business areas upload evaluation reports once projects have been finalised. However, use of the register as a tool to share information is not widely practiced. This is a significant weakness in DET's evaluation policy as it inhibits its ability to learn from the evaluations and improve policy or practices across the department.

4.4 Performance monitoring needs improvement

DET's Outcomes Framework intends to provide data that DET business areas can use to answer specific questions and understand the impact of reforms, strategies and policies. The Outcomes Framework provides a list of outcomes, describes the importance and impact of the outcomes on Victorians and outlines the indicators that will determine DET's progress towards its goal of providing a world-leading learning and development system.

While the Outcomes Framework contains the information needed to inform strategic planning at all levels across DET, there are no specific target measures against which to evaluate performance.

The Outcomes Framework only details a series of outcomes, states why each matters and lists the performance measures or indicators. For example:

Outcome:

VET students attain qualifications to support participation in the economy.

Why it matters:

Attainment of qualifications or certificates improves the employability skills, higher earning potential and employment of VET students.

Indicator:

Proportion of students who are attaining the qualifications they need to participate in the economy.

These indicators are useful in that they outline why specific outcomes matter. From a planning perspective, this encourages a focus on DET's objectives and helps determine whether proposed actions will contribute to achieving them or not. However, this framework would be better, and lead to greater accountability, if it incorporated specific targets against which performance could be assessed.

DET advised that the Outcomes Framework was designed to generate evidence on performance and be a tool to lift its culture regarding the development and use of evidence. For these reasons targets were not included as there was concern that their addition may result in a 'compliance' response.

As part of the government's Education State program, 10 targets have been identified for annual reporting and will be reported on publically to monitor key educational outcomes for schools. These are complemented by the indicators and measures reported through outcomes assessment reports.

4.4.1 Performance reports do not meet user needs

As part of its Outcomes Framework, DET produces four outcomes assessment reports biannually. The reports focus on how selected groups are tracking in relation to the early, interim and long-term indicators that support DET's strategic outcomes of achievement, engagement, wellbeing and productivity. These reports relate to the following groups:

• statewide—system-level performance across the life course
• low socioeconomic status(SES)—performance of Victorians from low SES backgrounds or attending services in low SES areas
• Indigenous—performance of Indigenous Victorians
• regional—performance in DET's four regions that cover all of Victoria.

These reports provide a substantial amount of data with which performance trends can be assessed, to determine whether performance has improved, remained the same or decreased. However, it is not possible to examine whether performance achieved the intended goals given there are no targets to compare against.

The Outcomes Framework is made available annually at a regional level. Regions still produce their own sub-set of data given what they considered were weaknesses with the framework. Regions also produce this data independently of each other, which means schools receive different types and levels of performance data across the state. This does not facilitate effective planning aimed at achieving statewide outcomes. DET plans to develop a Regional Reporting Framework to be implemented in 2016.

4.4.2 Performance measures do not adequately assess performance

Agencies need to have clear and measurable outcomes and outputs in order to effectively monitor performance. Performance indicators need to provide information on whether an agency is achieving its objectives. Output performance measures and targets are used to assess achievement of outputs an agency is funded to deliver.

The Victorian Government's annual Budget Papers outline how public money will be spent over a four-year period. BP3 outlines the government's service priorities and the expected and actual performance in delivering these priorities.

DET's BP3 measures are weak in parts. For 2014–15, DET had 86 BP3 measures covering six outputs, which account for $12.065 billion in funding. DET had the highest budgetary output-to-measure ratio of any government organisation—one measure for every $140 million of output, compared to a Victorian average of one measure for every $40 million. Having such a high ratio reduces the level of scrutiny of DET's performance.

In 2014, DET commissioned a review of its BP3 measures—Performance Measures Review October 2014 (BP3 review).

The purpose of the review was to update and refresh DET's suite of BP3 measures and targets to more closely align them with DET's policies and strategic direction.

While the review did not lead to a significant increase in the number of BP3 measures, the review's recommendations have created stronger measures by:

• better targeting measures
• addressing measurement gaps
• removing outdated and redundant measures.

A 2013 Department of Treasury and Finance base funding review found that some measures and targets did not encourage progress towards DET's overarching goals.

DET's BP3 review identified measurement gaps across DET's outputs. It recommended better targets, additional measures across most output areas and the removal of outdated and redundant measures. While DET has worked to improve its BP3 measures since the review, most of the review's recommendations have not been addressed.

Of the 39 recommendations, 28 were accepted and modified, and 11 not accepted. Of those that were not accepted or modified, reasons included:

• DET identified quality issues with the data
• DET anticipated that there would be another major BP3 review, or future work that would be more targeted
• the measure was still under consideration by DET.

It is concerning that DET has not addressed several BP3 weaknesses. DET should have addressed all weaknesses by now including data quality issues. The anticipated subsequent BP3 review has not yet been scheduled, meaning it is unclear when the remaining weak BP3 measures will be addressed.

4.4.3 Responding to underperformance

DET was not able to demonstrate that it adequately addresses underperformance, either in relation to BP3 measures or the performance measures listed in DET's strategic plan.

Despite having mechanisms in place, such as the annual retreat for the EB, and the annual planning conference, to reflect on performance and inform the development of priorities, strategies and actions for the year ahead, DET does not manage underperformance. Shifts in policy and reform strategies should be informed by underperformance in particular areas. But DET has no policy or process in place and could not adequately demonstrate that it was addressing underperformance when identified.

Without clearly defined targets, measures, outcomes and guidance, there is little capacity for DET to identify and monitor the underperformance of students, teachers and schools. VAGO has consistently found that DET is lacking in this area of governance.

Recommendations

That the Department of Education & Training:

5. develops targets for the Education and Training Outcomes Framework, where applicable, so that progress and performance can be properly assessed
6. establishes a central repository for strategic program evaluations and applies the lessons learned from evaluations to current and future programs.