Performance Reporting Systems in Education

Conclusions

DEECD has an inconsistent approach to managing information. This has led to good practice in some instances, but notable shortfalls in others. These inconsistencies are in part a result of DEECD not having an information strategy. Without a strategy it has no way to drive better practice in how it manages and uses the significant volume of information it collects. This includes ensuring that a consistent set of data standards and controls apply across all of its systems, and deriving the maximum possible value from its information resources.

We assessed six information systems and found that four are managed appropriately. However, two systems had notable deficiencies, which means there is no assurance that the information they contain is consistent and reliable. Weaknesses in these systems raise doubts about the quality of some the information DEECD relies on to make important policy and resource allocation decisions.

We found encouraging signs that DEECD is working to improve how it manages information. It is developing an information strategy that, if implemented fully, will drive a consistent, better-practice approach to managing information right across the department. DEECD is also working to make better use of the information it collects by developing new interactive ways for people to access data. It is important that DEECD follow through on this work.

Findings

Strategic approach to managing information and alignment with whole-of-government information principles

Efficient and effective organisations manage and use information well. The Department of Treasury and Finance (DTF) has published Whole of Victorian Government Guideline: Information Management Principles that, if adopted, will help government departments and agencies to do so. The principles underscore the importance of information and provide a framework for departments to monitor their activities, understand their impact and drive better services and outcomes. An information strategy is a practical way for departments to put these principles into practice.

DEECD developed a Knowledge Management Strategy in 2007. However, it did not fully implement its actions and, as a result, missed many of the potential benefits it sought to achieve. Without an up-to-date information strategy, DEECD does not have a way to embed DTF's information management principles into its practices. Information, therefore, has primarily related to service delivery rather than being seen as a strategically managed resource capable of being used to enhance operational efficiency and effectiveness.

While lots of information is gathered it is not always the most useful information and is not always easy to discover and use.

We reviewed nine separate performance audits of DEECD from the past three years and found a recurrence of information management related issues, including:

• inconsistent data collection methods and controls that have led to the reliability of some data being compromised
• lack of information transfer between the early childhood and school sectors
• lack of access to, or use of, important data in a range of different areas.

As a result of these repeated issues, DEECD and those who deliver services on its behalf cannot effectively:

• evaluate programs, determine resourcing requirements and develop policy
• undertake longitudinal studies based on robust and complete student data
• identify and target resources towards the most vulnerable children in their critical transitions from early childhood into school
• determine if program funding has been well spent and provides value for money.

Managing key information systems

We assessed the processes and controls associated with six of DEECD's key information systems. DEECD uses different processes and standards to control the data in each system. However, controls applied to four of the six systems were robust. We found multiple weaknesses in two of the systems examined—MCH and the Census of Computers in Schools. These weaknesses pose serious questions about the reliability of the information used and reported publicly from them.

Maternal and Child Health

The 79 local government authorities in Victoria operate MCH services, but DEECD provides 50 per cent of total funding. MCH services collect a wide range of information about infants and young children and use this information for case management. DEECD relies on this information to fund MCH services, but does little to verify and validate the data, and does not enforce consistent data definitions. Therefore, it has no assurance that the data it is basing its funding decisions on is accurate.

DEECD only has access to a very limited selection of the data collated by MCH services. Without access to the broader data, DEECD can't effectively evaluate the services being provided on its behalf. It also has limited information that it can analyse and feed back to the MCH services to assist them in their role.

Census of Computers in Schools

The Census of Computers in Schools gathers information on the number of computers available for teaching in individual schools. DEECD reports this information in Budget Paper No.3: Service Delivery, and uses it to inform national statistics, licence software and evaluate policy.

Given the importance of information and communications technology (ICT) to delivering a modern curriculum, it is critical that schools and DEECD accurately understand and monitor the age and composition of the school computer fleet.

Schools use different methods to collect data and do not consistently interpret data definitions. This means that there is a risk that obsolete assets are being included in the data reported and used by schools, which limits the usefulness of this information to both schools and DEECD.

Recent focus on improving information management

DEECD has acknowledged that there are issues with how it manages information and is taking steps to address these issues. It recently developed a new ICT strategy, is currently developing a four-year information strategy based on whole-of-government better-practice principles, and is further developing its capability to link separate information systems through a single data repository.

It is critical, however, that DEECD follow through on this work. Previous attempts to implement an information strategy have been unsuccessful. As a result, critical issues with information—its quality and its use—identified repeatedly over the years have persisted. If DEECD does not address these issues, its ability to understand the performance of its schools and other service providers will remain restricted, as will its ability to make well-informed decisions.

Recommendations

The Department of Education and Early Childhood Development should:

1. adopt system-wide processes and controls that ensure that all of its information systems are regularly reviewed, managed to consistent standards and contain reliable and accurate data
2. develop a detailed project plan for the implementation of its information strategy that describes the scope of activities, required resources and time lines, and implement actions in accordance with this plan.

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report was provided to the Department of Primary Industries with a request for submissions or comments.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

The Department of Education and Early Childhood Development (DEECD) is administratively responsible for Victoria's early childhood services, school education, and tertiary and further education. Early childhood and post-compulsory education services were previously administered by other departments, but machinery of government changes in 2007 and 2010 brought these divisions together under DEECD. These changes were underpinned by a policy agenda that sought to support a lifelong approach to learning and development.

By incorporating these additional portfolio responsibilities, DEECD assumed responsibility for providing services, every year, to more than:

• 400 000 children and families in early childhood programs
• 850 000 government school students
• 645 000 tertiary and further education students.

DEECD oversees the delivery of these services with an annual budget of more than $11.5 billion, but in a highly devolved environment, where it relies on councils, schools and vocational education and training providers to deliver services.

Effective performance reporting is critical for DEECD to demonstrate to Parliament and the public that it is achieving its objectives and providing equitable access to services that are appropriate, high quality, cost effective and that provide value for money. DEECD also needs timely access to high-quality information to understand how efficiently, effectively and economically services are being delivered, to know where and when to intervene to address poor performance, and to make sure that service providers are well informed and able to appropriately plan and deliver services.

1.2 Reporting requirements

Section 45 of the Financial Management Act 1994 (the Act) sets out compulsory reporting requirements for government departments and public bodies. It requires them to prepare annual financial statements and submit them to the Auditor-General for audit. Section 46 of the Act requires them to submit a report of operations and audited financial statements to the relevant ministers for tabling in Parliament. While financial statements provide a basis to assess departments' management of their finances, the report of operations provides an indication of whether agencies are delivering intended outputs and fulfilling objectives.

The Act is supported by the Standing Directions of the Minister for Finance. Standing Direction 4.4 requires government departments to set performance indicators and targets that report against their objectives. These must align with guidance published by the Department of Treasury and Finance (DTF).

Budget Paper No. 3: Service Delivery (BP3) outlines the government's priorities for goods and services. It details the government's budget decisions and explains how these decisions support the delivery of core services and infrastructure. It includes a breakdown of output funding, associated performance targets and historical performance against those targets.

1.2.1 Other reporting obligations

In addition to DEECD's reporting commitments under the Act, it is required to provide performance information to the Commonwealth Government, under the National Education Agreement, the Intergovernmental Agreement on Federal Financial Relations 'Public Accountability and Performance Reporting' and to the Australian Curriculum, Assessment and Reporting Authority.

In order to meet these reporting obligations, DEECD must assure that data-related risks are managed. This extends to assuring that risks associated with third parties' provided data are properly managed.

1.3 Department of Treasury and Finance guidance on information management

DTF identifies quality data as one of the most valuable assets in the public sector and that the completeness, consistency, validity, accuracy, timeliness and secure flow of data are essential to:

• meet public commitments
• fulfil legislative requirements and obligations
• support decision-making and policy making
• underpin service performance, development and maintenance
• support and assess political and legislative initiatives and programs
• measure program and service performance against strategy, objectives and key performance indicators.

It has published the Whole of Victorian Government Guideline: Information Management Principles and VPS data integrity manual (2010) which describes expected behaviours and best practice, and provide high-level guidance on how information should be managed to meet government accountability requirements. Figure 1A outlines DTF's information management principles.

Figure 1A

Information management principles

Source: Department of Treasury and Finance, Whole of Victorian Government Guideline: Information Management Principles.

These principles provide a sound framework against which agencies can base their own information management practices and policies. They also align with public sector information principles published by the Australian Information Commissioner in 2011, particularly around effective information governance and open access to and use of information.

Figure 1B details the five elements of data integrity as defined by DTF in its VPS data integrity manual.

Figure 1B

Data integrity requirements

Source: Victorian Auditor-General's Office from the Department of Treasury and Finance's, VPS data integrity manual.

The VPS data integrity manual provides sound guidance on the effective management of information. This audit used both the Whole of Victorian Government Guideline: Information Management Principles and VPS data integrity manual to assess the effectiveness of DEECD's information management practices.

1.4 Audit objectives and scope

The objective of this audit was to assess the effectiveness of DEECD's information management systems, processes and controls. Specifically, whether:

• DEECD's information management practices are informed by an overarching information management strategy and align with Whole of Victorian Government Guideline: Information Management Principles
• DEECD's information management practices fully reflect the requirements of DTF's VPS data integrity manual on managing data to ensure data integrity.

1.5 Audit method and cost

2.1 Introduction

Timely access to high-quality performance information is fundamental to the effective and efficient delivery of services. It provides a robust evidence base to underpin decision-making around policies and programs, and to achieve public accountability for the services delivered.

The timely flow of quality information is essential to many aspects of the Department of Education and Early Childhood Development's (DEECD) oversight of its portfolio responsibilities:

• Early childhood—Maternal and child health (MCH) services are administered by local government authorities, but are co-funded by DEECD. DEECD needs access to timely and accurate information about numbers of families accessing services to determine funding allocations.
• Schools—DEECD requires high quality and timely information about how well schools are being managed and how students are progressing. Without this information it cannot provide timely and targeted support to schools, and will not know when or how to intervene in poorly performing schools.
• Further education and training—For Victoria's market-led vocational education and training system to work effectively, DEECD must have access to robust information about what training is being delivered, to whom, and by which training providers.

This Part of the report examines the effectiveness of DEECD's information management systems, processes and controls and asks two questions:

• Are DEECD's information management practices informed by an overarching information management strategy and do they align with the Whole of Victorian Government Guideline: Information Management Principles?
• Do DEECD's information management practices reflect the best practice requirements of the Department of Treasury and Finance's (DTF) VPS data integrity manual on managing data to ensure data integrity?

2.2 Conclusion

DEECD's information management systems, processes and controls are largely effective, despite considerable variations in how it manages individual systems.

Its information management practices are not informed by an information management strategy and, as a result, DEECD has no way to drive better practice and greater consistency in how it manages and uses the significant volume of information it collects.

Four of the six systems examined reflect the requirements of DTF's VPS data integrity manual. However, the two remaining systems had some inherent weaknesses that limit DEECD's ability to use this information to good effect.

DEECD is currently developing a new information strategy which, if fully implemented, will address some of the issues identified in this audit and drive better practice. However, previous attempts to implement an overarching information strategy have been unsuccessful and the implementation of this new strategy will need to be carefully managed to make sure that past mistakes are not repeated.

2.3 DEECD's information management

DEECD has not had an operational information strategy in place for several years.

In a large department such as DEECD, with six major divisions spread across three portfolios, a whole-of-department approach to information management is critical to ensure that information is managed and used well. An overarching information strategy is an important component of this.

When designed and implemented effectively, an information strategy should enable consistent and rigorous management of multiple information systems across an organisation. It should also allow organisations to embed better-practice information management principles at a strategic level, signalling the importance an organisation places on its information resources, and a commitment to maximising the usefulness and value of its information.

DEECD's new information strategy is unlikely to overcome its long‑term information management challenges, because as at September 2013, the following key elements are missing:

• a long-term, high-level commitment to improving information management across all areas of the department
• clear leadership, accountability and governance arrangements not only for the strategy, but also for the operation, maintenance and development of the information systems
• detailed implementation and resource plans supported by appropriate funding
• robust and well-defined project management practices.

2.3.1 Absence of an information strategy

In 2007, the then Department of Education developed a Knowledge Management Strategy for the period 2007–11 to address issues it had identified with its management and use of information. The strategy sought to overcome issues such as:

• the fractured nature of information systems, which limit the department's capacity to link information together
• difficulty sourcing relevant information from the large volumes of available data
• challenges in applying learning outcomes data in program planning and evaluation
• barriers to community members accessing information on education options.

Although objectives and action areas were identified in the strategy, it was not comprehensively implemented and was not updated in 2007 or 2010 to reflect machinery of government changes when the department took on responsibility for early childhood development services and vocational education and training. As a result, many of DEECD's information management systems have operated in isolation and have lacked coordination.

2.3.2 Implications of not having an information strategy

As a result of not having an adequate information strategy in place to facilitate integration between systems and to drive a consistent, rigorous approach, DEECD has lacked the leadership needed to drive better practice in information management. Within government, better practice is characterised by adherence to the overarching principles that information:

• is recognised as a valuable asset
• meets business needs
• is easy to discover and use
• is shared to the maximum extent possible.

The absence of a strategy has not prevented divisions and work groups within DEECD from valuing and using information in a manner that is consistent with better practice. Later sections of this report highlight some instances of good practice we identified in our review of six key information systems.

Nine separate VAGO performance audits of DEECD, undertaken over the past three years show a recurrence of information management related issues, including:

• inconsistent data collection methods and controls that have led to the reliability of some data being compromised
• lack of information transfer between the early childhood and school sectors
• lack of access to, or use of, important data in a range of different areas.

These recurring issues compromise the reliability of information and limit DEECD's ability to make the best use of the information it does have, impacting its ability to discharge its responsibilities effectively and efficiently. It also creates challenges for DEECD to determine if program funding has been well spent and provides value for money.

Figure 2A highlights in further detail the information management issues identified by these performance audits.

Figure 2A

Findings of recent performance audits

2.4 Management of key performance information systems

There are notable variations in the management and controls applied to information systems across DEECD. While some were managed to a high standard, others had weaknesses that compromised the reliability of the information they contained.

2.4.1 Data integrity assessments

In the absence of an overarching information strategy, our audit looked at six of DEECD's key information systems to develop an understanding of how its information management works in practice. We assessed these systems to determine the extent to which they comply with the VPS data integrity manual. Together these systems represent more than half of the output measures reported by DEECD in Budget Paper No.3: Service Delivery (BP3). Figure 2B contains a summary of data integrity findings for each system.

Figure 2B
Audit assessment of the data integrity of information systems

Note: Definitions of assessment criteria are located in Figure 1B.

Source: Victorian Auditor-General's Office.

Figure 2B highlights some positive findings about the way DEECD manages individual information systems. It shows that—even without a department-wide strategy governing information management—four of the six systems had fundamentally sound processes and controls.

The Centaur and On Track systems were assessed as having the highest standards of data integrity. This means that there is strong assurance that the information they contribute to DEECD's public reporting is consistently accurate and robust.

Similarly, the Skills Victorian Training System (SVTS) and CASES21 systems were found to have accurate and reliable publicly reported information, with some minor weaknesses:

• DEECD relies on data from a separate system to verify the information entered into SVTS by service providers. This verification data is not consistently updated in a timely way.
• The issues with CASES21 related to some incidences of confusion around data definitions, leading to weaknesses in the accuracy and consistency of information.

Despite these weaknesses, we found their overall management nevertheless allows for reliable assurance as to the accuracy and reliability of publicly reported information.

Collectively, these four systems are responsible for one-third of the measures DEECD reports in BP3. Their robust management points towards veracity of the information they contain, which is a positive result.

2.4.2 Key information system weaknesses

Figure 2B also identifies multiple weaknesses in two of the information systems examined, raising questions about the reliability of the information they contain.

Maternal and Child Health

There are notable issues with the collection and use of MCH information which reduces assurance that DEECD is obtaining accurate, complete, consistent, valid and timely data on the delivery of these services.

While MCH care is provided by the 79 local government authorities in Victoria, DEECD provides 50 per cent of total funding. Each MCH service collects information about infants and young children primarily to facilitate case management, but passes some data on to DEECD, who then use it as the basis of funding allocation. At the individual service level, we found issues with how this information is collected, stored and then reported to DEECD. Specific issues include:

• each MCH service uses one of three programs to house the information it collects and DEECD does not have direct access to or control over these systems—this creates inefficiencies and, without full access, limits DEECD's ability to undertake meaningful analysis of data
• high levels of manual data entry—increasing the risk that data is entered incorrectly and casting doubts on the reliability of data
• inconsistencies in data definitions—leading to different MCH services recording the same information in different ways
• minimal verification and no validation of data—meaning DEECD cannot be assured that it is reliable and accurate.

Further, the most commonly used system—used by 61 councils—has a recognised flaw that causes errors when totalling figures, requiring staff to manually override the data. While this is a well-known problem amongst maternal and child health services, this additional manual manipulation of the data compromises its integrity and reduces the efficiency of administration and data collection.

Issues with these MCH information systems have been identified numerous times both by VAGO and by DEECD:

• VAGO's 2007 performance audit Giving Children the Best Start in Life and its 2011 performance audit Early Childhood Development Services: Access and Quality both noted that multiple systems and high levels of manual data entry leads to inefficiencies and increases the risk of transcription errors, which, in turn increases the need for better quality assurance measures. VAGO recommended that a single statewide database be developed and while this recommendation was accepted, to date no such database has been developed.
• A DEECD internal audit, conducted in 2010, found that there are significant and long-term risks with MCH information systems, however, these have not been addressed.

Almost one-third of the early childhood development performance indicators DEECD reports through BP3 come from MCH data. It is therefore central to DEECD's accountability for its early childhood development activities. Further, data collected by MCH services is a valuable resource to assist DEECD to identify and target services to vulnerable children. This is a particularly important capability for DEECD given the widely acknowledge importance of early intervention programs. DEECD has commenced work to overcome issues with MCH data accuracy and consistency, and to enable more extensive analysis of MCH data.

Census of Computers in Schools

The Census of Computers in Schools collects information on the number of computers available for teaching at individual schools. DEECD uses this information to:

• develop school, sector and state-level student-to-computer ratios
• inform BP3 performance measures
• inform national statistics
• comply with software licence requirements
• inform infrastructure planning, provision and support
• monitor implementation of policy initiatives
• monitor the age and composition of its computer fleet.

However, there are weaknesses in the way that schools input information into the system—specifically, inconsistent interpretation of data definitions, which are not addressed by any verification testing by DEECD.

These issues mean the figures that schools report to DEECD could be based on different assumptions, making them potentially unreliable as a measure of students' access to functional information and communications technology (ICT) equipment. While the age of computers is reported, this is a crude measure of functionality. A lack of data verification and testing by DEECD means that these inconsistencies are not addressed.

While this system contributes information to only two of the performance indicators DEECD reports in BP3, the lack of assurance about the reliability of the data it contains is still concerning. DEECD advised that this data remains an input into policy and service planning, and therefore its accuracy is important to underpin prudent decisions.

The importance of ICT to delivering a modern curriculum means it is essential that schools and DEECD have an accurate understanding of student-to-computer ratios so they can plan appropriately. VAGO's 2012 audit Obsolescence of Frontline ICT: Police and Schoolsfound that secondary school learning goals that depend on ICT assets are at risk due to insufficient oversight by DEECD, despite it knowing that schools do not routinely take into account the lifecycle costs of ICT assets. The processes and controls associated with the Census of Computers in Schools information system mean that, currently, the accuracy of the data it contributes to DEECD performing this oversight cannot be assured.