The State of Victoria's Children: Performance Reporting

Conclusions

SVC reports contain valuable information about Victorian children's health, development and wellbeing. They are recognised within government and beyond as valuable tools for understanding how children and young people are faring in Victoria, and they make an important contribution to the evidence base underpinning policy and program development.

The government's decision to publicly release SVC reports reflects a commitment to make information about the welfare, safety, learning and development of children available to those in a position to influence children's lives through policies and programs. With the community and the not-for-profit sector playing an increasing role in the delivery of government services to children, SVC reports have the potential to significantly influence service delivery.

However, SVC reports are not comprehensive, do not consistently report trend data and do not demonstrate how the outcomes for children are influenced by specific government programs.

Findings

Comprehensive reporting

The government developed and committed to a reporting framework and associated set of indicators to underpin SVC reports, which have been recognised as innovative and have informed approaches to reporting in other jurisdictions. However, SVC reports have not comprehensively and consistently reported against this framework.

The content of SVC reports has varied considerably over the six years of publication, which limits their usefulness. While each report provides important information, and in some cases a detailed focus on themes or cohorts which are important to stakeholders, the current approach is at the expense of systematic, comprehensive reporting. This reduces the capacity of the reports to demonstrate the changing outcomes for children over time. It also prevents reports from building on one another to present a wide reaching and increasingly rich evidence base about how well children are faring in Victoria.

Consequently, each report exists as a stand-alone document rather than a report in a longitudinal series. There is a role for both detailed and comprehensive reporting to facilitate a better understanding of how young people are faring.

Recognising the importance of comprehensive reporting against the Victorian Child and Adolescent Monitoring System (VCAMS) indicators, DEECD commenced work in 2008 to make the full set of indicators available online. Despite almost $2 million expenditure on this project to date, the information has still not been made publicly available.

The lack of inclusion of a complete, consistent set of indicators within SVC reports limits their usefulness to the Board as a mechanism to monitor the outcomes of government actions. It also limits their usefulness to the broader network of organisations involved in planning and delivering services to Victoria's children. These are the two key functions of SVC reports.

Understanding and evaluating state of Victoria's children reports

DEECD does not monitor the influence of SVC reports on policy and programs within government, nor has it evaluated the extent to which SVC reports have acted as a catalyst for improved service delivery within the non-government sector.

The intention of SVC reports has been to contribute to the evidence base informing policy and program development. Despite this, DEECD does not monitor and has not evaluated the impact its SVC reports have had on:

• facilitating cross-government coordination in development and delivery of policy and programs influencing children
• service design and delivery within the non-government sector.
• DEECD does not consult with non-government stakeholders to determine whether SVC reports are meeting their needs. This means that changes to the contents of SVC reports are occurring each year without any insight into whether previous reports have been effective, and whether content or format changes could improve any perceived deficiencies.

The Framework and VCAMS indicators that form the foundation of SVC reports were developed in 2005 and 2007 respectively. However, they have not been reviewed to determine their ongoing relevance since their inception. Further, there are a limited number of indicators in the Framework for which data sources have yet to be developed. It is important that the suitability of these indicators be revisited.

Reporting on the most vulnerable children in the community

The Act requires SVC reports to focus on the 'most vulnerable children in the community'. However, SVC reports do not define or focus specifically on this cohort. Neither the Framework nor VCAMS define 'most vulnerable', and there is no hierarchy of indicators that allows for focus on the most vulnerable children.

None of the six SVC reports published include any discussion of outcomes for vulnerable children. While some reports have focused on priority areas, this was not done as part of a strategy.

Notably, the 2012 report of the Victorian Government's Protecting Victoria's Vulnerable Children Inquiry made recommendations that a definition of vulnerable children be developed as part of a vulnerable children's strategy, and that information about these children and the services delivered to them be included in SVC reports. This recommendation highlights that SVC reports are not currently serving this purpose.

Work to include information about vulnerable children in future SVC reports as part of a wider strategy for vulnerable children and families is currently underway.

Recommendations

The Children's Services Coordination Board should:

1. review and clarify the role and purpose of state of Victoria's children reports in consultation with non-government stakeholders.

The Department of Education and Early Childhood Development should:

2. expedite completion of the project to make Victorian Child and Adolescent Monitoring System data publicly available online
3. evaluate the Victorian Child and Adolescent Monitoring System to determine its ongoing relevance for state of Victoria's children reports
4. establish agreed mechanisms to assure the quality of information being reported in state of Victoria's children reports.

Submissions and comments received

In addition to progressive engagement during the course of the audit, in accordance with section 16(3) of the Audit Act 1994 a copy of this report, or relevant extracts from the report, was provided to the Department of Education and Early Childhood Development with a request for submissions or comments.

Agency views have been considered in reaching our audit conclusions and are represented to the extent relevant and warranted in preparing this report. Their full section 16(3) submissions and comments are included in Appendix A.

1.1 Introduction

Good policy is informed by a robust evidence base underpinning decisions and actions.

For effective policy development, the availability of current and reliable information about target populations is essential to understand what impact existing policies and programs are having, and to identify areas to focus potential improvements on.

This information is equally valuable to those outside of government and provides transparency around the health, wellbeing and development status of populations. It also assists the work of the community and not-for-profit sectors which are playing an increasing role in service delivery on behalf of the government.

1.2 Policy context

During the 2000s, the Victorian Government increased the policy focus on children and young people. In 2003, the Premier established a Children's Advisory Committee to provide advice on ways for the government to improve outcomes for children.

In 2004, the Children's Advisory Committee released a report which made 37 recommendations to the government, including recommending changes to the administrative structures governing the planning, delivery and oversight of services to children to support more effective cross-government coordination. It also recommended that the government adopt principles to guide investment in children and develop a master plan for Victorian children by identifying desirable outcomes and specific targets for their achievement.

The Children's Advisory Committee also recommended that the government develop a coordinated statewide system to collate data on the education, health and wellbeing of Victorian children.

In response, the government implemented a range of initiatives from 2005 onwards aimed at improving outcomes for children. These initiatives included establishing:

• a Minister for Children to advocate for the interest of children across government
• an Office for Children to oversee service delivery, planning, review and performance measurement of children's services
• a Victorian Children's Council, comprising experts in children's policies and services, to provide advice to the Premier and Minister for Children
• a Children's Services Coordination Board, comprising heads of key government departments, to support an increasingly coordinated approach to service delivery
• a Child Safety Commissioner to provide accountability and oversight in relation to child safety.

These changes recognised that positive developmental outcomes for children require coordinated efforts across multiple areas, particularly in health, education, wellbeing and safety.

1.2.1 The Child Wellbeing and Safety Act 2005

A key component of the government's focus on children was the Child Wellbeing and Safety Act 2005 (the Act), which came into effect in November 2005.

The Act was intended to provide legislation to underpin the government's new policy focus on children, and the foundation for government and non-government services to work together to promote children's health, development, learning, wellbeing and safety.

It established principles to guide the development and provision of services to children, and mechanisms for whole-of-government collaboration to promote children's wellbeing and safety.

These principles state that services for children and families should be developed to promote continuous improvement in the quality of those services, based on the best available knowledge of children's needs and their stages of development.

The Children's Services Coordination Board

Another key component of the Act was the establishment of the Children's Services Coordination Board (the Board) as the mechanism to facilitate cross-government coordination of children's services. Under the Act, the Board was given two main functions:

• to annually review and report to the Minister for Children—now the Minister for Children and Early Childhood Development—and the Minister for Community Services (the Ministers) on the outcomes of government actions in relation to children, particularly the most vulnerable children in the community
• to monitor administrative arrangements to support coordination of government actions relating to children at local and regional levels.

The Board is comprised of the secretaries of the departments of Premier and Cabinet, Treasury and Finance, Human Services (DHS), Education and Early Childhood Development (DEECD), Health (DoH), Justice (DOJ), Planning and Community Development (DPCD) as well as the Chief Commissioner of Police.

The Board's role in coordinating and driving government action to address the needs of vulnerable children was considered as part of the Victorian Government's 2011 Protecting Victoria's Vulnerable Children Inquiry.

1.3 State of Victoria's children reports

To fulfil its obligation to review and report on the outcomes of government actions in relation to children, and reflect the commitment to a robust evidence base to underpin policy and program development, the Board produced a state of Victoria's children (SVC) report in 2006, led by the newly established Office for Children within DHS.

1.3.1 The Victorian Child and Adolescent Outcomes Framework

As a precursor to developing the report, the then Office for Children within DHS oversaw development of an outcomes reporting framework to create the prime content for inclusion in the reports.

Distinct from output reporting, which provides information on services delivered or activities performed, an outcomes reporting framework aims to provide information about the impact of services and activities on individuals and communities.

The Victorian Child and Adolescent Outcomes Framework (the Framework) consists of 35 desirable outcomes that relate to aspects of children's health, development, safety learning and wellbeing, as well as factors that influence them, such as families, communities and society.

By encompassing community, social and family outcomes as well as those that relate specifically to children, the Framework is intended to recognise and measure the multiple, integrated factors that impact on children's health, development and wellbeing.

The Framework was endorsed by the government in November 2005.

Figure 1A

The Victorian Child and Adolescent Outcomes Framework

Source: The Department of Education and Early Childhood Development.

1.3.2 The Victorian Child and Adolescent Monitoring System

In conjunction with development of the Framework, a range of indicators were selected to measure achievement for each outcome. This collection of indicators is referred to as the Victorian Child and Adolescent Monitoring System (VCAMS).

VCAMS indicators were selected on criteria requiring that they are:

• worth measuring
• measurable for diverse populations
• understood by people who need to act
• able to galvanise action
• relevant to policy and practice
• able to reflect results of actions
• feasible to collect and report
• compliant with national processes of data definition
• directly related to the Framework.

The indicators are drawn from 42 separate data sets held by various agencies, including DEECD, DHS, DoH, the Australian Bureau of Statistics, Victoria Police and Monash University.

A sample of indicators, and the outcomes they are associated with, is included in Figure 1B.

Figure 1B

A sample of Victorian Child and Adolescent Monitoring System indicators

Source: Victorian Auditor-General's Office.

While there are 150 core indicators, in some instances, an indicator has sub‑components or measures. This brings the total number of VCAMS indicators to 183. These indicators were endorsed by the Board in 2007.

While the early SVC reports were published prior to finalisation of the VCAMS indicators, the indicators they reported would eventually be included in VCAMS.

1.3.3 Developing state of Victoria's children reports

There have been six SVC reports published since 2006.

Figure 1C

Overview of each state of Victoria's children report

Source: Victorian Auditor-General's Office.

Production of the 2006 report was led by the Office for Children in DHS. Following machinery of government changes in 2007, the functions of the Office for Children were moved to DEECD. Since then, production of SVC reports has been led by DEECD.

SVC reports are developed with input from an Inter-Departmental Committee (IDC), established to support the Board, with representatives from the same departments and agencies as the Board. A steering group is also convened each year, again with representatives from the same departments and agencies as the Board. DEECD, the steering group and the IDC progressively review available data to identify the results to include in SVC reports, and highlight issues that may warrant greater focus.

Possible themes are then suggested to the Board, which makes a decision about the focus of that year's SVC report. DEECD then works with the IDC to gather further data, undertake analysis and develop the report. It is then presented to the Board for approval, and then submitted to the Ministers.

To date, the Ministers have then presented the SVC report to the relevant Cabinet subcommittee for approval to release it publicly. While not a requirement under the Act, this has occurred each year since 2006.

1.4 Audit objective and scope

The objective of the audit was to evaluate the quality of information contained in SVC reports, and determine whether they tell a clear and useful story.

It looked at whether:

• SVC reports are relevant, appropriate and complete
• information is well communicated and tells a comprehensive story.

1.5 Audit method and cost

The audit was conducted under section 15 of the Audit Act 1994 and in accordance with the Australian Auditing and Assurance Standards. The audit method involved meetings with DEECD, the Board secretariat and a variety of stakeholders as well as reviewing the content of SVC reports and associated documents.

Pursuant to section 20(3) of the Audit Act 1994, unless otherwise indicated, any persons named in this report are not the subject of adverse comment or opinion.

The total cost was $280 000.

1.6 Structure of this report

Part 2 of this report considers the usefulness of SVC reports.

2.1 Introduction

To effectively understand outcomes and identify opportunities to improve the way that services are managed and delivered, data needs to be measured and reported consistently over time. This allows continuous measurement of the impact of any changes or policies. It can also provide useful point-in-time information and evidence on which to base robust policy and program development.

The Victorian Child and Adolescent Outcomes Framework (the Framework) has been in place since 2005 as the conceptual framework to understand how Victoria's children are faring. The associated Victorian Child and Adolescent Monitoring System (VCAMS) is a suite of defined indicators to understand children's outcomes against the Framework. VCAMS data has been collected since 2007.

VCAMS provides an opportunity for the government to develop a long-term data collection system based on outcomes, to identify how Victoria's children are faring over time and also how government actions are impacting on them. Conversely, it can also highlight where government action may be necessary to address poor outcomes for children.

It is essential that the VCAMS data be presented in a comprehensive and consistent manner to effectively advise the Ministers for Children and Early Childhood Development, and Community Services (the Ministers) of the outcomes of government actions in relation to children, and to maximise the ongoing usefulness of the information gathered.

2.2 Conclusion

State of Victoria's children (SVC) reports contain valuable information about children's health, development and wellbeing. Their annual preparation and presentation to government and the public make them an important resource for policy makers and those involved in designing and delivering services to children and young people.

However, issues with comprehensiveness, the limited inclusion of trend data, and a lack of definition and focus on vulnerable children, compromise their usefulness.

A considerable amount of work has been done to develop a framework to monitor the outcomes of government activity on children and young people to underpin SVC reporting. However, this information is not comprehensively included in SVC reports or other mechanisms. At no time since the introduction of VCAMS have all indicators been reported.

Further, SVC reports do not define, identify or focus analysis on the most vulnerable children in the community, despite this being an explicit requirement of the Children's Services Coordination Board's (the Board) reporting function under the Child Wellbeing and Safety Act 2005 (the Act).

Themed SVC reports that focus on critical issues and cohorts can make a valuable contribution to understanding the wellbeing of Victoria's children. However, the lack of comprehensive reporting limits how effectively the reports fulfil their intended purposes—as reports to the government on behalf of the Board, and as information resources for stakeholders involved in delivering services to children and young people. There is a role for both kinds of reporting to best achieve the SVC report's purpose.

2.3 Systematic and comprehensive reporting

SVC reports were originally developed to allow the Board to fulfil its obligations under the Act. However, since their inception in 2006, SVC reports have had a further role as a means for publicly disseminating information about the status of Victoria's children.

The rationale for publicly releasing these reports has been that the information they contain is of value to a wide range of stakeholders involved in developing and delivering services for children. By adding to the pool of knowledge around how children in Victoria are faring, they are intended as resources to inform policy and program development and service delivery, and ultimately improve outcomes for children.

Fulfilling both roles is best achieved through consistent, systematic and comprehensive reporting against VCAMS indicators. The six SVC reports published to date have not adequately achieved this.

Fulfilling the requirements of the Child Wellbeing and Safety Act 2005

Section 15(a) of the Act highlights that the Board's function is not just to report to the Ministers on the outcomes of government actions in relation to children, but to review and report. The Board's ability to do so—and to assure that it is effectively identifying emerging issues—is best fulfilled when it is comprehensively assessing the outcomes and trends for all indicators that it has approved for this purpose.

Fulfilling the needs of other stakeholders

Similarly, for broader stakeholders, a point-in-time account of the status of children incorporating the interrelated influences over their health, wellbeing and development can best lead to informed policy and program development.

VCAMS has been designed to allow for a holistic understanding of not just children, but their families and the communities that they live in. It acknowledges that when trying to improve outcomes for children, policy makers need to also influence their family and community contexts. Therefore by its nature, VCAMS is a set of indicators that require comprehensive reporting to be meaningful.

2.3.1 Comprehensiveness of state of Victoria's children reporting

SVC reports do not include results for all VCAMS indicators. At no point has the Board been provided with, and in turn reported to the Ministers, a comprehensive report against all outcomes in the Framework, using approved indicators. No other mechanism has been used to provide the Board with these results.

While the 2006 and 2007 reports were published before VCAMS had been finalised, from 2008 the full set of 183 VCAMS indicators had been selected and approved by the Board. Despite this, SVC reports have included between 60 and 150 VCAMS indicators, as illustrated in Figure 2A.

Source: Victorian Auditor-General's Office.

While VCAMS data is not comprehensively reported, SVC reports include additional indicators from different data sources to demonstrate achievement of outcomes. However, this data is not consistently reported from year to year, nor is it consistently reported as time-series data.

VCAMS indicators were selected according to specified criteria, including that they be worth measuring, relevant to policy and practice, and understood by people who need to act. Choosing different indicators risks using information that is less meaningful to key audiences.

Source: Victorian Auditor-General's Office.

VCAMS indicators are also not distinguished from other indicators within the report making it difficult to identify what data relates to VCAMS and what is drawn from outside the Framework.

Separately identifying VCAMS indicators is required for the Framework to fulfil its purpose as a holistic, interrelated account of how children are faring. The strength of VCAMS is in the breadth of areas it covers. For results to be most meaningful to readers, they should be clearly identified as VCAMS indicators and presented in context with other VCAMS results. SVC reports do not present data in this way.

In addition to not reporting all VCAMS indicators each year, the six SVC reports published since 2006 have varied in fundamental ways. Rather than providing comprehensive results for all cohorts, SVC reports have focused on:

• adolescents—2007
• Indigenous children—2009
• key issues affecting children aged 0–17—2010
• rural and regional children—2011.

These themed reports provide a valuable, in-depth focus on those cohorts. However, this is at the expense of comprehensive reporting of outcomes for all children. There is a role for both types of reporting.

2.3.2 Strengths and limitations of the Victorian Child and Adolescent Monitoring System

Both DEECD and broader stakeholders recognise that the Framework and VCAMS provide a good model to report on the status of children. Other jurisdictions, including Tasmania and the Australian Capital Territory have drawn upon VCAMS to develop their own reporting structures. The Framework and data has also been cited in papers from the Council of Australian Governments and Australian Institute of Health and Welfare.

However, DEECD argues that comprehensive reporting of VCAMS is not possible or desirable due, in part, to some inherent limitations, including:

• data collection frequency—not all indicators are drawn from annual data collections, therefore new results are not available for reporting annually
• the number of indicators—there are a total of 183 indicators which could be seen as too many to report
• minimal fluctuation in results year on year—where data results do not show a marked difference from year to year, DEECD advised that it tends not to prioritise reporting them.

These may be legitimate considerations, but the nature of VCAMS means that it is most meaningful when comprehensively reported. Rather than continuing to report from VCAMS selectively, DEECD should review it to determine whether it remains the most appropriate set of indicators to underpin SVC reports.

This review should consider the practicalities of VCAMS as a vehicle to fulfil the Board's review and reporting role, as well as the needs of the broader stakeholder group who use the reports.

2.3.3 Making the Victorian Child and Adolescent Monitoring System available online

Despite spending $2 million since 2008, DEECD's project to make all VCAMS data available online has repeatedly stalled, and almost five years later, has not been realised.

In 2008, DEECD sought approval from the Board to make the full set of VCAMS indicators publicly available online. This was presented to the Board as a way of justifying the evolution of SVC reports into more focused, theme-based reports rather than the vehicle to comprehensively report against all VCAMS indicators.

This would also make data available to service providers, assisting them to target improvements. Stakeholders consulted through the course of this audit consistently voiced a strong interest in online access to all VCAMS data to help inform their work.

The lack of progress on this project means there is currently no means for all VCAMS indicators to be comprehensively reported either publicly, or to the Board. The importance of doing so is widely acknowledged, and DEECD is missing a significant opportunity by failing to execute this project to date.

2.3.4 Presentation of information in state of Victoria's children reports

Comparing results over time

SVC reports do not provide a useful or consistent comparison of results against VCAMS indicators over time. As a result, they do not provide sufficient information to clearly demonstrate trends or indicate the effectiveness of, or need for, government interventions.

In approving the 2008 SVC report—the first to be published following approval of VCAMS—the Board acknowledged that trend data would be of most interest to users of the report and should be included in SVC reports where possible. Despite this, the most recent SVC report included historical information for only 24 out of the 120 VCAMS indicators it reported.

While this is in part due to previously stated limitations with the VCAMS data, it also highlights that despite SVC reports' considerable length and large volumes of information, they miss basic opportunities to deliver data in a meaningful way, and to highlight critical issues with the results.

Figure 2C

Historical reporting on children in out-of-home care

Source: Victorian Auditor-General's Office.

Figure 2D

Trend in rate of children in out-of-home care 2006–11

Source: Victorian Auditor-General's Office.

In addition, there are two indicators that have never been reported in any SVC report. These are:

• clients receiving emergency housing services that are aged 13–17 years
• re-substantiations [of child abuse] within 12 months.

When VCAMS was finalised in 2007, there were no data sources to provide results for these indicators, and to date none have been developed. Despite this, they remain part of the VCAMS set and there is no evidence of a plan to either collect data or revise the indicator set.

Data quality assurance

DEECD does not apply any data quality assurance to the VCAMS data supplied by other departments. During the report development process, drafts are supplied to departments to confirm accuracy of the data and supporting analysis, which gives a level of assurance about the accuracy of information contained in SVC reports.

This process does not allow for any issues with the integrity of data that may be unknown to the source department. This is a particular problem when raw data is provided to departments from third party service providers.

VAGO's 2013 performance audit Addressing Homelessness: Partnerships and Plans found that the Department of Human Services did not validate data provided to it by service providers. Given the multiple sources of data that contribute to VCAMS indicators, it is likely that some draw on unvalidated data from third-party service providers. There is therefore scope for DEECD to strengthen data validation practices to assure the reliability of information included in reports.

2.3.5 Lack of definition of 'most vulnerable children'

A further issue compromising SVC reports' ability to fulfil their role as a report to the Ministers is a lack of definition of, and attention on, the most vulnerable children in the community.

Despite the Act specifying that, in fulfilling its review and reporting function, the Board give particular attention to 'the most vulnerable children in the community', this cohort has not been defined and is not separately identified within SVC reports. Neither the Framework nor VCAMS indicators differentiate the most vulnerable children from the broader population of children.

The 2006 SVC report did, however, identify four priority populations:

• Indigenous children
• children with a disability
• children from culturally and linguistically diverse backgrounds
• children affected by chronic disadvantage.

While not explicitly identifying them as the most vulnerable children in the community, the 2006 report provided a separate section analysing outcomes for these cohorts. However, this approach was not adopted for later reports.

The lack of definition of, and focus on, vulnerable children is an omission that compromises the Board's ability to fulfil its role under the Act, and limits the relevance and appropriateness of SVC reports as a vehicle to report to the Ministers.

Notably, the Victorian Government's Protecting Victoria's Vulnerable Children Inquiry (the Inquiry) highlighted the challenges associated with defining and measuring vulnerability, due to the multiple influences that can make children vulnerable and the shifting nature of these influences over their lives. The Inquiry recommended that the government develop and adopt a whole-of-government vulnerable children and families strategy, with a key element being a definition of vulnerable children and young people.

It made further recommendations that information about vulnerable children and services delivered to them should be included in SVC reports, specifically:

• performance against objectives of the vulnerable children's strategy
• results of a questionnaire about health and wellbeing of students transitioning from childhood to adolescence
• experiences of vulnerable children from culturally and linguistically diverse communities.

These recommendations suggest that there is a gap in knowledge about how vulnerable children are faring, and there is a need for SVC reports to more effectively convey information about their status in Victoria.

Since February 2012, when the Inquiry report was published, the Board has been working to develop a vulnerable children and families strategy. It is expected that this will include a definition of vulnerable children, and result in additional information being included in SVC reports, however, the precise nature of these changes is still being resolved. The strategy is expected to be released by mid-2013.

2.3.6 Evaluating state of Victoria's children reports

DEECD has not comprehensively evaluated SVC reports either as mechanisms to fulfil the Board's reporting role under the Act, or as resources for non-government stakeholders. However, while there has been no comprehensive evaluation, DEECD performed some limited evaluations of reports.

The Framework has been in place since 2005 and VCAMS since 2007, yet neither has been evaluated or reviewed to determine their ongoing relevance and value for SVC reports.

Understanding the value of state of Victoria's children reports to non-government stakeholders

All SVC reports have been publicly released with the stated aim of contributing to program development and service delivery. However, DEECD has not consulted with non‑government stakeholders when developing the reports.

This means that SVC reports have been developed without any input from one of the key audiences of the report. Given that non-government organisations play an increasing role in the delivery of services to young people, this is a significant gap.

DEECD has advised that the SVC report's primary role is to report to the Ministers on behalf of the Board; therefore, meeting the Board's needs is the only major consideration during its development. They also note that the Act imposes no obligation on the Board or the Ministers to publicly release SVC reports, but rather that this decision is at the Ministers' discretion.

This does not fully recognise that all SVC reports have been publicly released with a specific purpose of being a resource for stakeholders outside government—a resource that stakeholders consulted for this audit consistently stated they value. Achieving this purpose depends on SVC reports' inclusion of information that is useful and meaningful to those stakeholders.

DEECD has no processes to gain insight into how useful SVC reports are to non‑government stakeholders without consulting with them during development or comprehensive evaluation of the reports.

DEECD's evaluation of state of Victoria's children reports

The state of Victoria's children 2008: A report on how children and young people in Victoria are faring

DEECD developed a brief on-line questionnaire to elicit user feedback about the 2008 SVC report. However, this was poorly administered and received few responses.

The state of Victoria's children 2009: Aboriginal children and young people in Victoria

The 2009 state of Victoria's children report focused on Indigenous children and young people, and was widely promoted through a series of community workshops aimed at facilitating understanding of the data within the report. Some basic evaluation of the workshops was undertaken, which found that they increased the usefulness of the data and the data's potential application.

The state of Victoria's children 2010

DEECD commissioned a consultant to conduct interviews with users of the report, specifically 10 government stakeholders and five non-government stakeholders. The aim was to assess whether the 2010 SVC report was a useful evidence base for service planning and policy development by government and community sector organisations. The consultant's report acknowledged that the SVC report was a worthwhile and valuable reference document. The quality of the data and the rigour with which the information was presented within the 2010 report was noted by all respondents.

Notably, the evaluation found that there was a desire to have the data presented online, and more active promotion of the report.

The state of Victoria's children report 2011: A report on how children and young people in rural and regional Victoria are faring

DEECD is in the early stages of promoting the 2011 report through a series of regional workshops similar to those that took place to promote the 2009 report. Evaluation of these workshops is also proposed, and will be important to understand the value of reports to stakeholders.

While each of DEECD's past evaluation activities have provided some insight, the lack of a thorough overall evaluation means DEECD has little understanding of whether the SVC is meeting the needs of users or whether the investment required to produce it has been a prudent use of public resources.

2.3.7 Measuring the impact of state of Victoria's children reports on policy and programs

In addition to a lack of comprehensive evaluation, DEECD has not undertaken any work to measure the impact of SVC reports on government policies and programs. More specifically, DEECD does not know whether SVC reports have been a catalyst for coordinated, cross-government interventions designed to improve outcomes for children.

Through stakeholder consultations, we observed that SVC reports have influenced some policy development. For instance, work has been undertaken in strengthening education of children in out-of-home-care which has led to the Out-of-Home Care Education Commitment: A Partnering Agreement between the Department of Human Services, Department of Education and Early Childhood Development, the Catholic Education Commission of Victoria and Independent Schools Victoria'. A key objective of the Partnering Agreement is to improve educational outcomes for children and young people in out-of-home care, which is directly based on evidence in the 2008 SVC report.

Additionally, the Aboriginal Affairs Taskforce produced the Victorian Aboriginal Affairs Framework 2013–2018, which used information in the 2009 SVC report regarding outcomes for Aboriginal children as compared to other Victorian children, particularly the indicators developed for the Framework.

It is likely that SVC reports have led to other action, including action within agencies outside government. However, neither DEECD nor the Board has made attempts to evaluate this. Given the original intent of the Board and the importance of overcoming impediments to coordination across agencies to improve outcomes for children, understanding the role SVC reports play in achieving this is an important priority.

Recommendations

The Children's Services Coordination Board should:

1. review and clarify the role and purpose of state of Victoria's children reports in consultation with non-government stakeholders

The Department of Education and Early Childhood Development should:

2. expedite completion of the project to make Victorian Child and Adolescent Monitoring System data publicly available online
3. evaluate the Victorian Child and Adolescent Monitoring System to determine its ongoing relevance for state of Victoria's children reports
4. establish agreed mechanisms to assure the quality of information being reported in state of Victoria's children reports.