Introduction
This evidence summary seeks to address the following question relating to measuring the effectiveness of interventions in social work: how do we know whether someone has progressed or improved (or potentially regressed) as a result of the involvement from our social work team?
About the evidence presented below
To find relevant evidence for this Outline we searched academic databases containing publications in the fields of social work and social care (e.g. ASSIA and Sociological Abstracts), databases that contain academic and grey literature in the field of social care (e.g. SCIE and Google Scholar) and used the Iriss National Social Services Search, Scottish Government website and websites of other key organisations to search more widely for relevant grey literature.
We used search terms including effects, evaluation, impact and outcomes alongside social work, social care, social services, interventions and other related terms to identify relevant evidence and information about examples of methods of impact identification at the individual level in the context of direct social work practice and social services. We included international research and publications from academic and grey literature dating between 2010-2018, and earlier material that we considered particularly relevant to the original enquirers' needs if identified during citation mining of publications identified through searches.
We identified a large body of literature relating to the value of effectively measuring the impact of social work interventions, particularly in the area of children and families, and many different approaches to doing so. There is no consensus around the best way to approach impact measurement or any one standard approach. In their scoping review of the effectiveness of social work with adults, Moriarty and Manthorpe (2016) discuss how there is no one definition of effectiveness and argue that it depends on who is doing the measuring and what outcomes are being measured. It can take the form of aspects such as cost effectiveness, impact on service users and carers (for example, changes to quality of life), and user and carer views.
For this Outline, we focused on measures of effectiveness and impact relating to changes in the lives of the individuals receiving any form of social work intervention. In their review, the authors identified a particular lack of evidence around the outcomes of social work with adults with learning disabilities, which may be an indication of the unique challenges posed by collecting meaningful data about the changes established through social work intervention with people with learning disabilities.
Accessing resources
We have provided links to the materials referenced in the summary. Some materials are paywalled, which means they are published in academic journals and are only available with a subscription. Some of these are available through the The Knowledge Network with an NHS Scotland OpenAthens username. The Knowledge Network offers accounts to everyone who helps provide health and social care in Scotland in conjunction with the NHS and Scottish Local Authorities, including many in the third and independent sectors. You can register here. Where resources are identified as 'available through document delivery', these have been provided to the original enquirer and may be requested through NHS Scotland's fetch item service (subject to eligibility).
Where possible we identify where evidence is published open access, which means the author has chosen to publish their work in a way that makes it freely available to the public. Some are identified as author repository copies, manuscripts, or other copies, which means the author has made a version of the otherwise paywalled publication available to the public. Other referenced sources are pdfs and websites that are available publicly.
Background
Generic locality social work services in Scotland support people with a wide range of needs and situations. How the impact of the approaches and forms of intervention and support provided by social workers is captured effectively may be dependent on the nature of the intervention and the person or people with whom the service is working. No one approach is likely to suit all. This Outline therefore identifies a range of approaches and methods to capturing social services impact present in the evidence around this topic.
Examples of the kinds of issues social workers in generic locality services work with include physical and mental illness and disability, substance misuse and addiction and domestic violence. This wide range of work has been identified as an evaluative challenge:
Social work addresses some of the most complex and intractable human and social problems: poverty, mental illness, addiction, homelessness, and child abuse. Our field may be distinct among professions for its efforts to ameliorate the toughest societal problems, experienced by society's most vulnerable, while working from under-resourced institutions and settings. Members of our profession are underpaid, and most of our agencies lack the data infrastructure required for rigorous assessment and evaluation. (Proctor 2017)
Improving the quality of outcomes data for key areas of social work is a challenge for local and central government (Social Work Inspection Agency 2010). It is difficult to identify whether interventions have prevented events from occurring, and to attribute outcomes to specific interventions. It is therefore difficult to identify whether an individual's circumstances and outcomes have changed in a positive or negative way, or at all, as a result of the involvement from social work teams. However, there is a long-standing interested in collecting information and demonstrating improvements. For example, Moriarty and Manthorpe (2016) cite Mary Ellen Richmond's 1917 call for social workers to "subject the processes of their task to critical analysis" and to "measure their work". They go on to provide a history of social work evaluation. This Outline looks at suggested effective methods of capturing the impact of social work interventions and identifies sources that make recommendations for good practice or different approaches.
Defining impact of interventions
There is a growing need within social work to show that the services they provide are effective and lead to improved outcomes. In this context 'impact' is the 'influence' or 'difference' brought about by the interventions social workers have identified as appropriate.
We have considered the definition of interventions themselves as "practitioner behaviors that can be volitionally manipulated by practitioners (used or not, varied in intensity and timing), that are defined in detail, can be reliably measured, and can be linked to specific identified outcomes" (Proctor 2017).
Capturing impact
Cook and Morton (2016) describe the drive to capture evidence of impact in Scottish social services and the need to be able to do this in-house:
Understanding what a programme does, how it works and the difference it makes to outcomes is a fundamental part of good practice in public service delivery. Working in this way is vital to the ongoing development and improvement of services, systems and supports as well as demonstrating accountability to funders and stakeholders. In the current context of austerity, organisations, projects and teams experience increasing pressure from funders to evidence the impact of their work, who in turn need to demonstrate effective use of resources.
While working with external evaluation specialists can be valuable, it is not sufficient if an organisation is to continually learn and improve. Approaches to capturing evidence on impact need to be embedded in everyday work and capacity and skills built within teams to make sense of and respond to emerging findings.
Critical perspectives
The need to identify what impact interventions has had is considered by many as a contentious area of social work. From a critical perspective, the increasing pressure to demonstrate 'impact' can be seen as a symptom of deprofessionalisation (Malin 2017) based on a "narrow ends-means rationality" (Webb 2001) that does not appropriately represent or capture the complexity of the impact of social work interventions in people's lives. Webb (2001) argues that "the idea that a formal rationality of practice based on scientific methods can produce a more effective and economically accountable means of social service" is in part based on a misconception of the nature of social work and that the value-laden beliefs around the objective nature of evidence-based practice may not be a realistic approach to take. Webb advocates for the abandonment of mechanistic approaches:
Rather than conceiving of social workers and clients as social billiard balls, passively bounding through a world of causal and objective relations, they should be conceived as dynamic, interconnected agents who make decisions according to heuristic devices which are bound up with specific cultural formations.
If this is not done, he warns that social work's role in moral and political discourse will be neutralised and its professional autonomy will be undermined, suggesting that the "requirements of evaluative effectiveness may undermine traditional social work practice and further legitimate a harsher managerialist ethos of performance culture in social work" (Webb 2001).
Selected additional pieces that present a critical perspective are provided below for further reading.
Hood, Rick (2016) Be careful what you measure in child protection because it influences practice. Community Care (website)
The current system of measuring success is flawed, argues Dr Rick Hood, senior lecturer in social work at Kingston University and St George's.
Draper, Lucy (2001) Being evaluated: a practitioner's view. Children and Society, 15(1), pp.46-52 (paywalled)
Looks at the impact of evaluation exercises on people and services being evaluated. Considers whether evaluators are measuring the right things, whether the resulting evidence is valid, and the impact of evaluation on services. Identifies features of research and evaluation that are most likely to lead to improvements in services for children and families.
Holligan, C et al. (2014) The 'care' of children in need in contemporary Scotland: the role of positivism and performance indicators in official imaginings of childhood and wellbeing. Scottish Journal of Residential Child Care, 13(1) (open access)
Improving the wellbeing of children is an ambition of governments worldwide. This has led to increased activity to measure the implementation of policies intended to achieve this. This article argues that this is currently limited through the reliance on statistically-driven methods and that there needs to be a fundamental change in how policies are assessed. They examine this within the current policy context for vulnerable children in Scotland. They outline the policies and frameworks behind current strategies of official data collection and argue that whilst useful for serving bureaucratic needs critical to national governance and planning resource allocation, they overlook the children in their unique social settings.
Evidence
Although many services had adopted performance frameworks, research indicates that in 2010 few had reached the stage of having comprehensive systems in place to monitor and improve all aspects of social work performance (Social Work Inspection Agency (SWIA) 2010). Although further work may have been conducted in this area, we did not identify examples of up to date practice guidelines or examples from local authorities in our searching.
This section identifies some evidence that may support services to implement strategies for identifying outcomes of social work. In terms of overall principles for improving performance management, SWIA (2010) make recommendations around:
- Improving data quality
- Benchmarking
- Reporting of performance
- Taking action to improve performance
- Quality assurance
- Self evaluation
Overall, a key message from the evidence presented below is that the broad range of services provided, outcomes sought and people being supported by social services may not be suited to a one size fits all approach to identification of impact of social work and tailored approaches to measurement depending on each context may be more effective, accurate and robust in determining what changes different interventions have led to on an individual level. The evidence is divided into two main sections: theoretical approaches to capturing the impact of social work interventions; and approaches taken to capturing the impact of interventions in specific areas of social work including prevention, children and families, intimate partner violence, and drugs and alcohol.
Theoretical approaches
Grand challenges
The American Academy of Social Work and Social Welfare has applied the Grand Challenges concept to the field of social work. The Grand Challenges Executive Committee identified 12 Grand Challenges for Social Work from a larger set of challenges identified from a broad call for ideas. The 12 Grand Challenges are:
- Ensure healthy development for all youth
- Close the health gap
- Stop family violence
- Advance long and productive lives
- Eradicate social isolation
- End homelessness
- Create social responses to a changing environment
- Harness technology for social good
- Promote smart decarceration
- Build financial capability for all
- Reduce extreme economic inequality
- Achieve equal opportunity and justice
Proctor (2017) suggests that these Grand Challenges "provide a framework for communicating social work's goals" and reflect the added value of social work for wider social wellbeing and social justice. The framework requires the identification of specific benchmarks which can be used to identify to what extent interventions have achieved specific goals.
Implementation
Proctor, E (2017) The pursuit of quality for social work practice: three generations and counting. Journal of the Society for Social Work and Research, 8(3), pp.335–353 (open access)
This invited article is based on the 2017 Aaron Rosen Lecture presented by Enola Proctor at the Society for Social Work and Research 21st Annual Conference—"Ensure Healthy Development for All Youth"—held January 11–15, 2017, in New Orleans, LA. The paper describes the author's career of work, which seeks to improve the quality of social work practice by pursuing answers to three questions:
1. What interventions and services are most effective and thus should be delivered in social work practice?
2. How do we measure the impact of those interventions and services? (That is, what outcomes do our interventions achieve?)
3. How do we implement the highest quality interventions?
It demonstrates the substantive and methodological progression across the three questions, assesses what the field has learned, and forecasts a research agenda for what it still needs to learn. It provides a potentially useful overview for background context to the measurement of social work impact.
Craig, P et al. (2008) Developing and evaluating complex interventions: The new Medical Research Council guidance. British Medical Journal, 337, a1655 (open access)
Complex interventions are widely used in the health service, in public health practice, and in areas of social policy that have important health consequences, such as education, transport, and housing. They present various problems for evaluators, in addition to the practical and methodological difficulties that any successful evaluation must overcome. In 2000, the Medical Research Council (MRC) published a framework to help researchers and research funders to recognise and adopt appropriate methods. The framework has been highly influential, and the accompanying BMJ paper is widely cited. However, much valuable experience has since accumulated of both conventional and more innovative methods. This has now been incorporated in comprehensively revised and updated guidance recently released by the MRC. In this article the authors summarise the issues that prompted the revision and the key messages of the new guidance.
Strengths-based approach
Green, BL et al. (2004) The strengths-based practices inventory: a tool for measuring strengths-based service delivery in early childhood and family support programs. Families in Society, 85(3), pp.326-334 (paywalled)
Examines US research into the development of the Strengths-Based Practices Inventory (SBPI), a tool to evaluate service delivery, and interpret results and outcomes of early childhood prevention and family support programmes.
Personal outcomes approach
Daly, E (2017) Outcomes & CO. Iriss (website, pdf and physical resource)
Outcomes & CO: conversation openers was created to support practitioners and managers across the social services sector reflect on their views, values and experiences of a personal outcomes approach. Personal outcomes are defined as what matter to people accessing support and their unpaid carers (adult and young).
Outcomes & CO provides a framework for open, honest conversations about personal outcomes. It does this by supporting reflective practice and building common understanding. It is flexible and can used or adapted for a range of audiences, including health and housing.
This tool builds on a range of resources developed by Iriss and partners to support an outcomes-focused approach in the social services sector, in particular, the Leading for Outcomes series (Iriss, 2010-2013) and the Values and Principles Paper developed by the Personal Outcomes Collaboration. A list of other resources referenced in the tool are also available.
Anticipated outcomes:
- Increased awareness and understanding of the key components involved in a personal outcomes approach
- Improved understanding of the landscape in which a personal outcomes approach exists
- Improved awareness and understanding of the views, values and experience of a personal outcomes approach for individuals and teams
- Shared understanding of an outcomes-focused approach across teams and between practitioners, the people they support and their unpaid carers
- Increased confidence about understanding and practising an outcomes-focused approach
- Improved reflective practice
Miller, E (2017) Review of evidence on personal outcomes relevant to the Carers (Scotland) Act 2016 (pdf)
This resource provides an update to the "Talking Points Personal Outcomes Approach: Practical Guide" (Cook and Miller 2012), which summarised learning from implementation of an outcomes approach in Scotland from 2006 to 2012.
Miller, E (2012) Measuring personal outcomes: challenges and strategies. Iriss Insight 12 (website and pdf)
This report considers some of the challenges of measuring outcomes and emerging responses to these.
Key Points:
- A focus on personal outcomes offers the potential to refocus on what matters to people who use services, with potential benefits for the individuals involved, staff and organisations
- It is important to be clear about the purpose of measuring outcomes. In particular, whether the measurement is primarily for improvement purposes or for judgement - in practice it may well be both
- There is potential to link outcomes measurement to the organisational value base and a range of approaches and tools are emerging to support this
- There are many identified challenges of measuring outcomes, but the evidence highlights various recommendations and strategies that can help
- Outcomes tools are sometimes designed with a very specific user group in mind, whilst others can be used more generally with different user groups
Miller, E and Daly, E (2013) Understanding and measuring outcomes: the role of qualitative data. Iriss (pdf)
This guide has been developed to support the collection and use of personal outcomes data. Personal outcomes data refers to information gathered from people supported by health and social services and their unpaid carers about what's important to them in their lives and the ways in which they would like to be supported. The guide is divided into three parts:
- Part 1 explores the links between an outcomes approach and qualitative data, why qualitative data is important and what it can achieve
- Part 2 outlines a practical approach exploring collecting, recording, analysing and reporting qualitative data about personal outcomes
- Part 3 highlights different approaches to qualitative analysis through case studies of people using qualitative data about outcomes for the first time
Realist evaluation
Kazi, M (2003) Realist evaluation for practice. The British Journal of Social Work, 33(6), pp.803-818 (paywalled or author copy)
This paper outlines the new, emerging realist paradigm in evaluation research, and applies it to social work practice. This paradigm has the potential for a white box evaluation that not only systematically tracks outcomes, but also the mechanisms that produce the outcomes, the contexts in which these mechanisms are triggered, and the content of the interventions (or the generative mechanisms introduced by a programme). Two examples are provided, both studies with an extensive use of single subject designs by practitioners within a realist paradigm. This article is based on the authors invited keynote address at the Ohio State University's Thirteenth National Symposium on Doctoral Research in Social Work, 6 April 2001.
Other approaches
Van Dop et al. (2016) Measuring empowerment: development and validation of the Service User Psychological Empowerment Scale. Journal of Social Service Research, 42(5), pp.651-664 (paywalled)
Current registrations and measurements used in social services are criticised for not providing an accurate reflection of the social work practice. The Service User Psychological Empowerment Scale (SUPES) was developed with input from both service users and social workers in order to provide an alternative measure. The SUPES is a 28-item scale that can be used to measure the intrapersonal, interactional, and behavioural dimensions of psychological empowerment among service users. Through factor analysis, the SUPES was found to be valid and reliable in assessing the different dimensions of psychological empowerment among service users. (Abstract from SCIE)
Social work areas
General
Gardner, F (2000) Design evaluation: illuminating social work practice for better outcomes. Social Work, 45(2), pp.176-182 (paywalled or author copy)
One of the difficulties in social work lies in finding forms of evaluation that suit the various kinds of work people actually do. There is a need for methods that allow a valid and rigorous evaluation of process as well as outcomes, particularly in new areas of work. A valuable example of such evaluation evolved at St. Luke's, a voluntary agency in Victoria, Australia, that used design evaluation to articulate a new model of practice. Design evaluation involved having the evaluator work with staff to describe work practices as a series of stages with associated principles. This process of documenting, clarifying, and illuminating the model led to its progressive refinement and concurrent improved services delivery to clients. Testing the program model with clients provided some valuable feedback about its effectiveness. This form of evaluation is complementary to outcomes evaluation and can also provide the information needed for a subsequent outcomes evaluation.
Kazi, M (2000) Realist evaluation for evidence based practice. Journal of Social Work Research and Evaluation, 1(2), pp.101-109 (author copy)
This article is based on the experiences of social workers & other related practitioners in England, working in partnership with the Center for Evaluation Studies at the U of Huddersfield. The main purpose of this partnership is the development of evaluation strategies that can be incorporated into social work practice -- strategies that not only demonstrate the effectiveness of practice but also contribute directly to the development of models of social work intervention (Kazi, 1998). These strategies range from those that systematically track the outcomes of practice at a minimum, to others that can also address the wider dimensions of practice such as the systematic tracking of the content of interventions as well as the contexts in which practice takes place.
Prevention
Dillon, K et al. (2014) Evaluating prevention programs: tip sheet for measuring what doesn't happen. Saint Paul, Minn: Wilder Research (pdf)
This tip sheet briefly considers the challenges in evaluation prevention programmes. Sets out four steps for effective evaluation: define what prevention means; specify a programme theory; develop a process evaluation; develop an outcome evaluation.
Carers
Miller, E (2017) Review of evidence on personal outcomes relevant to the Carers (Scotland) Act 2016 (pdf)
This review identifies several long-term programmes on outcomes focused planning, some specific to carers, based in England, Wales, Canada, Sweden and Scotland. Key features of early work by the Social Policy Research Unit (SPRU) at York University influenced later work in Scotland and include adoption of similar frameworks for service users and carers, the inclusion of both process and quality of life outcomes, and the exchange model of assessment, which incorporates the views of service users, carers, practitioners and agencies in negotiating and agreeing outcomes. The SPRU programme adopted 'whole systems thinking' to reflect awareness that resolution of many health and social problems, and achievement of outcomes for people, lie beyond the ability of any one practitioner or agency. It was acknowledged that systems are complex networks of inter-relationships, which links to the recent emphasis on complex adaptive systems highlighted in the implementation paper that sits alongside this paper. A 'paradigmatic leap' in service designed and delivery is required. Literature on outcomes for young carers is more limited but growing and identifies long-term impacts of substantial caring on development and opportunities into adulthood. Separate work is underway focusing on young carers relating to the Act.
Children and families
Bloomfield, L and Kendall, S (2007) Testing a parenting programme evaluation tool as a pre and post-course measure of parenting self-efficacy. Journal of Advanced Nursing, 60(5), pp. 487–493 (paywalled or author manuscript)
This paper is a report of a study to measure changes in parenting self‐efficacy after attending a parenting programme and to determine if these changes are maintained over time. A total of 356 parents took part in the study over 53 parenting programmes in the UK in 2004–2005. Parents completed the evaluation tool as a pre‐ and post‐course measure of parenting self‐efficacy and at 4‐month follow‐up. The methodology may be particularly relevant.
Krohn, J (2015) Beyond outcome measures in child protection: using feedback to direct and evaluate social work practice. Practice: Social Work in Action, 27(2), pp.1-17 (paywalled)
This study used a qualitative methodology employing focus groups in a London borough to explore new ways of judging and improving the quality of social work practice. Following analysis six themes were identified: outcome measures maintain a sense of urgency; outcome measures are not practice orientated; a change of approach to using outcome measures is needed; a vision of good practice should be cultivated; feedback should be used to measure practice; and audits can be used to measure and improve practice. These findings are used to suggest ways in which reforms underway in the UK and around the world can be utilised at a local level to bring policy into more sympathetic alignment with social work practice.
Ferguson, H (2014) What social workers do in performing child protection work: evidence from research into face-to-face practice. Child & Family Social Work, 21(3) (paywalled or author copy)
Little research has been done into what social workers do in everyday child protection practice. This paper outlines the broad findings from an ethnographic study of face‐to‐face encounters between social workers, children and families, especially on home visits. The social work practice was found to be deeply investigative. Children's bedrooms were routinely inspected and were the most common place where they were seen alone. A high proportion of children were not seen on their own because they were too young and the majority of the time was spent working with parents and children together. Small amounts of time were spent with children on their own and some first encounters were so rushed that social workers did not even introduce themselves to the child. This arose from two key factors: firstly, organisational pressures from high workloads and the short timescales that social workers were expected to adhere to by managers and Government; secondly, practitioners had varying levels of communication skills, playfulness and comfort with getting close to children and skills at family work. Where these skills and relational capacities were present, social workers were found to have developed deep and meaningful relationships with some children and families, for whom it was apparent that therapeutic change had occurred.
Freiberg, K et al. (2014) The Parent Empowerment and Efficacy Measure (PEEM): a tool for strengthening the accountability and effectiveness of family support services. Australian Social Work, 67(3), pp.405-418 (paywalled or author preprint)
This article describes the development, validity, and reliability of the Parent Empowerment and Efficacy Measure (PEEM). Development was guided by theory and by challenges faced by Pathways to Prevention family support staff who required a short, practical, and reliable measure of parent empowerment. The measure's psychometric properties were tested using data from 866 parents of children aged 5 to 12, living in high to low socioeconomic status areas. The instrument measured parents confidence to be a good parent and their capacity to connect with informal and formal networks (ie when to seek help and how to access support and the ability to be part of a mutually supportive community or social network). Principal factor analysis revealed a strong general dimension with high internal consistency that correlated at 0.60 or more with three validation measures, as well as the existence of two hypothesised subfactors: efficacy to parent and efficacy to connect. Test–retest reliability (n = 200) was 0.84. The study concludes that PEEM exhibits excellent convergent and concurrent validity and is a reliable tool for use in planning services, monitoring participant progress, and evaluating program effectiveness. (Abstract from SCIE)
Jopling, M and Vincent, S (2014) Evaluation of reach of play@home: final report. University of Wolverhampton (pdf)
Play@home is a universal early years' play intervention programme, designed for children aged 0-5. It provides age and stage appropriate information and activities for parents/carers and children in three books.
Luckock, B et al. (2017) The Islington 'Doing what counts: measuring what matters': evaluation report. Department for Education (pdf)
The report evaluates the 'Doing What Counts (DWC) and Measuring What Matters (MWM)' Project, which aimed to improve the impact of direct social work practice with children and families referred for a statutory assessment of need (CIN) in the London Borough of Islington. A novel Motivational Social Work (MSW) practice methodology was developed and implemented as the core offer for CIN cases. Where need and risk were complex, an enhanced offer added in-house multi-professional team support to the core MSW intervention. Practice evaluation findings were fed back to practitioners (through a coaching relationship) and managers (in practice reports for the team and service) in order to form the basis of a new model of continuous practice improvement. Key findings include: parents reported positively about the quality of practice, and the Motivational Social Work methodology was shown to improve practitioner skill and confidence – but there is as yet insufficient evidence to demonstrate improved outcomes for children; novel models of practice evaluation, improvement and performance management, which focus attention on quality and impact, benefit significantly in their design, implementation and refinement when embedded researchers work alongside local authority project and service leads; whole service re-design consumes significant amounts of time and money without the guarantee of demonstrable returns on investment in the short and medium term. The evaluation suggests that that MSW might be an effective approach to enhancing practice impact for children in their family and wider social lives. (Abstract from SCIE)
Margolis, R et al. (2017) What are the best tools to measure outcomes for children and families? NSPCC (website)
In this piece the authors share their tips on how to evaluate services in practice, along with some of the tools they have found most useful to measure outcomes for children and families.
McGuffin, Valerie (2002) Family support: measuring the benefits to families. Child Care in Practice, 8(4), pp.251-261 (paywalled)
This article outlines the process undertaken by the family support agency Home-Start, to develop and implement a monitoring and evaluation format that would measure the outcomes for families who had been supported by the agency. Summarises organisational history, key developmental stages, content and format development, recommendations, implementation and benefits to families and volunteers.
Mulroy, E and Lauber, H (2004) A user-friendly approach to program evaluation and effective community interventions for families at risk of homelessness. Social Work, 49(4), pp.573-586 (paywalled or author copy)
This article demonstrates how a user-friendly evaluation of a federally funded homeless prevention program using an action research approach— and using a logic model as the analytic framework—informed multiple stakeholders, including members of Congress, other decision makers, and Family Center practitioners. The program's target population was very low-income families at risk of being evicted from public housing. The authors discuss the methods used, the application of the logic model, and the study's findings as they unfolded in four phases: (1) logic modeling as program planning, (2) conceptualizing the intervention, (3) delineating implementation processes, and (4) determining the range of client outcomes. Implications for social policy, social work practice, and evaluation research are discussed.
Sinha I et al. (2008) A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Medicine 5(4), e96 (open access)
In clinical trials the selection of appropriate outcomes is crucial to the assessment of whether one intervention is better than another. Selection of inappropriate outcomes can compromise the utility of a trial. However, the process of selecting the most suitable outcomes to include can be complex. The authors' aim was to systematically review studies that address the process of selecting outcomes or outcome domains to measure in clinical trials in children. They found that very few studies address the appropriate choice of outcomes for clinical research with children, and in most paediatric specialties no research has been undertaken. Among the studies assessed, very few involved parents or children in selecting outcomes that should be measured, and none directly involved children. Although this paper focuses on clinical interventions, the focus on the need to determine the best way to involve children and their parents in the selection of outcomes may be relevant in a social work context.
Rhodes, H (2009) Knowing what you do works: measuring your own effectiveness with families, parents and children: a short guide. Family and Parenting Institute (FPI) (pdf)
Guide for practitioners to measure their effectiveness with families, parents and children. Includes the following topics: pre- and post-measuring; reliability and validity; finding the right tools for the job; ethical and practical issues; information technology and measurement tools; helping families to answer the questions and what not to do. It is designed to help individual workers and whole services understand the terms and issues within the act of measuring and how to go about it.
Roberts, J et al. (2014) Measuring what matters: a guide for children's centres. UCL Institute of Health Equity (pdf)
Guidance to support children's centres, local authorities, health and employment services to implement and measure the impact of a whole-system approach to improving outcomes for children. Presents an evidence and measurement framework for children's centres which will help measure outcomes for children and families. Areas of focus within the framework include effective outreach; children developing well and positive parenting.
Philip, G and O'Brien, M (2017) Are interventions supporting separated parents father inclusive? Insights and challenges from a review of programme implementation and impact. Child and Family Social Work, 22(2), pp.1114-1127 (paywalled or author manuscript)
This paper reviews divorce-related parenting programmes, assessing the extent to which fathers are included and whether father inclusion influences outcomes. The paper also discusses limitations of the research evidence and implications for future intervention and evaluation design. Thirteen programmes met the criteria in the review period 2005-2012 but only four had been evaluated using randomized control trials or with independent measures from mothers and fathers. Analysis of these four programmes shows modest evidence of reduced couple-conflict, improved coparenting and some evidence of improved child outcomes. Key issues raised are the need for improving the quantity and quality of demographic data about fathers; the importance of incorporating analysis by gender of parent into evaluation design and the value of developing and routinely using father-related indicators to measure programme impact on men's parenting, fathering and co-parental relationships.
Wallace, E et al. (2011) Evaluation of Flying Start: findings from the baseline survey of families: mapping needs and measuring early influence among families with babies aged seven to 20 months: main report. Welsh Government Social Research Social Research (pdf)
Findings from a survey evaluating the Flying Start programme, which aims to improve child outcomes in deprived areas in Wales by providing targeted services to families with children under four. Analysis indicates that Flying Start has been shown to be successful in areas including: increased access to health visiting services; and increased focus on early language development.
Morris, C et al. (2014) Informing the NHS Outcomes Framework: evaluating meaningful health outcomes for children with neurodisability using multiple methods including systematic review, qualitative research, Delphi survey and consensus meeting. Health Services and Delivery Research, 2(15) (pdf)
The identification of suitable outcome measures will improve the evaluation of integrated NHS care for the large number of children affected by neurodisability, and has the potential to encourage the provision of more appropriate and effective health care. This research sought to appraise the potential of patient-reported outcome measures (PROMs) for children and young people with neurodisability. The study found agreement between young people, parents and professional regarding the most important health outcomes regarding neurodisability for identification through patient-reported outcome measures (PROMs). A systematic review and focus groups were used to examine the effectiveness of PROMs for measuring the specified outcomes, but found that none of the candidate PROMs in the review adequately captured all of these.
Families with complex needs
Bloomfield, L et al. (2010) Supporting parents: development of a tool to measure self-efficacy of parents with learning disabilities. British Journal of Learning Disabilities, 38(4), pp.303–309 (paywalled)
This article reports the development of a tool to measure the self‐efficacy of parents with learning disabilities, which will help to evaluate parenting initiatives specifically aimed at this parent group. A tool to measure parenting self‐efficacy (TOPSE) has been adapted to be accessible to parents with learning disabilities. Eighteen parents took part in the study to complete and comment on the tool with the help of a researcher from the community learning disabilities team. This tool, which consists of 45 self‐efficacy statements, now needs to be tested on a larger sample of parents with learning disabilities. The use of the tool and its measurement of impact may be of relevance.
Coates, D and Howe, D (2016) An evaluation of a service to keep children safe in families with mental health and/or substance abuse issues. Australasian Psychiatry, 24(5), pp.483–488 (paywalled)
The objective of this study was to evaluate a specialised service designed to improve parenting capacity, child safety and family functioning in the context of parental mental health, drug and alcohol and child protection concerns. Client data was collected over a period of 3 years and 3 months, including demographic characteristics, service usage, presenting issues and pre and post revised North Carolina Family Assessment Scale (NCFAS-G) scores. This methodology may be of relevance.
Intimate partner violence
Howarth, E et al. (2015) The effectiveness of targeted interventions for children exposed to domestic violence: measuring success in ways that matter to children, parents and professionals. Child Abuse Review, 24(4), pp.297-310 (open access)
In this paper the authors consider the issue of measuring outcomes that reflect the needs of those using the interventions. They consider this issue in relation to trials of interventions for children exposed to domestic violence and abuse (DVA). They drew on data collected as part of a larger study to consider whether the types of outcomes measured in clinical trials reflect: (1) the perceived benefit of interventions reported in qualitative evaluation studies; and (2) the views of parents, professionals and young people as to what constitutes a 'good outcome'. They found that trials most frequently evaluated changes in children's symptoms and disorders, whereas children and parents, along with practitioners, had broader concepts of success that extended beyond narrow health-focused outcomes. A number of studies measured other types of outcomes, although there was inconsistency in the types of outcomes that were measured. Based on these findings, the authors discuss the need to reach consensus on an expanded set of outcomes to be measured in child-focused DVA trials.
O'Doherty LJ et al. (2014) Selecting outcomes for intimate partner violence intervention trials: Overview and recommendations. Aggression and Violent Behavior, 19, pp.663–672 (paywalled or author copy)
Developing robust evidence with potential to inform policy and clinical practice around intimate partner violence requires greater clarity and consistency across studies in the selection and use of outcomes to evaluate interventions. Drawing on systematic reviews and individual trials aimed at reducing abuse and improving women's health, the authors discuss critical issues in respect of outcomes. They discuss primary, secondary, intermediate and proxy outcomes and measures used to evaluate interventions for women who experience IPV. They offer recommendations about which outcomes to assess and approaches to doing so within the context of trials in health care settings.
Westmarland, N and Kelly, L (2013) Why extending measurements of 'success' in domestic violence perpetrator programmes matters for social work. British Journal of Social Work, 43(6), pp.1092-1110 (open access)
There is an ongoing disagreement internationally about the effectiveness of interventions aimed at domestic violence perpetrators. Part of this disagreement stems from the failure to consider a broad range of potential outcomes, with most research focusing on an overly narrow understanding of what 'success' means (as no subsequent police callouts or incidents of physical violence). A total of seventy-three interviews with men on programmes, their partners/ex-partners, programme staff, and funders and commissioners were undertaken to explore what 'success' meant from their perspectives. Findings reveal that success needs to be redefined and connected not just to criminal justice, but also to health and social care agendas.
Drugs and alcohol
Cree, V et al. (2016) The challenge of measuring effectiveness in social work: a case study of an evaluation of a drug and alcohol referral service in Scotland. British Journal of Social Work, 46(1), pp.277-293 (paywalled or author repository copy)
This article explores the challenge of measuring effectiveness in social work, building from an evaluation of a service for those with drug- and alcohol-related problems in Scotland conducted in 2012. Drug and alcohol misuse have long been recognised as major issues for the Scottish economy, health service and population, affecting high numbers of individuals, families and communities in Scotland. For this reason, it is vital that the services provided give 'best value' for service users and the country as a whole. But what is an effective service and how might this be measured? The authors argue that demonstrating social work effectiveness is always difficult, but that the complex and interconnected nature of drug and alcohol problems makes it even more difficult to isolate the effectiveness of one intervention from another and from the context in which it is located. (Abstract from SCIE)
Hospice social work
Hansen, AG et al. (2015) Social work assessment notes: a comprehensive outcomes-based hospice documentation system. Health and Social Work, 40(3), pp.191-200 (paywalled or author copy)
This article describes the development of an integrated psychosocial patient and caregiver assessment and plan of care for hospice social work documentation. A team of hospice social workers developed the Social Work Assessment Notes as a quality improvement project in collaboration with the information technology department. Using the Social Work Assessment Tool as an organising framework, this comprehensive hospice social work documentation system is designed to integrate assessment, planning, and outcomes measurement. The system was developed to guide the assessment of patients' and caregivers' needs related to end-of-life psychosocial issues, to facilitate collaborative care plan development, and to measure patient- and family-centred outcomes. Goals established with the patient and the caregiver are documented in the plan of care and become the foundation for patient-centred, strengths-based interventions. Likert scales are used to assign numerical severity levels for identified issues and progress made toward goals and to track the outcome of social work interventions across nine psychosocial constructs. The documentation system was developed for use in an electronic health record but can be used for paper charting. Future plans include automated aggregate outcomes measurement to identify the most effective interventions and best practices in end-of-life care. (Abstract from SCIE)
Further reading
The following books may be useful for a wider picture of the position of evaluation in the social work context.
DePoy, E and Gilson, S (2017) Social work research and evaluation: Examined practice for action. London: Sage (Available through NHS Libraries)
This book applies systematically developed research knowledge to social work practice and emphasizes the "doing" of social work as a reciprocal avenue for generating research evidence and social work knowledge. Using the Examined Practice Model, the authors present research as the identification of a problem and then proceed to evaluate the efficacy of social work practice in its resolution. Diverse theories, actions, and sets of evidence from a range of professional and disciplinary perspectives are included to underscore the importance of integrating evaluation and practice in research.
Dickinson, H and O'Flynn, J (2016) Evaluating outcomes in health and social care (Available through NHS libraries)
Recent years have seen a shift in health care and social work that has moved collaborative work to the center of everyday practice. But has that change led to better outcomes for the people who use these social services? Evaluating Outcomes in Health and Social Care takes up that question--as well as the crucial underlying question of how best to measure those outcomes. This new edition brings the book fully up to date with the latest research findings and offers more tools, frameworks, and international examples of best practices to aid practitioners as they evaluate partnerships.
Shaw, I and Lishman, J (1999) Evaluation and social work practice. London: SAGE (Available through NHS Libraries)
Evaluation and Social Work Practice offers a comprehensive treatment of the central issues confronting evaluation in social work that links theory and method to practical applications. Evaluation is an integral part of social work and social care provision, for both practice and service delivery. Evaluation can improve effectiveness and increase accountability and help develop new models of practice and service delivery. The authors argue that evaluation should not just be applied to practice but should be a direct dimension of practice.
Spolander, G and Martin, L (2012) Successful project management in social work and social care: managing resources, assessing risks and measuring outcomes. London: Jessica Kingsley (Available from NHS Libraries)
Project management has become increasingly important in social work and social care. Managers in social work and social care contexts are required to manage a wide range of projects: long-term and short-term; on large and small scales; in partnership with other agencies; and covering a broad range of issues and contexts. This practical handbook sets out the core skills required for the project manager in the context of social work and social care. Topics such as managing resources, assessing risks, and measuring outcomes are covered, as well as how to start and end a project. The book acknowledges the values and ethics inherent to care environments, as well as the business skills necessary for good management. It is underpinned by detailed real-life case studies of project management at different levels and in different settings which demonstrate the ideas in action.
References
American Academy of Social Work and Social Welfare (2018) Grand challenges for social work (website)
Bloomfield, L et al. (2010) Supporting parents: development of a tool to measure self-efficacy of parents with learning disabilities. British Journal of Learning Disabilities, 38(4), pp.303–309 (paywalled)
Bloomfield, L and Kendall, S (2007) Testing a parenting programme evaluation tool as a pre and post-course measure of parenting self-efficacy. Journal of Advanced Nursing, 60(5), pp. 487–493 (paywalled or author manuscript)
Carter, R (2014) Adults chief Lyn Romeo orders evidence review in bid to 'revitalise' adult social work practice. Community Care (website)
Coates, D and Howe, D (2016) An evaluation of a service to keep children safe in families with mental health and/or substance abuse issues. Australasian Psychiatry, 24(5), pp.483–488 (paywalled)
Cook, A and Morton, S (2016) Critical friend evaluation of the scottish social services council self-directed support workforce development project (pdf)
Craig, P et al. (2008) Developing and evaluating complex interventions: The new Medical Research Council guidance. British Medical Journal, 337, a1655 (open access)
Cree, V et al. (2016) The challenge of measuring effectiveness in social work: a case study of an evaluation of a drug and alcohol referral service in Scotland. British Journal of Social Work, 46(1), pp.277-293 (paywalled or author repository copy)
Daly, E (2017) Outcomes & CO. Iriss (website, pdf and physical resource)
DePoy, E and Gilson, S (2017) Social work research and evaluation: Examined practice for action. London: Sage (Available through NHS Libraries)
Dickinson, H and O'Flynn, J (2016) Evaluating outcomes in health and social care (Available through NHS libraries)
Dillon, K et al. (2014) Evaluating prevention programs: tip sheet for measuring what doesn't happen. Saint Paul, Minn: Wilder Research (pdf)
Draper, Lucy (2001) Being evaluated: a practitioner's view. Children and Society, 15(1), pp.46-52 (paywalled)
Ferguson, H (2014) What social workers do in performing child protection work: evidence from research into face-to-face practice. Child & Family Social Work, 21(3) (paywalled or author copy)
Freiberg, K et al. (2014) The Parent Empowerment and Efficacy Measure (PEEM): a tool for strengthening the accountability and effectiveness of family support services. Australian Social Work, 67(3), pp.405-418 (paywalled or author preprint)
Gardner, F (2000) Design evaluation: illuminating social work practice for better outcomes. Social Work, 45(2), pp.176-182 (paywalled or author copy)
Green, BL et al. (2004) The strengths-based practices inventory: a tool for measuring strengths-based service delivery in early childhood and family support programs. Families in Society, 85(3), pp.326-334 (paywalled)
Hansen, AG et al. (2015) Social work assessment notes: a comprehensive outcomes-based hospice documentation system. Health and Social Work, 40(3), pp.191-200 (paywalled or author copy)
Holligan, C et al. (2014) The 'care' of children in need in contemporary Scotland: the role of positivism and performance indicators in official imaginings of childhood and wellbeing. Scottish Journal of Residential Child Care, 13(1) (open access)
Hood, Rick (2016) Be careful what you measure in child protection because it influences practice. Community Care (website)
Howarth, E et al. (2015) The effectiveness of targeted interventions for children exposed to domestic violence: measuring success in ways that matter to children, parents and professionals. Child Abuse Review, 24(4), pp.297-310 (open access)
Jopling, M and Vincent, S (2014) Evaluation of reach of play@home: final report. University of Wolverhampton (pdf)
Kazi, M (2003) Realist evaluation for practice. The British Journal of Social Work, 33(6), pp.803-818 (paywalled or author copy)
Kazi, M (2000) Realist evaluation for evidence based practice. Journal of Social Work Research and Evaluation, 1(2), pp.101-109 (author copy)
Krohn, J (2015) Beyond outcome measures in child protection: using feedback to direct and evaluate social work practice. Practice: Social Work in Action, 27(2), pp.1-17 (paywalled)
Luckock, B et al. (2017) The Islington 'Doing what counts: measuring what matters': evaluation report. Department for Education (pdf)
Malin, N (2017) Developing an analytical framework for understanding the emergence of de-professionalisation in health, social care and education sectors. Social Work & Social Sciences Review, 19(1), pp.66–162 (open access)
Margolis, R et al. (2017) What are the best tools to measure outcomes for children and families? NSPCC (website)
McGuffin, Valerie (2002) Family support: measuring the benefits to families. Child Care in Practice, 8(4), pp.251-261 (paywalled)
Miller, E (2017) Review of evidence on personal outcomes relevant to the Carers (Scotland) Act 2016 (pdf)
Miller, E (2012) Measuring personal outcomes: challenges and strategies. Iriss Insight 12 (website and pdf)
Miller, E and Daly, E (2013) Understanding and measuring outcomes: the role of qualitative data. Iriss (pdf)
Moriarty, J and Manthorpe, J (2016) The effectiveness of social work with adults: a systematic scoping review. King's College London (pdf)
Morris, C et al. (2014) Informing the NHS Outcomes Framework: evaluating meaningful health outcomes for children with neurodisability using multiple methods including systematic review, qualitative research, Delphi survey and consensus meeting. Health Services and Delivery Research, 2(15) (pdf)
Mulroy, E and Lauber, H (2004) A user-friendly approach to program evaluation and effective community interventions for families at risk of homelessness. Social Work, 49(4), pp.573-586 (paywalled or author copy)
O'Doherty LJ et al. (2014) Selecting outcomes for intimate partner violence intervention trials: Overview and recommendations. Aggression and Violent Behavior, 19, pp.663–672 (paywalled or author copy)
Philip, G and O'Brien, M (2017) Are interventions supporting separated parents father inclusive? Insights and challenges from a review of programme implementation and impact. Child and Family Social Work, 22(2), pp.1114-1127 (paywalled or author manuscript)
Proctor, E (2017) The pursuit of quality for social work practice: three generations and counting. Journal of the Society for Social Work and Research, 8(3), pp.335–353 (open access)
Rhodes, H (2009) Knowing what you do works: measuring your own effectiveness with families, parents and children: a short guide. Family and Parenting Institute (FPI) (pdf)
Roberts, J et al. (2014) Measuring what matters: a guide for children's centres. UCL Institute of Health Equity (pdf)
Shaw, I and Lishman, J (1999) Evaluation and social work practice. London: SAGE (Available through NHS Libraries)
Sinha I et al. (2008) A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Medicine 5(4), e96 (open access)
Social Work Inspection Agency (2010) Improving social work in Scotland (pdf)
Spolander, G and Martin, L (2012) Successful project management in social work and social care: managing resources, assessing risks and measuring outcomes. London: Jessica Kingsley (Available from NHS Libraries)
Van Dop et al. (2016) Measuring empowerment: development and validation of the Service User Psychological Empowerment Scale. Journal of Social Service Research, 42(5), pp.651-664 (paywalled)
Wallace, E et al. (2011) Evaluation of Flying Start: findings from the baseline survey of families: mapping needs and measuring early influence among families with babies aged seven to 20 months: main report. Welsh Government Social Research Social Research (pdf)
Webb, S (2001) Some consideration on the validity of evidence-based practice in social work (paywalled or author copy)
Westmarland, N and Kelly, L (2013) Why extending measurements of 'success' in domestic violence perpetrator programmes matters for social work. British Journal of Social Work, 43(6), pp.1092-1110 (open access)
