Every year around the end of the summer term, I evaluate the impact of our work in the schools and Trusts we work with. It’s a great time to reflect on what is working well and, of course, on how to make things better.
For us, there is always a fine balance between taking the time to measure what we’re doing versus using that time to do more doing! Although we’d all probably rather do the latter, measuring impact is important to make sure what we’re doing is working, to help schools show their return on investment and to help showcase the importance of intervention.
With that in mind, I’ll be sharing a snapshot of our approach to measuring impact, looking first at the numbers, with another blog to follow on more qualitative measurements.
Setting Objectives
Our focus on impact begins at the start of our relationship with schools, when we discuss the outcomes and impact they are seeking. Obviously, this can vary enormously. Every school is different. So, by being on the same page from the beginning, we are measuring the difference we all want to see.
Activity and Outcomes
Often at the highest levels of commissioning and leadership there’s a keen interest in our activity and outcomes, for example:
How many visits has the speech and language therapist made?
How many children have been referred and how many seen?
How many EHCP reports have been written by the speech and language therapist?
How many children have made progress?
However, schools are also interested in the quality of our service and the impact we are looking to achieve. The impact measures are naturally broader, longer-term, and often more difficult to measure. Though we all understand that to achieve lasting positive outcomes for children’s SLCN, we need to step back and broaden our focus.
Ways of working
Following SEND guidelines and working to an evidence-based practice model, we work with schools at three main levels. We support identification and intervention at each level, all through discussion and collaboration with school staff and leaders[1]:
Universal: supporting best practice in language for learning for all children, building awareness and identifying children who may be struggling.
Targeted: supporting identification and support of children who are struggling with the development of speech, language and communication skills.
Specialist: working directly and indirectly with children who have speech, language and communication needs.
Alongside these levels of practice:
We work to build professional learning in school staff to increase capacity and expertise.
We work with parents wherever possible to support home-school links and build parental understanding of SLCN.
We build student voice principles into our practice, taking account of student views on intervention, support, and what works best for them.
Ideally, we work with leaders on a strategic level to build implementation strategies[2] and systems that stand the test of time.
What does this mean for our approach to measuring outcomes and impact?
With any evaluation or assessment protocols, we must balance the need to capture change with the need to ensure change is happening. We try to avoid too much weighing and measuring with a focus on working to support the children and staff in school.
However, outcome measures that provide strong quantitative evidence of impact can be relatively easy to capture. For example, this year we are able to report:
Visit allocation fulfilled.
100% of referrals received have been processed.
95% of assessments completed, with 5% carried over until September.
Management plans for each child were completed where relevant.
Over 90% of pupils receiving support are achieving their short and medium-term targets.
Individual pupils
Our schools have access to a summary of their caseload, where we track progress through the academic year. Case studies are also a great way to exemplify pupil progress, e.g.:
This pupil with DLD has been receiving direct speech and language therapy, including
Collaborative work with his support assistant and class teacher.
A focus on building understanding of language.
Initial work on word learning.
Opportunities for exploratory talk, understanding, and inferencing in small group work.
We have captured positive impact on standardised assessments, with him moving from most scores representing significant difficulties, to most scores within the typical range as shown in the table, plus:
Positive feedback from his teacher regarding engagement in the classroom.
The pupil talked about his increased confidence in talking in the classroom and about the additional time the teacher gives him to answer.
Changes in his language were captured through observation in the classroom and intervention.
Targeted interventions
With our targeted intervention, we carry out baseline assessments and retest against the baseline to show outcomes. It’s a simple way to capture change with some inherent challenges, but gives useful information. For a more qualitative picture, we also gather feedback from staff and children, as well as observations of changes we see in the children’s language and interactions in the setting. Below are three examples of targeted interventions where we have captured outcomes via measuring against a baseline and retesting.
Early Years
In one academy trust, the schools screen all their children using the Wellcomm assessment. We ran our Chatterboxes language intervention in 8 schools with small groups of 5 children. The intervention ran three times weekly for around 15 minutes over a 4-month period. We then calculated averages of pupil data for each school.
Over the 4-month intervention period, the lowest level of progress was 9 months, and the highest was 21 months. Differences were often due to levels of fidelity. Schools that implemented as above were the ones where children tended to make the most progress. These schools also reported seeing higher levels of GLD at the end of the academic year than in previous years.
In primary KS2
We had a number of schools that referred children in KS2 (Year 5) with a combination of difficulties with spoken language and text comprehension. Our Language Legends programme is based on research that demonstrated an impact on text comprehension via a focus on spoken language, so this intervention was used.
The children had all been referred for therapy and were tested using the ACE 6-11 standardised assessment tool. This tests various aspects of language, including understanding of language, inference, and word meanings.
Before the intervention, all children were scoring below the typical range in the three subtests. Their understanding of word meanings was particularly low. After the intervention, scores showed all children were within the typical language range. Additional qualitative data from class teachers and students reinforced the positive impact in relation to confidence, engagement in class discussions, and improvements in text comprehension.
In secondary KS3
In one of our secondary schools, there were high levels of need in relation to language. As SLCN is difficult to identify, we use school data to identify ‘at-risk’ pupils, then screen those pupils using the Communication Trust progression tools. This helps identify need and allows us to check pupils’ language across four categories:
Understanding and verbal reasoning
Vocabulary
Sentence structure and narrative
Social communication
We implemented our News Hackers programme with 20 pupils in this school to support language. We then retested using the tool and gathered feedback from students and teachers. The majority of pupils made good progress.
Individual pupil data and student feedback were included in our impact report. As students did this work as part of their studies, they did not need to be withdrawn from their usual classes to complete the work. This was seen as a positive for many pupils.
In another of our secondary schools, the year 7 cohort was identified as having high need during the Y6/7 transition process. Due to high need across the cohort, a communication curriculum was put in place for these pupils, with a focus on a combination of Oracy exploratory talk, dialogic teaching, and targeted speech and language work.
In this school, pupils were tracked using school based Rising Stars data, which we used in combination with other measures to track progress.
We saw great progress for all the classes taking part in the communication lessons, with the majority making expected progress or better by the summer term.
Part 2 coming next week
As with many areas of life, the numbers are just a part of the story. They focus mainly on the measurable short-term outcomes. As important as this is, we want to capture the bigger picture, so as well as collating the numbers, we gather information on the wider impact of our work.
In our next blog, we will share how we capture the wider impact of our work, alongside these outcome measures. We’d love to hear from you – what is your vision of positive impact on children’s speech, language and communication? What has worked well for you in capturing outcomes and the wider impact of speech and language therapy support?
[1] International Journal of Language & Communication Disorders: Vol 54, No 1 (wiley.com)