Assessment and accountability were on the agenda big time for the Kentucky Legislature’s June 6, 2023 meeting of the Interim Joint Committee on Education, and the slide in Figure 1 showed up in a briefing from the Kentucky Department of Education (KDE) regarding what could be gleaned from state test results over the past seven years.

Figure 1

Impacts from COVID were obvious in the lack of scores for both 2020 and 2021. State tests weren’t even given in 2020 due to the pandemic-induced shutdown of classrooms, and lingering impacts led to only limited participation in the 2021 launch of the new Kentucky Summative Assessment (KSA) program.

It wasn’t until 2022 that things were back to near normal, except now the KSA tests are being used instead of the former the Kentucky Performance Rating for Educational Progress (KPREP) tests.

Overall, KDE’s presenter admitted that the impacts of COVID and switching from the KPREP to the KSA impairs the ability to compare trends across the past few years. But maybe we can find out more than the legislators heard on the sixth.

While Kentucky’s state tests have changed, the National Assessment of Educational Progress (NAEP) assessments for Grade 4 and Grade 8 in both math and reading were consistently being given against a stable set of testing standards since the early days of KPREP. Let’s see what we can glean from putting up the KPREP/KSA and NAEP results side-by-side.

 Figure 2

Figure 2 matches up elementary school level results from the KPREP from 2015 to 2019 and from the 2022 KSA to results from the NAEP Grade 4 Reading Assessment over the same time frame (Note: I pulled the 2015 KPREP scores from the 2015 Accountability_Achievement_Level spreadsheet from the KDE’s web site and all the NAEP results from the NAEP Data Explorer. Other state test results are from the briefing slide in Figure 1).

Some observations:

KPREP reading proficiency rates for elementary schools (the blue bars) stayed pretty much constant at around 55% throughout the period from 2015 to 2019.

In notable contrast, Kentucky’s NAEP Grade 4 Reading proficiency rate (orange bars) was steadily declining from 2015 to 2019. Note this decline was a pre-COVID trend.

In consequence, between 2015 and 2019 the difference in reported proficiency rates on KPREP versus NAEP increased from 14 to 20 points (gray bars). There was a decline, but KPREP wasn’t showing it.

Then, along came COVID.

By 2022, the new KSA was telling us that Kentucky’s elementary school level reading proficiency rate was lower than the last reported KPREP score by a full 10 percentage points. This brought the difference in the KSA figure and the NAEP figure down again to 14 points, the same variation seen between NAEP and the KPREP in 2015.

However, it is notable that the NAEP Grade 4 Reading proficiency rate dropped only 4 percentage points from 2019 to 2022 while simplistic examination of the Kentucky state test results would indicate more than twice that decline occurred. So, the real decline in reading performance might only be about half of what the different state tests indicate. In other words, cautions about comparing KPREP to KSA, at least for elementary school level reading, are probably valid.

But what will happen going forward. Will the KSA start to diverge from the NAEP results again as happened with the KPREP (and with other previous Kentucky tests such as CATS and KIRIS) – who knows?

Now examine the elementary math situation in Figure 3

Figure 3

During the KPREP period shown (2015 to 2019), both KPREP and NAEP showed essentially flat scores and the difference in reported proficiency rates was also consistent at nine percentage points.

Flash forward to 2022, and both the KSA and NAEP show declines in performance.

In addition, the KSA proficiency rate is only 5 points higher than the NAEP rate.

So, was there a decline in elementary school math in the COVID period? Most certainly.

Will the KSA elementary math results continue to track the NAEP closely? We’ll have to wait to see.

 Figure 4

Figure 4 shows the middle school reading situation.

First, note that during the latter part of the KPREP era, the Kentucky’s state testing scores were growing while the NAEP was in consistent decline. That raises concerns about what the KPREP was telling us. When one test shows improvement while the other shows decline, real issues are present.

Now, look at the 2022 KSA versus NAEP situation. There has been a big drop in the Kentucky test score results, bringing them closer to NAEP. But, NAEP only dropped by 4 percentage points. So, while there was certainly a real performance drop, there might not be as much need for panic as some school folks (always looking for more money) would like us to believe based on Kentucky state test results. Again, we need more time to know for sure.

Finally, let’s see the middle school math picture, shown in Figure 5.

Figure 5

NAEP was pretty much flat from 2015 to 2019 while KPREP was slightly more up and down. Overall, KPREP consistently showed much higher proficiency rates than NAEP through this whole period.

In 2022, KSA reported a notably lower proficiency rate for Kentucky middle school math than KPREP had reported, but NAEP fell off a fair amount in 2022, too. Thus, the KSA to NAEP proficiency difference isn’t much changed from the earlier KPREP to NAEP situation.

Summing up

It looks like NAEP shows more of a COVID-related decline in proficiency rates for math (down 7 points from 2019 to 2022 in Grade 4 Math and 8 points in Grade 8 Math) than it does for reading (down 4 points in both school levels). Since NAEP has been given in a consistent manner throughout the time considered by this blog, that is probably a fairly decent picture of what happened due to COVID. It’s a definite problem, but not a huge problem.

In comparison, attempting to use Kentucky’s changing test systems to gauge anything about the size of COVID impacts is problematic, at best. The tests clearly are not being scored to the same standards, and the variations of both Kentucky tests from NAEP are notable, as well.

It also remains to be seen if the KSA will inflate over time compared to the NAEP. We will have to stand by on that question, for now.