On Tuesday, the dubiously reorganized Kentucky Board of Education got an update on recent assessment results. This included a very brief and incomplete discussion of the state’s 2019 performance on the National Assessment of Educational Progress (NAEP). It was an inadequate briefing, but the dubiously reorganized board members never questioned it. Let’s fix that.

First, our regular readers know that simply looking at overall average scores on the NAEP isn’t a good way to compare state to state results. One major reason why is that racial demographics vary widely across the states today, which makes simplistic comparisons only of overall average scores an exercise in apples to oranges.

Consider this:

In the 2019 NAEP Grade 4 Reading assessment, Kentucky’s public school enrollment was listed as 75% white. In notable contrast, the national public school enrollment is now only 46% white, and some states have much lower white proportions, including Texas at only 27%, New Mexico at only 23%, California at only 21% white and Hawaii at just 13% white. Overall, only seven states have white enrollment percentages higher than Kentucky’s.

What this means is that when only the “all students” scores are compared between the states, the analysis winds up comparing a lot of white students in Kentucky to a lot of children of color in other states. That just won’t work because large racial achievement gaps are found in virtually every state. Even though Kentucky’s whites score low, as we will shortly see, they still outscore virtually all the minority students, so comparing all student scores across the states loads the deck in Kentucky’s favor.

To do state to state comparisons somewhat more accurately, you really need to break out the results separately for each race. To see how that works out, just click the "Read more" link.

Since only whites have scores reported for all 50 states, looking at white scores provides a better analysis than just looking at overall scores. With that said, here are the slides about the NAEP that the Kentucky Department of Education included in its briefing on Tuesday (Slides 30 to 32) along with similar map presentations for the same grades and subjects for white students only, which I developed using the NAEP Data Explorer, which is also the source for the demographic percentages I mentioned earlier.

Grade 4 Reading

Let’s start with Grade 4 reading. Figure 1 is what the Kentucky Board of Education was shown on Tuesday.

Figure 1

A key here is to look at the number of “jurisdictions” that had lower scores than Kentucky. In this case the slide says 13 jurisdictions had lower scores (by a statistically significant amount). Supposedly, Kentucky ranks 23rd, but such simplistic rankings concern me because they ignore the statistical sampling errors in the NAEP. It’s more accurate to talk about the states that statistically significantly outscored, tied, or scored statistically significantly lower than Kentucky. But, everyone loves rankings….

Now, look at Figure 2, which looks at whites only. (Note I didn’t rank the Department of Defense Schools (DS) on this map, but they also scored higher than Kentucky, so pretend their dot is also in dark blue).

Figure 2

When we get away from comparing apples to oranges and only look at white scores, Kentucky only outscores just two states, West Virginia and Alaska, by a statistically significant amount. We also scored below the national public school average by a statistically significant amount.

Doing a little more work with the NAEP Data Explorer, using KDE-type rankings, which completely ignore those sampling errors, and including the DOD schools in the mix, Kentucky ranks 39th for white grade 4 reading. That’s a whole lot different from 23rd place!

Grade 4 Math

Figure 3 shows the department’s map for all student score results for NAEP Grade 4 math.

Figure 3

We don’t do nearly as well in math, but still supposedly outscore 9 other jurisdictions. How does that compare to the white student results? Check Figure 4.

Figure 4

Now, only whites in West Virginia score lower. Again, the DS schools also score statistically significantly above us and should share dark blue coloring.

Once more using the dubious ranking of scores without considering sampling errors, Kentucky would rank only 48th out of the 52 jurisdictions covered on the map. Again, that is A LOT lower than the already rather low 31st place the Kentucky Board of Education thinks we occupy.

Grade 8 Reading

Figure 5 shows what the state board was told about Kentucky’s Grade 8 NAEP reading ranking in 2019.

Figure 5

Again, looking at all student scores, which rank a lot of Kentucky whites against lots of kids of color elsewhere, we rank 26th. We supposedly outrank 16 other jurisdictions, too. But, Figure 6 gives us another, much more revealing picture.

Figure 6

The sobering story is that Kentucky’s whites only outscore whites for reading in just two other states. Again, you can color that DS (DOD Schools) circle in dark blue. Using the same, dubious, forget-the-sampling-error-ranking the department employs, Kentucky ranks 47th. That is WAY different from the 26th place ranking our new state board members heard about.

Grade 8 math

Finally, Figure 7 deals with Grade 8 NAEP math, as the Kentucky Department of Education reported it on Tuesday. Already, even using the all student results, Kentucky’s ranking for Grade 8 math doesn’t look that great. But, we supposedly outscored nine other jurisdictions in this 52-jurisdiction group and ranked 37th.

Figure 7

But, take a look at Figure 8!

Figure 8

OUCH! Kentucky only outscores West Virginia by a statistically significant amount (color the DS circle dark blue again). Worse, using KDE’s simplistic ranking approach, Kentucky places 50th out of the 52 jurisdictions with data reported.

Now, there is still more to this story. Back on December 4, 2019, at its last meeting the ousted Kentucky Board of Education got a far more expanded briefing on the NAEP results. This discussion started at about 2 Hours, 37 Minutes into the Video 1 webcast you can access below.

If you listen to this, you will quickly see that the members of the now ousted board ask lots of informed questions about the NAEP, and board member Gary Houchens even points out that some of the slides are misleading due to student demographics. Point is the old board understood a lot about the NAEP.

But, in the new presentation to the new board on February 4, 2020, nary a question was raised about the possibility that the very limited presentation’s slides might have issues. That presentation starts at 1Hour, 57 minutes into the webcast. As you listen to the board mostly just accept without insight what they are fed about the NAEP, it seems no one on the new board understands very much about the NAEP? Considering that the new board is composed mostly of people with education backgrounds, all I can say is, “Wow!”

Also note in the first video that the old board members and now fired education commissioner Lewis were all over the fact that Mississippi was making great progress in the NAEP and needed to be considered.

In contrast, in the video from the February 4, 2020 meeting, the only notable mention of Mississippi I caught was during a 3-minute public comment from member of the public Brigit Ramsey of the Prichard Committee, who did bring up that state and urged the board to pay attention (which the old board had already doing). This didn't come from the department or any current board member.

The new board members seemed to know nothing about Mississippi. The goings on in the Magnolia State were not mentioned during the new board’s brief, seriously incomplete – if not outright misleading – discussion about the NAEP slides.

That total loss of all corporate memory is what happens if you throw an entire board out on the street at the same time. This isn’t what KERA required for change-outs of the state board, and we may be reaping some serious consequences – at the very least more delay – in the board getting up to speed on things that could really help Kentucky education.