How is Kentucky’s education system really performing? — The Bluegrass Institute for Public Policy Solutions

Kentuckians hear it all the time. The state supposedly has made dramatic improvement on things like “National Tests” since KERA began. For example, the Prichard Committee proclaims that Kentucky ranks “8th in fourth-grade reading,” which is actually where the state ranks if you only look at overall 2015 scores for fourth grade reading from the National Assessment of Educational Progress (NAEP). A Prichard representative made similar claims on the February 5, 2018 Kentucky Tonight show.

But, is this an accurate picture? As the late Paul Harvey used to put it, there is a “Rest of the Story” here, and the rest of Kentucky’s education performance picture is important.

Want to see “Page 2” in this story? Just click the “Read more” link.

The reality about Kentucky’s educational performance cannot be told by only looking at overall “All Student” scores from the NAEP. There are several reasons for this, but the most important one is that the student demographics for Kentucky are now very different from the rest of the country. These dramatic differences give Kentucky a huge, but undeserved, advantage whenever overall average scores on NAEP get compared between the states.

Figure 1 shows the racial makeup of NAEP Grade 8 test takers in 2015 testing. Note the percentage of white students in Kentucky’s public schools – 82 percent – is absolutely huge, way above the national public school white percentage of 51 percent and astonishingly higher than California’s 25 percent figure.

Similar statistics are found in the fourth grade, too.

Figure 1

Comparing Kentucky's 2015 Grade 8 NAEP Racial Makeup to National Public and California Makeups, Percentage of Students in Each Racial Category

Because whites dramatically outscore all the other races except for the small numbers of Asians in some states (but not all states), simply having more whites is a huge advantage if only overall scores are considered.

So, when various folks talk about “stuff” like Kentucky ranking eighth on national testing but don’t tell you this is a comparison of overall scores only, those folks are actually providing you with a very apples to oranges comparison of white student scores in Kentucky to scores for lower-scoring racial groups in other states. That is highly misleading and inappropriate, which is why NAEP’s own report cards have carried cautions about doing simplistic rankings using only overall average scores (See Page 32 in the 2009 NAEP Science Report Card for a specific Kentucky example).It turns out that the picture can change dramatically when the comparison student groups are more closely aligned. Since Kentucky still has a huge proportion of white students, we’ll see in a moment how things change when you only consider white student scores on the NAEP.NAEP report cards also point out something else regarding the simplistic rankings games. The NAEP is a sampled test. For example, while fourth grade class sizes in Kentucky run somewhere around 49,000 students, the entire NAEP Kentucky sample was only around 3,000 for the 2015 NAEP Grade 4 Math Assessment (See Page 1 in this report). The actual number NAEP tests can be considerably lower due to higher than average exclusion of some students due to severe learning disabilities or being newly arrived in the US English language learners.

Due to sampling, ALL NAEP scores have sampling errors. These errors often make what appears to be “wins” into nothing more than “ties” for different states’ scores. Those who simplistically claim Kentucky ranks eight for reading are not allowing for NAEP’s sampling errors.

So, let’s do some NAEP state comparisons in a more statistically valid manner, allowing for the sampling errors as well as the demographic issues.

Let’s start with the fourth-grade reading situation. Figure 2 shows you how Kentucky’s white students match up to the rest of the states plus the District of Columbia schools for this 2015 NAEP assessment.

Figure 2

Note that the notion of ranking 8th falls apart if we look at the white student results for Kentucky. The facts are that nine states, shown in dark blue, performed statistically significantly higher than Kentucky and only 10 states scored statistically significantly lower (using NAEP’s standard of a 95 percent level of confidence). Kentucky tied with 32 states after the sampling errors are considered.

So, at least to a 95 percent level of confidence, Kentucky’s real rank lies somewhere between 10th place and 40th place out of the 50 states and DC schools. That’s all that the NAEP can accurately tell us. Anyone who tries to say we rank high against that reality is clearly saying more than the NAEP can really prove.

Figure 3 gives you the real NAEP picture for white student scores in fourth grade NAEP math in 2015.

Figure 3

There is no above average claim supported here. In fact, Kentucky’s real position lies somewhere between 26th place and 48th place. We can only claim with a relatively high level of confidence that Kentucky’s white fourth grade students outscored white counterparts in three other states. That is all the NAEP can show us.

Figure 4 provides a picture of Kentucky’s Grade 8 NAEP reading performance for white students in 2015.

Figure 4

Note in Figure 4 that the NAEP only allows us to claim with reasonable confidence that the state outscored four states. That is all we can really claim.

Figure 5 contains the worst story for Kentucky on the NAEP, which comes from Grade 8 math.

Figure 5

In this dramatic case, Kentucky can only claim its white eighth grade students outscored counterparts in just two other states. That is all.

However, even with NAEP sampling errors considered, 43 other states and the District of Columbia schools’ whites did statistically significantly outscore Kentucky’s white eighth grade students. Kentucky can rank no higher than 45th place and might rank as low as 49th place!

And that is what the NAEP can really tell us. Folks who try to claim more with the NAEP either just don’t understand this assessment and its limitations, or they are hoping you don’t know.

Technical Note: The maps were generated with the New NAEP Data Explorer.

Caution is advised when using this new tool as some software issues have been identified. For example, as of the morning of February 9, 2018, the Comparison feature in the statistical significance test section was not working properly.