Beware rankings of state education systems

A half-year-old news article from the Courier-Journal about a study that ranks Kentucky’s education system as the 20th best in the nation has somehow resurfaced in the Twittersphere, and several educators are gushing over it. That makes it worthwhile to point to just a few reasons why the news article, the report and those who cite both really don’t have a good handle on education statistics. We want our readers to be sharper than that.

Curiously, the news article wasn’t written by either of the Courier’s lead education reporters. That is probably part of the explanation for why this ever surfaced in the first place. Education is a complex news beat and there are lots of ways to trip up.

The report mentioned by the Courier comes from a crowd called WalletHub. WalletHub’s main emphasis seems to be banking-related stuff like credit reporting and monitoring. How that relates to complex issues surrounding education data eludes me. In fact, there is good evidence of the lack of understanding of such education data once you dig into WalletHub’s education ranking report, which is titled “States with the Best & Worst School Systems.

”WalletHub’s report has some serious short-comings. You already have to know a fair amount about education data and still have to dig around quite a bit to figure out what WalletHub is ranking because the sourcing and referencing is abysmal. In the days of relatively easy hyperlinking, that’s really hard to understand, but there it is. Click the "Read more" link to see some comments about data I think WalletHub is ranking.

Ranking really dubious dropout rates

WalletHub’s specific rankings incredibly include extremely dubious high school dropout rate statistics from each state. These education data fell into disrepute years ago, which is why federal legislation since No Child Left Behind has consistently required states to use high school graduation rates for school ranking programs. But, that isn’t all WalletHub gets wrong.

Ranking dubious high school graduation rates

In an action that seems a bit like double-counting, WalletHub also ranks high school graduation rates. There is no indication of where those data came from, however.

In any event, there are NO consistent standards across the states for what is required to earn a high school diploma. So, expecting high school graduation rates will somehow indicate stronger or weaker state education systems is nonsense.

By the way, WalletHub probably ranks graduation rate numbers currently reported by each state to the federal government. These use the 4-Year Adjusted Cohort Graduation Rate formula, but there are no references to tell for sure. Regardless, the lack of consistent graduation standards makes comparisons worthless.

One more point about high school graduation rates: we do know that while Kentucky has a relatively large high school graduation rate, the numbers include a lot of hollow diplomas for students who can’t pass muster under any of a number of ways to be considered transition ready for either college or the workforce.

For sure, comparing Kentucky’s graduation rates to those elsewhere is a complete exercise in irrelevance.

Ranking “all student” scores from NAEP

Another of WalletHub’s major ranking gaffes almost undoubtedly is based on the National Assessment of Educational Progress (NAEP) although again the report doesn’t identify the math and reading tests that are actually being considered. The report just says it ranks reading and math scores for all the states for the fourth and eighth grade (which happen to be the same grades the NAEP assesses).We have written a ton about what needs to be done to intelligently compare testing results across states. A major problem is that student demographics now vary dramatically by state; thus, if you only look at overall average scores as WalletHub apparently does, you wind up comparing scores for a lot of white kids in Kentucky to scores for a lot of minority kids in other states. Thanks to the major achievement gaps in our schools, that winds up giving Kentucky a large and unearned advantage.

All NAEP Report Cards from 2005 on talk about demographic variations and other things that should be considered before you try to rank states on the NAEP. Apparently, WalletHub doesn’t pay attention to this guidance from the people who design and administer the NAEP.

How much difference do student demographics make? I’ll just cite one example from the latest, 2017 NAEP testing for Grade 8 Math. Look at how Kentucky ranks when we examine apples to apples scores for only white students.

Grade 8 Math Whites Map for All States

Recall again that WalletHub says Kentucky ranks 20th for education. But, when you look at how the state’s white students stack up to whites in other states for Grade 8 Math, the Bluegrass State got outscored by a statistically significant amount by 41 states plus the District of Columbia schools. Only 2 states scored statistically significantly worse.

Does that look like 20th place to you?

As you consider this, keep in mind that about 80 percent of Kentucky’s students are white.

Ranking states for ACT and SAT

This is another incredibly poorly informed ranking from WalletHub. In most states taking both of these tests is totally voluntary; only students who want to go to college participate. Furthermore, there is a tendency to favor one of these tests over the other in each state, further distancing these tests from any sort of a valid random sample situation.

In Kentucky, for example, taking the ACT is mandatory for all public school 11th grade students, so 100% of Kentucky’s graduates take this test each year. But, ACT reports only 7 percent of the 2018 high school graduates in Maine took the ACT. There is NO intelligent way to compare Kentucky to Maine on this statistic.

When it comes to the SAT, very few Kentucky students take this assessment, reversing the situation for Maine and Kentucky because many students take the SAT there (In fact, through at least 2017 the state tested all of its students with the SAT).

The bottom line

The data on education is often complex to understand and it can take years to learn most of the ins and outs of the available information. Ranking state education systems isn’t for the young, or apparently for bankers and some reporters, too. Too often, even those in education don’t seem to understand issues with the data they generate. But, when Kentucky’s white students only outscore their counterparts in just two other states for NAEP Grade 8 Math, someone trying to claim the state ranks 20th for education is going to face a very hard sell.