This week Kentucky Auditor of Public Accounts Adam Edelen delivered an audit on the functioning of the Jefferson County Public School (JCPS) District. Plenty of problems are listed in this 301-page report, but the auditor also provides over 200 suggestions on how to improve. That could be helpful – IF – Jefferson County folks ever get past their initial, knee-jerk denials.WAVE-3 TV’s article, “JCPS fires back after audit claims wasted millions,” carried the first round of denials from a Louisville schools crowd that seems to place the status quo interests of highly paid school staffers ahead of those of Jefferson County Public Schools’ (JCPS) generally under-served students.

I dealt with some of the more general whining yesterday. Now, let’s look at a somewhat more technical denial, found at the end of the WAVE-3 post, where it says:

“JCPS said comparisons would have been difficult if Edelen did try to tie test scores to his audit. Three of the districts don't have a testing system that can be easily compared with JCPS. As for the other two, JCPS said their scores trail Charlotte and Austin is a mixed bag. JCPS is roughly equal in reading but behind that Texas city in math scores.”

Very simply, I don’t think that is accurate. Once you look at the test data involved, which comes from the National Assessment of Educational Progress, I think Jefferson County does not look too good on the academic issues, either.

If you have a taste for details, read on.

To begin, the test scores we are talking about come from the 2013 administration of the National Assessment of Educational Progress’ special Trial Urban District Assessment program. Besides Jefferson County, two other comparison districts in the audit, Austin, Texas and Charlotte, North Carolina, also took part in this test, which is often called the TUDA from its initials.

As you read above, even the Jefferson County schools folks admit that the more efficiently run Charlotte school system beat them hands down in math and reading in 2013 TUDA, but they say it is a “mixed bag” when we look at Austin.

Let’s look at the data.

Before we do that, however, I want to review a few important considerations whenever you want to compare test scores across different states/districts on NAEP.

• First, the NAEP is a sampled assessment, and all the scores have plus and minus statistical sampling errors. So, a “win” may only be a “tie” when scores are close.

• Second, the exclusion rates of students with learning disabilities and English language learners can vary considerably across different school systems on the NAEP. That can inflate scores, although the exact amount of inflation has never been determined.

• Third, thanks to achievement gaps and different student racial demographics across different school systems, looking only at overall “all student” NAEP scores can be deceiving. It is important to break the data out by race to insure one district with a lot more whites does not get an unfair advantage.

• Fourth, different poverty rates can also be a factor.

Now, with those points in mind, let’s look at the situation on the 2013 TUDA for Austin, Charlotte and Jefferson County.

This table shows the overall exclusion rates for students with learning disabilities and English language learners in the three school districts in the 2013 TUDA.

As you can see, Jefferson County across the board excluded a higher percentage of its special students than either other districts, and the difference was substantial in reading.

Thanks to excluding more of these students, Jefferson County’s TUDA math results are somewhat inflated compared to the other districts, but the reading results (remember, the WAVE-3 article says the deniers claim they tied Austin for reading) are possibly substantially higher thanks to exclusion-based inflation.

Sadly, though the problem was identified a decade and a half ago, there is no consensus at present on how to correct for exclusion rate differences. So, we’ll just leave this area with a comment that there is some inflation, more so in reading than in math, in Jefferson County’s TUDA scores due to exclusion rate differences.

Now, let’s talk about student racial demographics in these three districts. This next graph helps tell that tale.

Very simply, Jefferson County has a whole lot more white students, proportionately, than either Austin or Charlotte. That is important because whites outscore blacks and Hispanics by a wide margin on the NAEP. So, if a school system has a lot more whites, they get an unfair advantage in any comparison that only considers overall scores. To get a fairer idea of what is really going on, you have to break the data out by race, which I do in the next set of graphs for the two predominant racial groups in Louisville.

Here is what we find: In mathematics there is no contest. Jefferson County scores statistically significantly lower than the other two school systems for whites and blacks. The differences are all substantial. End of story.

In reading, Jefferson County is also outscored across the board for whites. There is no “mixed bag” here. In fact, the differences here are substantial as well as statistically significant.

For blacks in reading, the small differences in favor of Austin are not statistically significant, but don’t forget that the Jefferson County scores are inflated to an unknown degree by excessive, rather large exclusion rates. If exclusion rates had been equal, would Jefferson County’s score spread look worse? Probably.

To sum up, if we are talking about a white student, regardless of whether we are talking about math or reading, Jefferson County is left behind.

For black students, when we talk about math, Jefferson County is also left behind. In reading, it looks like blacks in Austin and Jefferson County might do about the same, but the full picture is confused by the excessive exclusion rates in Jefferson County.

One last point: poverty rates in the three cities based on eligibility for the federal school lunch program are pretty close. Lunch eligibility in Austin’s NAEP Grade 4 Reading sample was 62 percent and in Charlotte it was 57 percent. Jefferson County’s rate was 65 percent, very close to Austin’s. So, the good old poverty excuse doesn’t seem to work for Jefferson County, particularly in relation to Austin, either.

So, let’s hope that – this time – folks in Louisville will finally calm down and understand that they really do have some very significant problems in their schools. They owe the auditor some time to review those 200-plus recommendations and to make a good-faith effort to do all that they can to fix their problems.

Technical Notes:

All data come from the NAEP Data Explorer except for the exclusion rates, which come from the 2013 TUDA Report Card Summary Tables for Reading and Mathematics.