Credibility gap for Johns Hopkins graduation report deepens

Erroneous data in report misrepresents performance in several Kentucky school districts

I have already written here and here about credibility problems with the recently released “For All Kids, How Kentucky is Closing the High School Graduation Gap for Low-Income Students” from Johns Hopkins University and Civic Enterprises.

This week the report’s credibility slipped even more.

Page 16 of the report includes this slam on Kentucky’s Beechwood and Burgin independent school districts for their 2013-14 graduation statistics:

“The number of low-income students served by different districts is not correlated with their graduation rates for those students. For example, Beechwood Independent and Burgin Independent both have low-income populations under 30 percent, but both have low-income graduation rates of 75 percent, at the low end of the Kentucky graduation spectrum.”

As soon as I saw this, I wondered if the claim for the generally very high-performing Beechwood was correct. It didn’t take much searching in the “Data Sets” part of the Kentucky School Report Cards database to find out that Beechwood and Burgin actually posted superior graduation rates for their school lunch eligible students of 92.3 percent and 100 percent, respectively, in the 2013-14 school year (you have to dig all the way to Page 52 in the Hopkins report to learn that what they call “Low-Income” is actually based on students eligible for the federal free and reduced cost school lunch program).

Obviously, neither district’s true performance for low-income students was even close to the 75-percent graduation rate figure alleged by Johns Hopkins.

Amazed, I started to spot-check other data in the Johns Hopkins report appendix beginning on Page 47 where the full listing of supposed 2013-14 graduation rate data for all Kentucky school districts is found. It didn’t take long to discover that the Hazard Independent School District was listed in the Hopkins report for a 90 percent graduation rate for low-income students while the Kentucky Department of Education says the real figure is 100 percent.

With my attention really aroused, I took the opportunity to talk to one of the Kentucky Department of Education’s data experts this week. It turns out the department also spotted problems with the data in the Hopkins report (In fact, they said they knew I would be looking so they had better look, too!). By the time I talked to them, the department had already double-checked its 2013-14 submission of this information to the US Department of Education because the Hopkins report says their researchers got their data from a US Ed data file titled the “Provisional data file: SY2013-14 District Level Four-Year Regulatory Adjusted Cohort Graduation Rates (ACGR)” (Sorry, have not found this online).

The department’s expert was very firm in stating the data submitted to US Ed for 2013-14 was correct when it left Kentucky.

So, the Hopkins report now faces a disturbing credibility problem regarding the very basis of its report: the data it used. Some of the numbers in the Hopkins report’s appendix simply do not agree with the correct data from the Kentucky Department of Education. Even worse, the errors are not uniform across all districts, but the Kentucky Department of Education says there are other errors besides those I mention above.

Obviously if the Kentucky numbers are not accurate, how much confidence can we have in the numbers that Hopkins cites for graduations in other states?

And, as we earlier pointed out, even if the data were accurate, what evidence is available that shows the education level required to get a diploma in Kentucky is even remotely similar to what other states demand? Hopkins doesn’t even consider that critical and obvious question.

Here’s another thought: an alternate interpretation of the Hopkins data – even if the data were accurate – could be that Kentucky is just handing out a lot more hollow diplomas than other states award. That isn’t a mark of success – it’s just a major and misleading problem.

Going forward, at least a couple of things need to happen:

• Hopkins needs to apologize to Burgin and Beechwood and issue a public correction.

• Hopkins needs to find out where the data corruption occurred. Was the problem with US Ed or with Hopkins researchers? If with US Ed., a lot of people need to get concerned.

• The Kentucky Department of Education, which earlier posted a favorable news release about the Hopkins report, needs to correct the erroneous assertions about Beechwood, Burgin and other districts and consider more careful reviews of reports in the future before publishing glowing News Releases about them.

• Even if it corrects the erroneous data, Hopkins needs to either provide more research that shows comparison of graduation rates between states is a relatively valid indicator of actual education performance or retract this obviously troubled report.

By the way, I am not the only intelligent person who is troubled by the Hopkins report. Professor Gary Houchens from Western Kentucky University, who is a member of the Kentucky Board of Education and also is a member of the Bluegrass Institute’s Board of Scholars, has also spent time examining the paper. Professor Houchens has a number of well-stated, major concerns as well.(Corrected Link 30Aug16 at 8:57 pm)