Johns Hopkins gets honest about altering federal data – sort of

It took a lot of questioning and independent research, but the authors of a recent report from Johns Hopkins University called “For All Kids, How Kentucky is Closing the High School Graduation Gap for Low-Income Students” finally admitted – only after we collected some rather compelling evidence – that they altered high school graduation rate data in a federal report. The Hopkins researchers did that without initially being up front about what they did.

The impact of this tardy admission coupled with a number of other problems – such as the continuing flaw in the report’s fundamental assumption that high school diploma awards in different states indicate comparable academic achievement – significantly undermine the value of the report.

The graduation rate data in question are federally reported high school graduation rates for Kentucky’s public school districts for the 2013-14 school year. These figures are officially called “Four-Year Adjusted Cohort High School Graduation Rates.” While all states are supposed to follow the same reporting guidelines for these graduation rates, cautions we found on Page 9 in a federal document that accompanies the official graduation data in the EdFacts web site makes it clear that – even on a technical basis – the federally reported graduation rates might not be fairly comparable from state to state. The document says,

“Although the regulatory adjusted cohort rates are more comparable across states than were rates submitted in previous years under the Elementary and Secondary Education Act of 1965 (ESEA) as amended, there are still some differences in how states have calculated their rates. These differences include: how students are identified for inclusion in certain subgroups, how the beginning of the cohort is defined, whether summer school students are included, and which diplomas count as a regular high school diploma.”

A lack of comparability of data from state to state undermines the fundamental assumption in the Hopkins report and all the conclusions derived from that assumption. As a note, this federal caveat about their data is not repeated in the Hopkins report.In any event, Hopkins manipulated the actual graduation rate data the feds published in ways that introduce some serious errors. Hopkins’ inappropriate presentation also misleads the reader to believe the federally reported data is a lot more precise than it really is.As we were researching the report, Hopkins issued at least two report changes. We located the first revision on August 30, 2016, right after a webinar about the report ended. This update deleted some seriously erroneous comments found on Page 16 in the initial version of the report, which came out on July 26, 2016. Those erroneous comments concern supposedly low performance in the Burgin and Beechwood Independent School Districts. The obvious error about Beechwood, which is a very high performing school district, was our first indication that there was a problem with the Hopkins data.

It was a matter of concern that the webinar totally ignored the Beechwood and Burgin errors although the lead author of the report and the moderator of the webinar had both been informed in advance that this issue existed and needed to be addressed.

Subsequent to the release of the revision we found on August 30, 2016, yet another version was released. We located that second version on September 3, 2016.We have been unable to find any indication in either of those revised reports that they are updated/altered/corrected/revised from earlier versions – chose your own term. However, we found changes in each update that were not present in the earlier versions.

Furthermore, only the most recent, September 3 version finally adds a statement at the beginning of the Appendices section on Page 47 that describes how the Hopkins team manipulated the real federal data.

After explaining that the federal data is purposely blurred by reporting the graduation rates in ranges only for smaller student groups (e.g. 80 to 84 percent) to comply with student data privacy provisions in the Family Educational Rights and Privacy Act (FERPA), this Hopkins statement says:

“In those cases where rates have been reported as ranges we have taken the midpoint as the method of estimating the true graduation rates for those groups/subgroups while maintaining and protecting the right to privacy of individual students.”

You won’t find this statement on Page 47 in the first version of the report, which is still online as of our posting of this blog (Look quickly. This may drop off the web at any moment).

What the federal government really reports

One place the real federal data can be found is in a comma delimited csv file from the US Department of Education’s EdFacts web site. I downloaded that file and followed instructions in the EdFacts web site to convert it to a more usable Excel file, which you can see here Provisional Data File - SY 2013-14 Four Year Regulatory Adjusted Cohort Graduation Rate, by State 10192015-graduation-rate-1. This EdFacts file is not the exact file Hopkins cites (that file apparently isn’t on line and so far no-one we have contacted has been able to locate it). However, the EdFacts file is apparently consistent with whatever federal data Hopkins worked from, just as it should be if both files really come from the US Department of Education.

The EdFacts graduation rate file for school year 2013-14 contains a huge number of “blurred” entries for Kentucky’s school districts. Not only are virtually all the data for what EdFacts calls “economically disadvantaged students” (called “low-income students” in the Hopkins report) only presented as a range of possibilities, but even most of the “all cohort” (“all student”) numbers are blurred for most Kentucky districts, too.

In many cases, Kentucky data available in the EdFacts web site for the 2013-14 school year shows the graduation rate blurring range runs about five units wide. As explained in an EdFacts document, one of the standard reporting ranges runs from 80 to 84 percent and the next runs from 85 to 89 percent. That purposely creates uncertainty to protect student privacy. However, it also makes it difficult to compare results from different districts or for different student subgroups with any sort of precision.

For example, a district with a real graduation rate of 80 percent would be reported with a band of 80 to 84 percent. Meanwhile, a district with an 89 percent true rate would be listed in the 85 to 89 percent range.

So, the real gap in performance for these school districts would be a rather large nine points. However, using the inappropriate Hopkins estimations, the first district would be shown with an 82 percent graduation rate and the second with an 87 percent rate. The true 9-point gap would be misleadingly cut nearly in half to only a 5-point spread.

It gets worse

Some small Kentucky districts have even wider score ranges for their all cohort graduation rates in the federal data. Jackson Independent, for example, has an all cohort graduation rate listed in the federal report as greater than or equal to 80 percent. That is a pretty big range of uncertainty.

In the case of Jackson, the September 3 version of the Hopkins report says the district had an “all students” graduation rate of precisely 90 percent. This is right in the middle of the possible range actually reported by the federal government, of course, but it is highly inaccurate.

The real 2013-14 high school graduation rate for all Jackson Independent students reported in the Excel “Data Sets” area of the Kentucky Department of Education’s Kentucky School Report Cards Database is 100 percent.

The Hopkins “estimate” is off by 10 percentage points, ranking Jackson’s all student graduation rate considerably lower than it should be. Of course, questions still remain about the education value behind a diploma from Jackson as compared diplomas from other Kentucky school districts (See references at the end of this blog for more on the serious quality control problem with Kentucky’s high school diplomas), but if we are just focused on graduation rate reporting, the Hopkins report provides very misleading data for the school district.

When we look at the focus group of interest in the Hopkins report, the “Low-Income Students” – which the EdFacts file actually labels “Economically Disadvantaged Students” – the situation deteriorates even more. As a note, these are apparently listed as “Free/Reduced-Price Meals” eligible students in the Kentucky Department of Education’s reports.

The federal EdFacts file for 2013-14 lists Kentucky’s Burgin Independent School District with a graduation rate for the economically disadvantaged students of “GE50.” This means the real graduation rate for these students in Burgin is greater than or equal to 50 percent. That creates a huge band of uncertainty. The actual rate could be anywhere between 50 and 100 percent so far as you can tell from the official federal data.

Hopkins, on its own, changed this. In the original version of the Hopkins report, the Burgin “low-income” student rate was shown as a hard 75 percent number. After we pointed out to Hopkins researchers that this rate was grossly in error, the Burgin rate depiction in the Hopkins report was changed to “>75.” This means the rate was greater than 75 percent, according to Hopkins. Here is a math fact for you, saying something is greater than or equal to 50 percent is not the same thing as saying something is greater than or equal to 75 percent. Hopkins had no right to change that federal figure. What makes this much worse is the true Burgin rate for lunch eligible students is easily found in the Kentucky School Report Cards Data Sets for 2013-14 as 100 percent. Even examining the Hopkins data is just a waste of time.

By the way, you can read more about the federal blurring policies for 2013-14 here.

Here is where we stand

Hopkins was not forthcoming in its initial report about where it really got its numbers. Significantly manipulating real federal data without admitting it – sometimes generating very large errors in the process – is a big problem.

Even now that Hopkins has admitted how it altered the true federal data, it is clear that as of the September 3, 2016 version of the Hopkins report, the Hopkins’ Kentucky school district graduation rate statistics are not accurate. Furthermore, the amount of uncertainty in the numbers varies widely depending upon district enrollment counts.

Kentuckians need to know that far more accurate graduation rate information is readily available from the Kentucky Department of Education. It is a mystery why the Hopkins report completely ignored this far more precise and accurate data source for Kentucky’s school district graduation rates.

Obviously, if we want to draw good, data-based conclusions about Kentucky’s high school graduation situation, the Hopkins report is not the place to go.

Other references

We’ve written a lot about Kentucky’s major problem with high school diploma quality control. This quality control problem obviously makes the Hopkins’ reports sole focus on diploma awards as some sort of metric for real educational performance very dubious, at best.

Here are some blogs and articles of interest regarding this important topic:

Kentucky’s Effective High School Graduation Rate

Bluegrass Beacon: Social promotion enabled by watered-down diplomas

News release: The uneven quality of Kentucky’s high school diplomas

More on the quality control problems with Kentucky’s high school diplomas – Part 1

More on the quality control problems with Kentucky’s high school diplomas – Part 2

More on the quality control problems with Kentucky’s high school diplomas – Part 3

More on the quality control problems with Kentucky’s high school diplomas – Part 4

Is Kentucky handing out ‘hollow’ diplomas?

(Update September 27, 2016)I talked today to Dr. Mike Stacy, superintendent at the Beechwood Independent School District. So far, he has received no apology from Johns Hopkins for the incorrect charges made in the initial version of the report that Beechwood had an abnormally low, 75 percent graduation rate for its low-income students. He is very unhappy about Hopkins' activities with their report, and I agree with him.

Also, the Kentucky Department of Education (KDE) has not issued any update/correction to its July 27, 2016 News Release No. 16-069 that unquestioningly presented the Hopkins report as valid and reliable information. At the very least, the department owes it to Beechwood and Burgin Independent to set the record straight about their real graduation rate performance. Common, KDE, you have a responsibility here to protect your school districts from inaccurate characterizations.

Others who jumped on this report when it first came out also need to provide their followers with the rest of the story. That is how real research is done.