New Johns Hopkins University report builds on shaky foundation

A new report from Johns Hopkins University (JHU) and Civic Enterprises is generating considerable media attention due in no small measure to a favorable Op-Ed from former Kentucky First Lady Jane Beshear and a press release from the Kentucky Department of Education.

Unfortunately, the attention might be rather misdirected due to a number of problems with the “For All Kids, How Kentucky is Closing the High School Graduation Gap for Low-Income Students” report. I think the short-comings are important enough that Kentuckians need to hear “The rest of the story,” as the late Paul Harvey always so nicely put it.

Based on some interesting evidence I have assembled, the basic metrics of performance the JHU report is built around – high school graduation rates in Kentucky – are not reliable indicators of education performance and should not be compared to graduation data for other states.

Unfortunately, due to pressures created by increasing accountability for graduation rates in both federal and state policies, there is reason for considerable concern that a lot of students are being socially promoted all the way to a high school diploma in Kentucky’s schools.

Worse, because there are absolutely no common quality standards between the states for what it really takes to earn a diploma, the true comparability of reported high school graduation data from state to state is completely unknown.

To drill down a bit deeper, there are ways to examine the relative quality of diplomas awarded across the Bluegrass State’s school districts. I have been working with the related data for a number of months now. I’ve already written about some of this research.

For example, in the Bluegrass Institute’s “Blacks Continue Falling Through Gaps in Louisville’s Schools, The 2016 Update” report that released earlier this year, I examine the credibility of diploma awards as a real indication of educational performance for the high schools in the Jefferson County Public School System.

In doing that Jefferson County analysis, I discuss one new way to examine the real validity of diploma awards using a metric I developed called the “Effective High School Graduation Rate.” The Effective High School Graduation Rate shows the percentage of entering ninth grade students that graduate on time four years later with sufficient education to enable them to qualify as college and/or career ready using the Kentucky Board of Education’s own, formally-adopted methods for determining readiness.

You can read about the current, very unsatisfactory graduation rate situation in Jefferson County in the Blacks Falling Through Gaps report starting on Page 11. Of particular note for the new JHU paper is evidence in the Blacks Falling Through Gaps report that shows far more blacks than whites are being socially promoted to hollow diplomas in the school district. The evidence shows that even within this single Kentucky school system, quality control over diploma awards is sorely lacking. The absence of quality control makes Kentucky’s high school graduation data unreliable for conducting comparisons of educational performance between school systems.

Let’s take an even deeper look at what Kentucky’s data can show us about the unreliability of the state’s high school graduation rate data as a metric for education system performance. For this examination we look at the statewide data for Kentucky and for five school districts that are highlighted in the JHU report. The table below shows the information (click to enlarge, if necessary).

Summary of Effective Grad Rate and Algebra II Analysis for Study Districts and State

Here is how to read this table. Concentrate first on the bottom line, which summarizes statewide, “Kentucky Average” statistics. The first column shows the “Reported 4-Year Adjusted Cohort High School Grad Rate 2015.” These official graduation rate numbers are reported in the Kentucky Department of Education’s School Report Cards database. Note that the department reports the statewide average graduation rate in 2015 using the federally required 4-year cohort calculation is 88.0 percent. This tells us that out of every 100 ninth graders who started high school in the 2011-12 school year, 88 graduated on time in 2015.

Now, according to Kentucky regulations, demonstrated competency in Algebra II – or at least the content covered by Algebra II – is a high school graduation requirement. Furthermore, the department reports the proficiency rate on its Kentucky Performance Rating for Educational Progress (KPREP) Algebra II End-of-Course (EOC) Exams each year. After checking with the department, it appears the majority of Kentucky’s high school students are taking this course during their junior year in high school.

So, the next column in the table shows the officially reported percentage of students that was proficient or more on the KPREP Algebra II EOC in the 2013-14 school year, the year the Class of 2015 was in the 11th grade. That Algebra II EOC Exam proficiency rate was a disturbingly low figure of only 37.9 percent.

Now, while it actually is not a state requirement for students to pass the Algebra II EOC Exam to get credit for passing the course (another quality control issue), it must also be kept in mind that this assessment is created for Kentucky by the ACT, Inc. This is the same organization that creates the ACT college entrance test, so scoring proficient or more on the Algebra II EOC test is probably a rather valid indication of true competency in the subject.

With that said, recall again that Kentucky’s statewide average Algebra II EOC Exam proficiency rate in 2013-14 was only 37.9 percent. This means, as the next column in the table shows, that across Kentucky there was an enormous, 50.1-point difference between the percentage of students who got a diploma and a fairly compelling indicator that many of those students actually did not truly meet the state’s Algebra II-to-graduate competency requirement. This Algebra II evidence raises very serious doubts about the quality control of diploma awards in the state.

But, there is more.

The Effective High School Graduation Rate mentioned earlier can be calculated for any school or school system in Kentucky as well as for the state as a whole. You can read a more extensive description about the calculation of the Effective High School Graduation Rate in the Blacks Falling Through Gaps report previously mentioned; however, the number basically shows the percentage of entering ninth grade students who graduated on time with at least the minimally necessary education to qualify as either college and/or career ready using the Kentucky Board of Education’s officially adopted methods to determine such readiness.

The percentage of graduates that demonstrated readiness is shown in the “Reported College and/or Career Ready Rate For 2015” column in the table. For Kentucky’s high school class of 2015, this figure was only 66.9 percent. Thus, among the 88 out of every 100 entering ninth graders in the Class of 2015 cohort that graduated on time, only 66.9 percent of those 88 students, or about 59 students [which comes from rounding the 58.9 figure shown in the column headed “Effective High School Graduation Rate 2015 (Calculated),”] actually got a meaningful education even using the somewhat undemanding college and/or career criteria established by the Kentucky Board of Education.

The other students that entered the ninth grade with this class either didn’t graduate on time, or at all, or got a watered down, “Hollow Diploma” that actually does not signify readiness for things the state deems meaningful.

Very simply, if Kentucky were using its own college and/or career ready statistics to hold diploma awards honest, the state’s 2015 high school graduation rate would only be 58.9 percent.

Statewide, the difference between the officially reported 4-year adjusted cohort high school grad rate for 2015 and the Effective High School Graduation Rate was 29.1 points.

The table above also includes an analysis of the graduation rate statistics for the five school districts used for case studies in the JHU report. As you can see, there is considerable variation in the quality of diploma awards even across just these five Kentucky districts. Four of the five districts have differences between their officially reported cohort graduation rates and their Algebra II proficiency rates that exceed the already very disturbing statewide average.

When we examine the evidence from the college and/or career ready indicator, three of the districts exceed the disturbing statewide difference between the reported and effective graduation rates. In fact Covington actually leads the state with the biggest disparity of all.

In closing, evidence regarding the lack of consistent standards for the award of high school diplomas in Kentucky is compelling and very disturbing. Clearly, there are no consistent standards for diploma awards, not even within single districts like Jefferson County. It is also clear that the state isn’t even paying attention to its own stated requirement that competency in Algebra II is a must to graduate.

Given the obvious and very uneven inflation in the Kentucky graduation rates, it is clearly impossible to fairly compare those published rates to those from other states (which could have even higher inflation in their numbers, of course).

Thus, the basic metric used in the For All Kids report is fatally flawed. Attempts to draw education performance judgements from the current Kentucky graduation statistics are simply not valid.