New “Diagnostic Reviews” of what is happening in four troubled Jefferson County Public Schools have been released, but the findings seem inconsistent. While the principal in only one school, Doss High, has been recommended for removal, test data for another school seems essentially the same, and just as dismal.

Overall, one is left wondering if it’s the reviews themselves that need some reviewing.

What is the school audit process and where did it come from?

In 2010 Kentucky gave up some of its sovereignty over its school system to the US Department of Education and started identifying and sanctioning its very worst performing schools due to requirements to get federal School Improvement Grants and Race to the Top funds.

Over the following years leading up to the start of the Unbridled Learning program in the 2011-12 school term a total of 41 schools – many in Jefferson County – were flagged as “Persistently Low-Achieving Schools” (PLAs) after posting the very lowest results for math and reading proficiency on state tests.

Among other things, PLAs were required to undergo a school leadership analysis, sometimes referred to as a “School Audit” or a “Diagnostic Review.” These audits, conducted by people from outside of the local school system, started in each school shortly after it became a PLAs. The audits are repeated on a two-year cycle while the school remains in PLAs status.

These audits are fairly serious business. Among other things, the audit team is supposed to include a finding on whether the principal in each low-achieving school has the capability to manage the needed improvement in that school. Those findings are supposed to be very carefully considered by the authority – usually the Kentucky Commissioner of Education – who renders the final decision on whether the principal stays or leaves.

Serious questions about school audits first raised in Fleming County High

For the most part, the audits I have seen seemed fairly well put together, but lately problems have started to surface. The first big question marks were raised following release of a follow-up audit in the Fleming County High School conducted not quite a year ago. As I wrote in early May of 2014, the auditors’ call for removal of the principal at Fleming County High stood at serious odds with the school’s very strong performance on Kentucky’s Unbridled Learning school accountability program. The problem wasn’t that the auditors were wrong – in fact the school’s performance in math and reading declined despite the overall Unbridled Learning scores – but rather that the audit team made no mention what-so-ever of the Unbridled Learning performance and why that performance didn’t override the call to remove the principal. The truth is that the audit’s findings raised serious concerns about Unbridled Learning. However, the audit team never discussed those very important issues.

In any event, Fleming County High students were true to their school and principal. The students staged a walkout that turned into a press event over the obvious inconsistency of calling for removal of a principal who led his school to amazingly high rankings in Unbridled Learning.

In the end, it was the Kentucky Commissioner of Education who found himself bridled. The commissioner was left with little other choice but to either condemn his own Unbridled Learning program or to override the recommendation of the audit team. Unbridled Learning remained and the principal remained, too, raising questions about the value of the leadership review.

New questions follow recent Jefferson County audits

Last week, the Courier-Journal ran “State review: JCPS should oust Doss Principal,” which discusses the results of four diagnostic reviews in the Doss, Seneca and Iroquois high schools and the Knight Middle School in Jefferson County (available here).

Among all four of the schools, only Doss High’s principal was recommended for removal.

After looking at the diagnostic reviews, I have to ask if that lone Doss recommendation is consistent with the evidence. In fact, I have to ask if the diagnostic reviews are even consistent on a purely technical basis.

Inconsistencies in new Jefferson County reviews

Consider some technical issues:

• Each high school’s report has an attachment for “Student Performance Data.” However, the attachment numbers vary. Even the collection of data found in each report varies. For example:

• Doss High and Iroquois High have their "Student Performance Data" sections posted as Attachment 3, but in the Seneca High report it is Attachment 1.

• Page numbers were omitted in the attachments sections in all the reports (a really big mistake -- how do you reference and easily find anything in reports running over 80 pages without full page numbers?).

• Despite the lack of printed page numbers in the report attachments, Adobe Acrobat does assign its own page numbers electronically. Using Adobe-assigned numbers, Doss High's Student Performance Data section starts on Page 74. In notable contrast, Seneca's begins on Page 37. That sort of major variation in the location of information makes it hard to find similar material in different reports.

• The Doss and Seneca reports have a data section titled "School Achievement of College and Career Readiness (CCR) and Graduation Rate Delivery Targets (2013-2014)." However, in the Iroquois report the closest comparable section is titled "School College and Career Ready Delivery Target (2013-2014)."

• The college and career readiness data listings in the Doss and Seneca reports list the graduation rate, but the graduation rate – again inconsistently – isn't listed in Iroquois’ report.

• There is an interesting number difference, too. The Doss and Seneca reports say the statewide College and Career Readiness Average Score was 62.4 in 2013-2014. The Iroquois report says it was 62.5. Since the Iroquois and Seneca reports were completed during the same, 3-day period, I doubt that the numbers were revised by the Kentucky Department of Education during the writing of these two reports. Anyway, according to the statewide School Report Card for 2013-14 (Get that here), the correct number is 62.5. Two of the three reviews have it incorrectly.

Overall, inconsistencies in the technical presentation of information in these reports make them more difficult to read and raise questions about the orderliness of the evaluations.

Things get more interesting when we compare school data from the new Jefferson County reports to see if Doss really is the only dog in this race. Keep in mind that only the review for Doss called for the principal to be removed. The other reviews say leave the Iroquois and Seneca high school leaders in place.• Consider each school's "Annual Measurable Objectives (AMO)" data section: In 2013-14 Doss met its AMO Goal, Participation Rate and Grad Rate. Iroquois missed its Grad Rate. Seneca flubbed its AMO. So, Doss performed better.

Wouldn’t meeting all goals indicate leadership is on track?

• Check the "Percentages of Students Scoring at Proficient/Distinguished (P/D) Levels on the K-PREP End-of-Course Assessments at the School and in the State (2011-2012, 2012-2013, 2013-2014)" section for Doss versus Iroquois: In 2013-14 Doss outscored Iroquois in English II, Algebra II, and Language Mechanics. The only area where Doss was notably behind Iroquois was biology.

• On the PLAN assessments, the only subject where Iroquois outscored Doss in 2013-14 was math.

• On the Grade 11 ACT, Doss outscored Iroquois in every subject area.

When we look at scores for the dominant racial minority in Louisville – the blacks – we learn still more. First, you won’t find that data in the diagnostic reviews. Apparently minority performance isn’t of much interest to the review teams. We have to go to the school report cards for that information.

It turns out Iroquois’ blacks slightly outperformed Doss’ blacks in 2013-2014 for reading, but neither school’s score is laudable. In Doss only 13.4 percent of the black students scored proficient or higher on reading. In Iroquois it was a scarcely different 14.6 percent. Furthermore, scores in both schools declined in this subject between the 2012-2013 and 2013-2014 school terms.

For math, blacks in both schools scored lower proficiency rates in 2013-2014 than they did in the launch year for KPREP testing in 2011-2012. The most recent math proficiency rates of 17.5 and 19.2 respectively are scarcely different, and certainly depressing.

The bottom line on this performance data seems to be that if Doss' principal needs to go, Iroquois' does, too. According to the school report card from the 2010-11 school term for Iroquois, Mr. Chris Perkins has been principal at the school since at least the 2011-12 school year. According to the new diagnostic review, he is still there. So, he has been in place for more than three school years. If the student performance data means anything, if Doss’ principal needs to go, I have to ask why Iroquois’ does not.