Is there a backlash growing over Kentucky’s proposed school accountability system?

Unbridled Learning, Kentucky’s current public-school assessment and accountability system, is on the way out, and Education Commissioner Stephen Pruitt and the Kentucky Department of Education have been working on an as-yet unnamed replacement accountability system for some time. Pruitt and his team have held two sets of public hearings seeking Kentuckians’ input into the new program and he formed several advisory committees to further develop ideas for the new system.

Now, a proposed system is starting to take form. The Kentucky Board of Education took its first formal look at the proposal in June, and a follow-up discussion is expected during the board’s August meeting.

Surprisingly, amid this movement toward finalizing Kentucky’s new accountability program – which, by law, must be submitted to Washington, DC for approval by mid-September – a curious letter appeared last week, co-signed by leaders of several organizations including the Kentucky Chamber of Commerce and the Prichard Committee for Academic Excellence, who were members of one of Pruitt’s key advisory committees.

It almost seems like the letter is, to use technical lingo, a minority dissenting report.

Here are some thoughts about major areas of concern mentioned in the letter.

1) The comment about "Performance of all students and each student group should be included in the ratings – not just averages" relates to something major that IS NOT in the current version of the draft proposed 703 KAR 5:270 regulation.

The proposed regulation talks about averaging all sorts of things together to reduce everything down to just one, overall one-to-five-star rating for each school, PRECISELY the same error the KDE has made with each past accountability system.

When you average everything together before making accountability decisions, you ALWAYS will wind up overlooking serious performance problems for student subgroups.

For example, as pointed out in the 2016 BIPPS update report about blacks still falling through gaps in Louisville (online here), Dunn Elementary was top rated as a "Distinguished School" under Unbridled Learning despite its enormous, fifty-point plus white minus black math proficiency achievement gap. It’s just not right to rank a school with such enormous achievement gaps so highly, but after averaging a lot of scores and other things together, Unbridled Learning – the state’s most recent accountability program – didn't fault Dunn in any way.

We’ve also blogged about how averaging hides problems on several occasions such as here and here.

Now, it looks like the Chamber, Prichard and others have finally figured this out. We welcome them aboard the train we’ve been driving for quite awhile now. More importantly, it’s problematic for the proposed accountability system when these groups and BIPPS all see an unresolved issue here.

2) Regarding "Specific, ambitious goals must be specified," the draft regulation lacks many specifics needed to fully understand the proposed new system. In more than one place in the regulation, scoring tables needed to compute scores are still left to be worked out. That can’t happen until after the first round of testing with what will probably be new tests because all testing contracts are up for rebid soon.

Right now, no one can project what the results from the proposed accountability system will look like because major work – such as developing new-cut scores for what will be a new scoring scheme with "Low Novice," "High Novice," "Low Apprentice," "High Apprentice," "Proficient" and "Distinguished" – simply yet been done.

3) "Achievement must reflect growth toward and beyond proficiency." Place holders are in the draft regulation to examine student performance growth over time, but scoring specifics are another to-be-determined area. I doubt anyone at this time knows how this will work out.

4) The letter’s comment that "The five-star system lacks clarity and definition" is certainly due in part to many unknown scoring items. Do the letter writers think this five-point rating system will need to be rethought?

5) "The Local Measure indicator must not weaken local accountability" is indeed a problem area.

This new, potentially highly problematic proposal allows each local school district to pick one item for its performance measure from a list of possibilities.

Such an approach, which makes each school system’s rating somewhat unique and thus non-comparable to other schools in the commonwealth, will likely not be popular with Kentuckians. As a locally-selected measure, this item could be subject to something similar to the self-grading inflation found in Kentucky’s previous assessment programs. Self-grading inflation is something we’ve heavily criticized in the past.

Again, we’re pleased that the Chamber, Prichard, etc., apparently are now in agreement with the Bluegrass Institute’s concerns about accountability-inflating elements.

6) The demand that “full and complete performance information must be made available to stakeholders” somewhat surprises us.

While it appears Pruitt is favorable towards such transparency, the letter’s writers seem to share our concerns that the currently proposed rating system in the draft regulation will largely ignore significant gap issues by averaging the gap scoring with a lot of other elements before making accountability decisions. Could this just repeat the mistake that led to the situation with Dunn Elementary where enormous achievement gaps are present even though the school gets absolutely top Unbridled Learning accountability scores?

One REALLY troublesome idea in the draft regulation is the creation of a racial "Reference Group" rather than consistently using white student scores to compare to other racial groups. Using this proposed approach will result in Asians – whose scores can be much higher than whites – becoming the “Reference Group” in around 10 percent of our schools. The other schools will use white scores for a MUCH different gap reference.

A look at the math results from London, Kentucky’s North Laurel Middle School, extracted from the school's 2016 Kentucky School Report Card, makes the potential problem quite clear:

North Laurel Middle 2016 KPREP Math by Race

In 2016, North Laurel had exactly 10 Asian students, the minimum number to report scores under Unbridled Learning and the current proposed minimum reporting enrollment for the new accountability program. In math, the North Laurel Middle School’s Asian students scored 50 percent Proficient and 50 percent Distinguished. Overall, the Asian proficiency rate was thus an astonishing 100 percent. That sets up a very tough gap target for even the white students in North Laurel Middle School, who only scored 52.8 percent Proficient or above.

However, if just one fewer Asian had enrolled in North Laurel, the Asian scores would have been suppressed. In that case, the proposed regulation says white students – who have a proficiency rate only about half of the Asians – would be the reference group at North Laurel Middle School. Clearly, the gap situation then would be dramatically different.

Could this type of system create pressure to force an Asian student out of North Laurel?

As a note, the actual gap calculation proposed in the new regulation would not use proficiency rates. The proposal would develop an index for each school based on the percentages of students scoring Novice, Apprentice, Proficient and Distinguished. This rather non-revealing index would then go through a somewhat complex statistical analysis to determine actual school performance for gaps. Details of how all of this would work out and how many schools would be identified as having gap problems are currently are unknown.

Thus, major questions remain unanswered regarding the treatment of achievement gaps. That is problematic when gaps are supposed to be one of the primary emphasis areas for the new accountability program.

7) "Parents must be more involved."

This is correct, but inclusion of this concern in the letter is a real surprise. Apparently, the folks writing the letter must not think Pruitt's efforts to date have been strong enough in this area. For Prichard, which has a major hand in running the Governor’s Commonwealth Institute for Parent Leadership, this comment comes as a special surprise. Prichard says their institute was

“created to provide multiple training opportunities for parents and community members to develop the capacity to support and advocate for successful public schools for all students.”

Can it be that after more than a quarter-century since passage of the Kentucky Education Reform Act of 1990, parents are still getting locked out too often, even when it comes to input into the accountability system that will impact their children’s schools? The letter seems to imply that is the case.

Anyway, most of the complaints in this astonishing letter seem on target. The big surprise is folks like the Chamber and Prichard now are saying this, and thereby agreeing with concerns we have held at BIPPS for some time, too.