The first Program Review audits are in – and “Just awful”
The Kentucky Board of Education is meeting as I write this, and I am listening in to the webcast. They just discussed results from an important audit related to the state’s Unbridled Learning school accountability program. This audit looked at scoring from the new Program Reviews part of Unbridled Learning. The audit found a solid majority of the Program Review scores were inflated.
At best, “It’s kind of discouraging” as KBE chair Roger Marcum summarized.
It is no surprise to me and our readers that the audit found significant inflation in the Program Review scores. After all, school staff self-award their own Program Review scores. What else do you think would happen when those scores are used to hold schools and staff accountable?
These scoring errors impact the validity of the state’s Unbridled Learning school accountability program.
Furthermore, a real fix quite likely will not be found.
How can the state board be so naïve about human nature? That really is “discouraging.”
What is a Program Review?
One feature of Senate Bill 1 from the Kentucky Legislature’s 2009 Regular Legislative Session was an attempt to do something about the dubious results and possible adverse consequences from tests that were being administered in highly subjective areas like the arts and the humanities. There also was recognition that the state wasn’t getting reliable scores from the writing portfolio part of the old CATS assessments, either.
Thus was born the idea of “Program Reviews.” These reviews would use experts to examine how well programs in things like writing portfolios and the arts were doing in each school. Student scores would not be considered. In most subject areas, no tests would be used.
Hopes collide with reality
The Program Review seemed like an interesting idea, but there was an unfortunate hitch – WHO would conduct all those reviews?
In the end, fiscal realities took over. Ignoring the lessons of human nature from the old KIRIS and CATS writing-portfolios-in-assessment mess, Kentucky’s education leaders decided that local school staff would score their own Program Reviews. This decision was almost sure to cause problems, and it didn’t take long for the first evidence to appear. I already pointed to evidence of Program Review score inflation here, long before today’s state board meeting.
Now, with the undeniable results of its own audit in hand, the state board has been confronted with the evidence that Program Review scoring is indeed highly problematic.
For whatever reason, the actual audit report was not made available in the online advance agenda for the board’s meeting. I’ll try to get a copy, but as I heard it this morning, the shocking overall summary of the main finding is that in 63 percent of the cases audited, the audit team found schools self-awarded a Program Review score that was too high. Under questioning from the board, it was revealed that all the scoring errors were on the side of inflation. No school awarded itself a score that was too low.
Inevitably, some board members rushed to tone down the seriousness of the report. However, board member Sam Hinkle laid it out rather succinctly, saying that this is just awful, right?
Program Reviews are a big problem for the board because starting with the 2013-14 school term the first three Program Reviews count in Kentucky’s Unbridled Learning school assessment formulas. Two more Program Review areas are going to be added, as well. Erroneous scoring of the Program Reviews not only provides bad information to policymakers and the public about the performance of those specific programs, but now those errors reduce the overall validity of the final scores from Unbridled Learning as well.
The overall impact on Unbridled Learning could be notable. At one point, the five proposed Program Reviews were going to collectively count for as much as 30 percent of the overall Unbridled Learning score. The state board is now looking at reduced weighting for Program Reviews, but providing any weight to these easily inflated items will certainly impact Unbridled Learning scores overall. Also, the board heard that next year a grand total of just 24 schools will be audited. With 1,233 schools in Kentucky, that drop in the bucket audit plan isn’t likely to fix much.
Furthermore, the problem with teachers and schools self-scoring Unbridled Learning items is going to get worse. Aside from those two additional Program Reviews (World Languages and K to 3 Program) still coming on line, the new Professional Growth and Effectiveness System (PGES), in which schools score their teachers’ and principals’ performance, also uses self-scoring.
Basic human nature tells us that all the inflation found in the new Program Review audits will be replicated with the PGES, as well.
In the end, we could wind up with schools that have pretty good overall Unbridled Learning scores even though their math and reading test results are lousy. We’ll be watching for that. Absent some major changes to the Program Reviews and PGES models, that inflation is almost inevitably going to happen.