Courier-Journal picks up our concern about new state assessment elements

A lot of creative thinking has gone into Kentucky’s still relatively new Unbridled Learning school accountability system. However, Unbridled Learning is also running up against some tough “facts of life” that sometimes interfere with the picture this program presents about school performance. Things might get even more problematic when the new Unbridled Learning results for 2014 come out on Friday.

Veteran Courier reporter Chris Kenning, who has covered Kentucky education issues for many years, picks up on some of our concerns in his article today about, “School test results to include new element.” Kenning is also attuned to the possibility that new scoring elements in Unbridled Learning could add more “facts of life” problems for scoring accuracy on Friday.

We discuss some of those tough Unbridled Learning “facts of life” in our recent report, “Kentucky’s Unbridled Learning School Assessment Program: Can we trust what it’s telling us about schools?” The report shows that Unbridled Learning sometimes misses serious performance issues. For example, Unbridled Learning reported the Fleming County High School was doing great even though the math and reading scores at the school declined between 2012 and 2013. While Unbridled Learning awarded Fleming County High a “Proficient” rating overall, a recent audit of the functioning of the staff at the school called for removal of the principal. This is a very dramatic example of dubious ratings that Unbridled Learning sometimes provides.

There is also a very small indication that scores may be starting to inflate, at least for some subjects and grades. The graph below shows the proficiency rates in eighth grade math testing from the National Assessment of Educational Progress (NAEP) and from the Kentucky Performance Rating for Education Progress (KPREP) tests for the past few years.

G8 Math KPREP vs NAEP 2011 to 2013

Notice that Kentucky’s NAEP proficiency rates stayed essentially flat for whites, blacks and Hispanics across the years when KPREP showed notable increases in proficiency rates for all of these racial groups. KPREP also reported notably higher proficiency rates than the NAEP did.

Because KPREP is used for several calculations in Unbridled Learning, score inflation in KPREP will cause overall inflation in the state accountability figures for each school, as well. We really need to see more data before forming firm opinions, but this limited trend information indicates we need to pay attention.

Unfortunately, the inflation issue could take on an added dimension when the new, 2014 Unbridled Learning results are released this Friday. The reason involves new Unbridled Learning “Program Review” elements that were added this year. The new Program Reviews evaluate and grade things like the operation of the writing portfolio program in each school and each school’s arts and humanities programs. The score inflation potential is created by the fact that all three of the new Program Reviews for 2014 – plus two more still to come – are being self-graded by each school’s staff. Based on human nature and Kentucky’s previous experience with another staff self-graded portfolio program, there is significant chance for score inflation.

Self-awarded Program Review scores will have an important impact on the school’s final Unbridled Learning score (counting for 23 percent of the total this year, and even more in the future as other self-graded items are added). Temptation for educators to fudge is rather strong.

Certainly, it is hard to evaluate things like Kentucky’s writing portfolio program and a school’s real performance in arts and humanities subjects using a standardized test. We actually tried to do that with our old KIRIS and CATS programs. The facts of life are those tests never worked out.

While valid program reviews seem like a better idea that testing in some areas, the “facts of life” are that providing a lot of weight to school staff self-graded items creates potential for overall score inflation in Unbridled Learning. Absent some serious revisions to Unbridled Learning, we could see Kentucky’s new assessment program running into some severe credibility issues over the next few years, just as happened with both the old KIRIS and the more recent CATS assessments. Those old systems didn’t survive the loss of credibility.

I sincerely hope that does not happen, this time.