Kentucky drops out of Common Core testing group – Part 2 — The Bluegrass Institute for Public Policy Solutions

Is this related to testing problems?

A January 31, 2014 letter from Kentucky Governor Steve Beshear and the state’s education leaders again focuses national attention on the Bluegrass State’s implementation headaches with the Common Core State Standards. For sure, as the leading state for Common Core implementation, headaches here are better developed than anywhere else in the nation.

The Beshear letter says:

“Kentucky anticipates that it may have to engage in a Request for Proposal (RFP) process as part of the next wave of assessments for the Commonwealth. Kentucky state law requires a fair and equitable RFP process. We want to ensure that PARCC has the opportunity to participate in this process as a potential bidder, if it wishes to do so, without any perception of a conflict of interest or bias being created. Accordingly, it makes practical sense for Kentucky to withdraw from PARCC, before any competitive bid process is initiated.”

Potential bidding preferences aside, Kentucky’s new action in the testing scene raises a number of questions, none more serious than the indication that the state is dissatisfied with its current Common Core State Standards testing program and may be getting ready to rebid it.

The state’s current Common Core State Standards based testing program is called the Kentucky Performance Rating for Educational Progress (KPREP). It includes two separate elements, grade-by-grade testing in math and reading in grades three to eight, currently provided by Pearson Publishers, and high school End-of-Course (EOC) testing from the ACT, Inc.

I discussed the notable problems with the EOC program – total removal of open-response questions and collapse of online testing last year – in the first blog on this topic.

Now, I examine initial test results from the first two years of grade three to eight KPREP, which – while limited – does raise concerns about scoring inflation.

To begin, we have two years of testing data from KPREP for 2012 and 2013. That gives us both scores and the beginning of a trend line.

We can compare that KPREP information to Kentucky’s performance on the National Assessment of Educational Progress (NAEP) reading and math assessments for fourth and eighth grade students. Unfortunately, NAEP was not offered in 2012, but it was offered in 2011. So, I can “sandwich” the KPREP data within NAEP data for nearly the same time frame. Because Kentucky’s NAEP data has not been changing much in short intervals, this gives us a useful, though admittedly not perfect, comparison.

Now, examine this first graph, which shows how the NAEP and KPREP in Grade 4 reading compare over the past few years.

Notice that for each racial group, the first data bar shows the 2011 NAEP proficiency rate. Then, the two years of KPREP data for 2012 and 2013 are displayed. The “sandwich” for each racial group is completed with the 2013 NAEP testing results.

Let’s look at the most disturbing information in the graph first, which is the right hand set of bars for Kentucky’s Hispanic student performance. NAEP in 2011 reported that 35 percent of Kentucky’s fourth grade Hispanics read proficiently. Two years later, the NAEP reported a notable, 6-point drop in their reading proficiency rate to just 29 percent.

Now, look at what the two “sandwiched” KPREP scores show. Per KPREP, Hispanic reading improved from 36.1 percent to 37.2 percent proficiency between 2012 and 2013. The fairly close agreement between the NAEP in 2011 and the KPREP in 2012 was erased in 2013 when the proficiency gap between the two tests increased to more than eight points.

A similar situation occurred for Kentucky’s black students. NAEP showed a 4-point drop in reading proficiency. KPREP reported a slight improvement. The disparity between the 2011 NAEP and the first year of KPREP testing was 7.5 percentage points. It rose to 12.1 percentage points in 2013.

In contrast, the amount of change in Kentucky’s White trends in Grade 4 reading tracked pretty well between the two assessments, but the discrepancy in proficiency rates was already high. In 2013, the same cohort of whites only were 39 percent proficient on the NAEP, but KPREP claimed more than half, 52.3 percent were proficient, a gap of 13.3 percentage points.

Here is a similar graph covering the eighth grade math comparison. The differing trends and the growth in KPREP inflation are quite notable here for all three races.

Taking the data in order from left to right this time, Kentucky’s white proficiency rate on NAEP Grade 8 math stayed completely flat at 33 percent between 2011 and 2013. In notable contrast, KPREP math testing reported a nearly 4-point rise in proficiency for Kentucky’s whites over nearly the same time interval. In 2013 the NAEP-KPREP proficiency gap was 15 percentage points.

Blacks experienced a 1-point decline in NAEP proficiency over the same time frame while KPREP reported a rise in math performance for these students. As of 2013 the reported Grade 8 math proficiency rate discrepancy was 13.4 percentage points for Kentucky’s blacks.

Hispanic KPREP-NAEP discrepancies were even more pronounced. While NAEP shows a statistically insignificant 1-point drop in proficiency, KPREP claimed a substantial improvement of more than 5 percentage points. The gap in Hispanic proficiency rates reported by NAEP and KPREP in 2013 was 22.2 percentage points, a very serious difference pointing to considerable inflation in the KPREP scores.

I also have graphs for Grade 4 math and Grade 8 reading along with overall student averages, as well. Click the “Read more” link to see those. The situations are not quite as dramatic as those in the graphs above, but KPREP is still generally over-reporting proficiency compared to the NAEP.

And, that might also explain why Kentucky’s educators seem to be sending signals that they are planning assessment changes after only two years of KPREP operations. It also might explain why a bill to end Kentucky’s involvement, while of uncertain future, does have multiple sponsors in the legislature.

Here is the comparison graph for NAEP and KPREP testing in Grade 4 math.

Note in 2013 that white performance differences on the two assessments were quite small and the performance trend also seems to match.

For blacks, a 2-point rise in NAEP proficiency was slightly outpaced by a somewhat larger 3.4-point rise on KPREP. However, the discrepancies for blacks are also not very notable.

Now, look at the Hispanic performance. NAEP reported totally flat Hispanic performance between 2011 and 2013. KPREP in sharp contrast says Hispanic math proficiency in Kentucky’s fourth grade classrooms shot up by more than eight percentage points. That sharp difference raises some questions about differential test functioning for different racial groups in KPREP math.

Now, here is the eighth grade reading graph.

For both whites and blacks, the rather shallow trend on NAEP isn’t matched by a more notable rise in proficiency rates on KPREP Grade 8 math.

However, once again, it is Hispanic performance that seems most out of line. Perfectly flat NAEP performance for Hispanics in Kentucky stands in sharp contrast to a rather notable claim of a one-year, 5.8 percentage point climb in KPREP proficiency.

In 2013, the KPREP to NAEP proficiency rate gap was 14.5 points for whites. Kentucky’s white grade 8 enrollment in the 2012-13 school year was 41,331 students. Thus, the KPREP reported nearly 6,000 more white eighth grade students were proficient readers than the NAEP reported. That is a whole lot of students and their parents who think they are on track when they probably are not.

The discrepancies for blacks and Hispanics total well over 1,000 more students that KPREP said were proficient and NAEP said were not, as well.

Finally, here is an overall summary for the average performance of all students of all races.

Once again, even with the smoothing of differences such averaging always brings, the discrepancies in KPREP and NAEP reporting are evident across the board and point to faster increases in KPREP than in NAEP, as well.

All of this contributes initial evidence (I wish we had another two years of data) that all is not well with KPREP. It seems like the test may already be inflating, just as happened in the past with Kentucky’s now defunct KIRIS and CATS assessments.

So, there is reason to believe Kentucky’s education leaders may be thinking about testing changes. That reopens some questions from the first post.

What happens to KPREP testing trend lines if totally new testing contractors come on board, probably with new tests?

Changing the tests only a year or so from now (PARCC tests are supposed to be available for the 2014-15 school year) would mean the current KPREP tests and their trend lines would be set aside after only around three years of use. We would probably have to start totally new testing trend lines. That would leave Kentucky without solid evidence about Common Core performance for an awfully long time. That leads to another question.

Will Kentucky’s patience with its educators finally wear out?

Kentucky has run a series of educational reform efforts ever since the passage of the Kentucky Education Reform Act of 1990. The state’s nearly quarter of a century of expensive education reforms has already seen the rise and demise of two major testing programs (KIRIS and CATS). At present there is legislation pending (House Bill 215) that would totally end Kentucky’s involvement with Common Core State Standards. While that bill’s future is far from certain, the fact that it has multiple co-sponsors certainly indicates a growing level of unease with Common Core and all related activity in Kentucky.