More on the Harvard study that ranked Kentucky so highly
A couple of days ago, I discussed problems in a new report from a group at Harvard University that claims Kentucky made some of the best progress in the nation on federal testing between 1992 and 2011.
Earlier today, I talked about that report live on WLLV radio in Louisville. Some of the test results we discussed on air help explain my concerns about the Harvard study, so I thought our readers statewide would like to know the story, too.
The reading proficiency rates in this table are from the 2011 National Assessment of Educational Progress (NAEP) eighth grade reading assessment. These are the percentages of students who score at or above what the NAEP has determined to be a “Proficient” level of reading.
G8 Read NAEP P Rates for US, KY and JeffCo
If you look carefully, you will see that Kentucky exceeds the national reading proficiency rate when you only look at overall scores (yellow highlighted section of table). However, Kentucky actually lags the national proficiency rates for BOTH of the two dominant racial groups in the Bluegrass State’s public schools (whites in the white shaded section, and blacks in the black shaded section).How can that be? How can we supposedly be better overall, but behind for both of our two main racial groups?
Why does this raise issues for the Harvard report?
Click the “Read more” link to find out.
The first part of the table, highlighted in yellow, shows the overall average percentages of all students in each geographic area who scored at or above the performance level the NAEP grades as “Proficient.” The nationwide average is listed first (US), followed by the Kentucky statewide average (KY) and finally the average for students in the Jefferson County Public School District (Jeff. Co.).
Note in particular that when all students’ scores are considered, Kentucky notably outperformed the US average, scoring 36 percent proficient against the national average of only 32 percent proficient. You’ve heard about this before from our department of education, the Prichard Committee and a bunch of others for some time, of course.
However, things change when we look at scores only for whites. Kentucky’s whites score behind the US average for whites in reading. Whites comprised about 84% of the student enrollment in Kentucky’s public schools in 2011.Things get more interesting when we examine the situation for black students’ scores (NAEP uses the term “black,” by the way, though one radio call-in commenter quite properly pointed out today that African-Americans are not actually black in color and some consider this an insult. There is no insult intended here. For clarity, I am just using the terminology used in the actual NAEP reports).Repeating the situation for the whites, Kentucky’s blacks also score behind the national average for eight grade reading. Furthermore, blacks score WAY below the whites.
As a note, the NAEP Reading Report Card for 2011 indicates that blacks account for about 10% of Kentucky’s school enrollment as of 2011. The remaining six percent comes from a small number of Hispanics, Asians, Pacific Islanders, American Indians and multi-racial students.
To summarize to this point, broken out by race, the two dominant populations in Kentucky’s public schools BOTH score below their counterparts in other areas of the country. How is it therefore possible that the overall scores for all students trend the other way?
The answer is that, proportionately, Kentucky has a lot more whites than almost every other state in the nation. Even though Kentucky’s whites score below the national white average, they score well above the averages for the minorities. So, by having a lot more whites than other states, Kentucky gets an unfair boost in any simple comparison that only looks at “all students” scores.
One more point: Kentucky had a much higher than average rate of exclusion of students with learning disabilities on the 2011 NAEP reading assessments in both fourth and eighth grade. In the eighth grade, Kentucky excluded a nation-leading 7 percent of the entire raw sample of all students, disabled and non-disabled, that the NAEP wanted to test. Nationally, only three percent of the entire student population was excluded due to learning disabilities. Jefferson County exclusion was also higher than average. As a result, the Kentucky and Jefferson County proficiency rates shown in the table above are somewhat inflated, which makes the disparities even worse than I describe here. The Harvard study neatly overlooks that fact, too.
To sum up: Once you break the NAEP data down, as all NAEP Report Cards since 2005 say we should do, it becomes very clear that Kentucky is not really a leader in reading in the eighth grade.
Still, that didn’t stop a Kentucky Department of Education spokesperson from commenting about the Harvard report, telling the Glasgow Daily Times on July 21. 2012 (subscription) that she didn’t “see how demographic changes affect the validity of the report, let alone the validity of the NAEP, or it’s long-standing, professional relationship with the KDE.”
Well, I guess the department isn’t reading NAEP report cards or really talking to the professionals who run the NAEP and write those report cards.
Apparently, folks at Harvard are not, either.
Because, as I already mentioned, the need to consider student demographics and exclusion rates isn’t a new deal. NAEP Report Cards since 2005 have pointed out that you need to do this.
And, the Kentucky example I outlined above proves the people who run the NAEP know what they are talking about.
Technical Notes:
Data comes from the NAEP Reading Report Card for 2011 and the NAEP Trial Urban District Reading Report Card for 2011
Also, I have ignored the sampling error in the results shown for clarity. In fact, some small differences in proficiency rates shown above are probably not statistically significant, but that does not materially alter the main point of the discussion.