TV ads from the Kentuckians for Reform in Education, or KARE coalition, which recently formed to fight for charter schools in Kentucky, have stirred the Prichard Committee to again venture into dubious analytical waters with the National Assessment of Educational Progress (NAEP).

The Bluegrass Institute is not a part of KARE, but we are always concerned when school performance data is incompletely, if not outright misleadingly, presented. So, I decided to add a very important and pesky ‘asterisk item’ back into the discussion.

Very simply, Prichard ‘conveniently’ overlooks the fact that Kentucky’s way-above-average exclusion of students with learning disabilities from the 2011 NAEP reading assessments brings the validity of our reading scores and the comparability of our scores to other states into serious question.

Prichard cites Kentucky’s performance on the NAEP reading assessments in 2011. So, lets get this big ‘asterisk’ note out there loud and clear: the validity of NAEP reading results for states with high exclusion rates (Kentucky led the nation for reading exclusion in 2011) are called into question by multiple agencies in multiple sources. Two very recent examples include:

• The new “Report Card on American Education” from the American Legislative Research Council (ALEC) refuses to even rank Kentucky’s reading results in most of its graphs because our state excluded a large number of students with learning disabilities from this federal testing program. The ALEC report specifically uses Kentucky data as part of their argument to justify this refusal to rank our NAEP reading scores (see page 31 of the report).

• Florida’s Commissioner of Education is asking the people who run the NAEP to suppress scores in the future for states that don’t meet NAEP inclusion guidelines for learning disabled students and students with learning disabilities. The Commissioner is tired of people like Prichard making unfair comparisons with this biased data.

Just a few of the older examples of people questioning NAEP score validity when states exclude more students include:

• A July 2005 study from the General Accounting Office, which looked at NAEP inclusion rates in a study on how states were doing with No Child Left Behind. The GAO report says:

“Because states had different exclusion rates, ranging from 2 percent to 10 percent in the 2002 NAEP, comparisons of student achievement across states may have limitations.” (Pg 3)

• The NAEP Validity Studies Panel looked at the exclusion issue in “NAEP Validity Studies: An Agenda For NAEP Validity Research” in 2003.The panel concluded:

“With regard to students excluded because of disabilities or limited English proficiency, variations in exclusion rates over time have already led to challenges of the validity of NAEP’s reports of gains in some states. Panel members therefore attached “High” importance to research into methods to include “excluded” students in population estimates.”

• A somewhat recent report, “Evaluation of the National Assessment of Educational Progress,” conducted for the National Center for Educational Statistics in 2009, continues to raise concerns that uneven exclusion of students impacts score validity.

This report says:“…the lack of consistency in the inclusion of students with disabilities has called the validity of NAEP trends into question (Forgione 1999; McLaughlin 2000, 2001, 2003).” (Pg. 1)

So, to close this blog item, there is plenty of concern about the validity of NAEP scores in states like Kentucky with unusually high exclusion rates (In both fourth and eighth grade reading, Kentucky led the nation for its percentage of students that got excluded from the raw samples NAEP wanted to test). Some now refuse to recognize them.

But, don’t expect to find any of this information in a Prichard blog. Prichard has been well advised of this issue several times before and refuses to publish this huge, pesky ‘asterisk note’ to their claims about NAEP reading. That might be clever politics, but it’s not the way to do good research.

And, as far as Prichard’s claim that Kentucky’s educational system is doing just ‘hunky dory,’ take a look at the NAEP grade 8 math results. In math, Kentucky’s exclusion rates are in line with the rest of the country. When we look at the data, and allow for statistical sampling error in that data (another necessity that Prichard also ignores), it is painfully clear that for the vast majority of our students – whites (which comprise 84 percent of the students in Kentucky’s schools) – Kentucky is performing behind almost all of the rest of the nation, including a host of charter school states. Kentucky’s math performance shows we need some dramatic changes in our school system, which is actually the basic thrust of the KARE commercials.