“Our NAEP scores are in question” AND HOW!

Kentucky’s fourth and eighth grade NAEP reading scores probably untrustworthy

Kentucky leads nation for exclusion of students with learning disabilities on 2011 NAEP

Why did so many Kentucky kids get totally excluded from this test?

The remark caught my attention when he made it to the Kentucky Board of Education on October 5, 2011. Kentucky Commissioner of Education Terry Holliday admitted something I have been saying since 1999 – Kentucky’s testing policies for learning disabled students impact the validity of the state’s reading scores, both on state reading assessments and on the National Assessment of Educational Progress (NAEP).

Holliday specifically told the board:

“Our NAEP scores are in question.”

He’s right. Now I know more about how right he is.

To learn why, click the "Read more" link and prepare to be shocked.

The graph tells the story. It shows the percentage of all students that the NAEP wanted to test for fourth grade reading each year that ultimately got excluded as too learning disabled to sit for the NAEP assessment. Data for every year that the state fourth grade NAEP reading assessment has been conducted are included for both Kentucky and the nation.

G4 NAEP Exclusion Rate Graph for KY and Nation

Table Sources: Data Source: NAEP 2009 Reading Report Card Table A-5NAEP 2011 Reading Report Card, Table A-6Back in 1992, even though no testing accommodations were allowed, Kentucky only excluded four percent of the total raw sample of students the NAEP wanted to test. Across the nation, the average exclusion rate was one point higher.

That all changed dramatically by 1998. Even though the NAEP began allowing some testing accommodations for students with learning disabilities, Kentucky’s exclusion rate increased dramatically.

That raised questions about whether Kentucky’s scores could be fairly compared to scores in other states, or even to its own earlier scores from 1992 and 1994.

Flash forward to 2011. Across the nation, most states have gotten much more aggressive about teaching learning disabled students to read and evaluating how well they are doing. As a result, in 2011 the NAEP national average exclusion rate for learning disabled students DROPPED from the historic level of five percent of the total raw sample to only three percent.

In notable contrast, Kentucky’s already high exclusion rate for learning disabled students on the fourth grade NAEP reading assessment ROSE yet another point in 2011 from the level posted in 2007 and 2009. In consequence, while the national exclusion rate in 2011 was only three percent – a historic low – Kentucky’s rate was now far more than two times higher, at eight percent.

Very simply, Kentucky excluded eight out of every 100 students from the 2011 forth grade NAEP Reading Assessment. Those excluded students as a group would unquestionably score very low on NAEP had they been allowed to participate. In fact, for reasons I discuss further below, most of those students probably can’t read at all and would likely receive close to a non-performance score if they were presented with a NAEP test booklet.

Sadly, the situation is almost as bad for the eighth grade NAEP reading results. Kentucky excluded seven percent of its eighth grade NAEP reading sample, while across the nation only three percent got excluded.

So, I won’t waste much more time on NAEP reading results for 2011. After more than a decade of research, the NAEP has been unable to develop a methodology to correct scores for states like Kentucky that don’t have equitable participation of learning disabled students on the NAEP. The best I can advise is to treat our NAEP reading scores as untrustworthy.

Why is this happening?

From the early days of KERA, it actually has been permissible for Kentucky’s teachers to read all parts of the state assessments, including the so-called reading assessments, to a notable proportion of the learning disabled students in this state.

So long as a learning disabled child’s Individual Education Plan (IEP) calls for tests to be read to that student, then all tests that student takes, including state reading tests, are indeed read to the student.

One serious consequence of this very bad policy is that schools can escape all responsibility to try to teach the child to read – ever. The school system can just create IEPs with the reading accommodation for the student’s entire school experience – elementary school through high school graduation.

Of course, state tests indicate to the public that all is well, because the read-to student’s scores are reported and counted just the same as other students’ scores. Schools look good, parents and students are lulled into a sense of well-being, but the child winds up graduating (or dropping out) totally illiterate.

I don’t know precisely how many children have been denied the opportunity to learn to read because of this horrible policy, but I suspect over the past two decades this runs into many thousands. I’d bet more than a few of these poorly served individuals are currently found in Kentucky’s burgeoning penal system.

Kentucky’s extraordinarily bad testing policy also set up a collision with the validity of Kentucky’s NAEP reading scores.

The federally operated NAEP always intended its “reading” test to be a real test of printed text reading and comprehension, not just a spoken word comprehension test. Thus, NAEP test administration guidelines specifically prohibit reading the NAEP Reading Assessment to any student. When that NAEP rule conflicts with the student’s IEP, this federal test’s directions say the student is to be excluded from testing all together.

However, no-one at the NAEP anticipated how many students were going to get the reading accommodation in their Kentucky IEPs.

The conflict in rules came to a head with the 1998 NAEP Grade 4 Reading Assessment. Kentucky’s exclusion of students with learning disabilities on NAEP skyrocketed by that year, increasing from 4 percent of the raw sample of all students NAEP wanted to test in 1994 to a whopping 10 percent exclusion in 1998 in the unaccommodated test sample. That’s right; one out of 10 of all students the NAEP wanted to test for fourth grade reading got excluded in 1998. Even in a trial run of the NAEP reading assessment that allowed test accommodations, Kentucky’s exclusion rate stayed high at seven percent (this 1998 trial run data is shown in the graph above. The unaccommodated information is not shown but can be found in the 1998 NAEP Reading Report Card).

Even today, with NAEP now allowing some other accommodations – but not the reading nonsense, Kentucky still excludes learning disabled students on the NAEP reading assessments at rates much higher than the national average.

You can’t exclude so many of your most educationally challenged students without inflating your test scores. After fighting this fact for several years, the folks who administer the NAEP finally had to agree.

NAEP’s administrators also were hearing from other states crying foul about Kentucky’s supposed nation-leading improvements in NAEP reading. Other states, you see, don’t allow such testing shenanigans on their reading assessments. They want their educators to at least make the attempt to teach all kids to read. In fact, in 2011 the NAEP 2011 Report Card data shows that 40 states had exclusion rates half or less than Kentucky just posted.

How come teachers in all those states can do a good enough job with so many of their kids that they CAN take the NAEP reading assessment while our kids cannot?

It’s now 2011. KERA has been on the books for more than two decades, and kids in Kentucky have been getting short-changed on reading instruction the whole time. State and NAEP reading tests results have both been getting corrupted all of this time as well.

Finally, it looks like the Kentucky Board of Education may act to fix this problem. The board is considering new testing rules that will bring us in line with NAEP standards and the normal standards in other states, where reading means READING, not listening.

Of course, there are some inevitable rumbles from some Kentucky educators who would prefer for this deceptive nonsense to continue.

So, stay tuned on this, but I think the board is leaning towards doing the right thing.

Too bad it’s taken 21 years – and, it trashed the value of our NAEP reading scores.