Getting test results wrong leads to bad education decisions — The Bluegrass Institute for Public Policy Solutions

A good example of the problems with interpreting real state academic performance surfaced today during a radio talk show at WLMW FM in New Hampshire.

Michael Petrilli, Executive Vice President of the Thomas B. Fordham Institute, and show host Rich Girard were discussing examples of states that actually had to lower their standards to sign on to the Common Core State Standards. Massachusetts naturally came up, but then Petrilli claimed (around 24 minutes and 43 seconds into the second segment) that while California also had great standards before Common Core, those standards had not worked. Said Petrilli, California’s “achievement’s been terrible.”

Well, that’s just not right.

But you have to really understand testing data to learn what has actually been going on in California and why that state’s accepting Common Core could be a huge mistake. Fortunately, it really isn’t complicated, so if you want to learn more, just click the “Read more” link.

California’s progress in education is largely hidden by a stunning demographic fact of life. Since the National Assessment of Educational Progress (NAEP) started to do state level testing in the early 1990’s, California’s public school population has done a complete flip-flop. Data captured by the NAEP shows whites comprised the majority of the state’s fourth graders in the 1992 NAEP Grade 4 Reading Assessment. By 2011, it was Hispanics who filled the majority of seats in the Golden State’s public school classrooms.

And, because Hispanics score much lower than whites, having a much larger percentage of Hispanics drags California’s NAEP scores down. So, let’s look more deeply than Mr. Petrilli and his sources did at what is really going on.

This next graph shows two sets of scores for California. The first set of data, shown by the blue line, includes the actual overall average scores for all students that NAEP reported for Grade 4 reading in California.

Graph, CA G4 Read for Reported and Constant Demographics

Between 1992, the first time NAEP did state testing for Grade 4 reading, and 2011, California’s overall scale score average increased 9 points.

But, what would happen if California’s demographics had not changed for whites, blacks, Hispanics and Asian/Pacific Islanders? The red line shows how the scores for each of these racial groups would average together if California’s racial demographics had remained at the 1992 levels. Absent those demographic shifts, California’s score rise would have been – not just 9 points – but a much more remarkable 17 points, nearly twice as much progress.

By the way, the national average NAEP Grade 4 scale score in 2011 was 220. If California had not experienced a Tsunami of immigration, it would have had a statistical tie with the nation.

Let’s break this down in another way. The next graph shows actual scores for Grade 4 NAEP reading for whites in California and across the nation. Back in 1992, California’s whites lagged their counterparts elsewhere by six points. By 2011, California closed the gap to one point, which probably isn’t a significant difference.

Now, look at what happens for black students.

In 1992, California’s blacks lagged their racial counterparts elsewhere by 10 points. Things look very different by 2011, where California’s blacks moved three points ahead of the rest of the blacks in the nation.

The story generally repeats for Hispanics. In 1992 the state lagged the national Hispanic NAEP Grade 4 reading scale score by 14 points. By 2011 California cut that deficit in half despite the fact that many of the state’s Hispanics in 2011 also were English Language Learners.

Finally, here is how California’s last significant student population, the Asian/Pacific Islander group, performed.

The Golden State’s Asian/Pac. Islanders cut their gap from eight points to only an insignificant one point between 1992 and 2011.

Thus, for every single significant population in California, there has been progress on the NAEP reading assessments.

Oh, one more point. Regular readers know I make a regular issue about exclusion rate changes on the NAEP and how they can impact results.

Well, in 1992 California excluded four percent of its raw sample of all students NAEP wanted to test due to learning disabilities. By 2011, they cut that to just two percent of the raw sample. In 1992 California excluded 11 percent of its English Language Learners. In 2011 California dropped that to just one percent of the raw sample NAEP wanted to test. Those trends should have further depressed California’s scores somewhat.

Basically, if you understand how to really interpret the NAEP data, by any reasonable measure California actually has come from notably behind to basically match the nation for reading.

And, that, as radio host Girard would point out, makes California’s signing on to the Common Core very disconcerting. If California’s real trends of progress I highlight above are not continued because the state adopted lower standards, that will be a real shame.

Tech. Notes

The NAEP scores were extracted from the NAEP Data Explorer.

The exclusion information comes from the 2009 and 2011 NAEP Reading Report Cards.