On problems with education research, state education rankings and teacher preparation

I wrote a couple of days ago about my concerns with an annual education rite of passage – Education Week’s latest report on education across the United States, known as Quality Counts (subscription).

Part 1 of my discussion addressed concerns about Quality Counts continuing to use its own Cumulative Promotion Index (CPI) calculations for high school graduation rates when extensive research completed for the US Department of Education in 2006 shows now readily available data from the Averaged Freshman Graduation Rate (AFGR) calculation is more accurate (find many years of AFGR data for all the states here).

In my first blog, I focused on Kentucky’s impacts. Using a less accurate graduation rate formula matters for Kentucky’s Quality Counts rankings.

In this addendum blog, I examine data for all the states to point out that Quality Counts’ use of less accurate graduation rate estimates matters a lot all across the nation. Arizona, Rhode Island, South Dakota and New Mexico have been hurt the most, but many other states also got poorly treated, as well.

Click on the “Read more” link to see data on all 50 states that shows which got very poorly treated by Quality Counts’ continued use of a less accurate, obsolescent graduation rate formula instead of the now standard Averaged Freshman Graduation Rate calculation.

I’ll use this table (click on it to enlarge, if necessary) to explain. The table was assembled from data on graduation rates from the Quality Counts’ “K-12 Achievement” section as well as the AFGR data mentioned above. It is sorted by the data in the green column, which I explain below.

Here is how to read this table. The yellow-shaded columns show the CPI data published in Quality Counts 2012. Looking at the top row of data for Hawaii, that state’s 2008 CPI graduation rate was 65.8 and this supposedly had increased by 3.5 percentage points from that state’s rate in 2000.The tan-colored columns show data published by the US Department of Education in the latest Digest of Education Statistics. According to this more accurate (based on extensive federally sponsored research) Averaged Freshman Graduation Rate data, Hawaii more likely had a 2007-2008 school year graduation rate of 76.0 percent, and this increased by 5.1 percentage points from the 70.9 percent rate back in the 1999-2000 school year.

Reading further to the right for Hawaii, we see in the green-shaded column that the difference in Hawaii’s 2008 CPI and AFGR was very large, 10.2 points.

Finally, from the rightmost column and the two columns showing changes from 2000 to 2008 in the CPI and the AFGR, we learn that Hawaii’s improvement in graduation rates according to the more accurate AFGR was not the 3.5 points Quality Counts showed, but rather was 1.6 points higher at 5.1 points.

So, Hawaii didn’t get treated fairly by the Quality Counts graduation rate data, given that federal research looked at both the CPI and AFGR and found the latter to be a better formula. In fact, the graduation rate for this state was under-reported by more than 10 points in Quality Counts 2012.

At the other extreme, New Jersey’s 2008 graduation rate was over-reported in Quality Counts by 2.3 percentage points.

Now, let’s sort the data in the table using the last column.

Here it is easy to see a lot of states didn’t get treated very well in Quality Counts 2012’s graduation rate section.

Worst treated was Arizona. According to Quality Counts, this state’s high school graduation rate dropped 0.2 point between 2000 and 2008. Using the more accurate AFGR, we learn the state actually improved its graduation rate by 7.1 points between those years, a difference of 7.3 percentage points and a different trend direction, as well.

Several more states, Rhode Island, South Dakota and New Mexico are not far behind, with improvement rates all better than five percentage points higher than Quality Counts reported.

At the other end of the spectrum, Florida received too much credit for its improvement in graduation rates. While Florida did make a very nice 5.9 percentage point improvement between 2000 and 2008, it was nowhere near the 14 point increase that Quality Counts claims. The difference was 8.1 points. Other states getting far too much credit for improvement (more than five points too much) include Washington, North Carolina, South Carolina and Nevada.

It is important to understand that every state in the union is being held to account under No Child Left Behind with either the AFGR or the still more accurate cohort graduation rate calculation, which eventually will become the universal standard once all states are able to calculate it (Kentucky will not have the capability to report its first cohort rate until 2014).

Why does Education Week continue to use its researched and found less accurate CPI? The CPI versus AFGR argument was settled way back in 2006 in Volume 1 and Volume 2 of “User’s Guide to Computing High School Graduation Rates,” a report from the National Center for Education Statistics. It’s way past time for Quality Counts to get on board with what really works and what is the current standard pending all states moving to the cohort graduation rate formula.(Updated 24 Jan 12 to insert correct CPI term and minor gramatical edits)