Why I am uncomfortable with trying to tie specific improvements in school quality to increases in Kentucky’s economy
At the outset, I want to make it clear that I think boosting the level of education in Kentucky can reap good economic benefits for the state.
For one example, folks in Northern Kentucky tell that robotics-based manufacturers in this part of the state can’t find enough qualified workers. The situation is so serious that those manufacturers are now asking Northern Kentucky business recruiters not to attract more robotics-based companies to the region. The qualified labor pool is already stretched too thin. Jobs already are there. Workers with the education needed to perform those jobs are not.
To be sure, this is a subjective example, and I can’t add any dollar impact estimates.
Sadly, it a real challenge to find specific data that can be credibly used to tie actual, dollar amounts of economic improvements to specific amounts of better education. The Kentucky Center for Education and Workforce Statistics was recently created to develop such data based pictures, but KCEWS, as we call it, is still rather new. It has issued a few reports about the impacts of college education and economic outcomes, but platforms that fully blend data from the K to 12 education system with postsecondary education data and resulting workforce impacts is still in the future.
Given the limitations of currently available data, I am uncomfortable about trying to estimate specific dollar amounts of economic improvement related to specific amounts of educational improvement. However, limitations in the data are not stopping others from trying to connect specific dollar estimates for economic improvement to educational improvements.
The newest case in point, Education Next, an education-oriented journal published at Harvard University, released an article in its summer 2016 edition titled “It Pays to Improve School Quality” (hereafter “It Pays”).It Pays bravely explores new territory, trying to estimate the current levels of adult work force education in each state and then using that to project how further improving education levels could boost each state’s economy. It’s a worthwhile pursuit, but I am uncomfortable with the approaches the report uses to work around the obvious limitations in currently available data. For sure, It Pays’ estimate of the current educational level in Kentucky’s workforce looks very wrong, and that might point to more credibility issues for the report’s other findings.
Furthermore, while It Pays’ authors’ intention is to boost education and productivity – certainly a laudable goal – inaccurately portraying Kentucky’s current adult education level as remarkably high might do more harm than good to the process of trying to really improve the Bluegrass State’s school system.
Some of It Pays’ assumptions concerning the overall quality of education received by each state’s workforce are impossible to evaluate. Those assumptions are part of an attempt to define education levels of all the workers in each state, a hugely complex issue. A portion of each state’s adult workers are native born and educated. Others are US born but were educated in another state. Finally, foreign-born and foreign educated immigrants present still more challenges to developing an accurate picture of the overall education level of each state’s workforce. Whether each of the various approaches adopted in It Pays are really on target is difficult to argue one way or the other.
For example, in assessing the education level of foreign-educated immigrants, It Pays and a companion report I received indicate the approach somewhat arbitrarily assigns a rather low math test scores to those from Mexico but uses much higher scores, the 90th percentile score, no less, for math tests from immigrants who came from other countries, including all the other South American countries where it is possible for the very poor to walk their way to the US just as many from Mexico have done. At best, It Pays uses a highly subjective system to evaluate the education of current US workers who were educated in another country.
There are plenty of other issues I could discuss; but, what really concerns me about the report is a bottom line implication found in Figure 4 from It Pays. This figure shows how the report’s authors believe each state fits on a scatter plot of GDP growth per capita versus each state’s supposed overall eighth grade math test score performance for the state’s native born and immigrant adult working population.
I highlighted Kentucky’s position in Figure 4 with an added yellow arrow. I also added a vertical line through the state’s plot position to help show that only a small number of states, Montana, South Dakota, Texas and North Dakota, supposedly have adults that overall enjoy higher levels of education than Kentucky’s workforce possesses. I know that will raise eyebrows among those familiar with Kentucky.
Figure 4 from It Pays
Hanushek's Figure 4 from It Pays
After my Kentucky-smart readers stop chuckling, let’s examine some information that indicates It Pays has really missed the mark.
Consider It Pays’ primary test score source -- NAEP
It Pays says it primarily relies on NAEP Grade 8 Math results from 1990 to 2011 to develop its education metric.
Aside from using just one, eighth grade test result as a proxy for the entire K to 12 education program’s performance in each state, this becomes even more problematic when we examine Kentucky’s NAEP scores.
The Main NAEP Data Explorer provides tools to determine how Kentucky ranks against the other states for various administrations of NAEP Grade 8 Math. I examined results from 1990 and 2000, the first and last years when state participation was optional in NAEP. I also looked at 2003, the first year when all states participated in NAEP and the most recent year used by the It Pays analysis, 2011.I considered both simplistic rankings of the published scores (It Pays uses only the published “all student” scores) and a more defensible approach which considers the statistical sampling error present in all NAEP scores (something It Pays notably ignores).Table 1 summarizes Kentucky’s Grade 8 NAEP Math results over the years.
Table 1
Kentucky NAEP G8 Math Rank for 1990 through 2011
The data in Table 1 show there is absolutely no way that the NAEP Grade 8 Math results support the notion that Kentucky’s education system ranks in the top five among the 50 states.
If we only look at the simplistic ranking of the reported scores, shown in the two medium blue-shaded columns, Kentucky’s ranking has never been above the 50th percentile (As an aside, we have consider this data in terms of percentiles because not all states participated in the early years of state NAEP testing, creating more apples to oranges data quality issues for results from different states used in the It Pays analysis).If we look at the more valid data analysis for Kentucky which does consider sampling errors, shown in the yellow shaded portion of the table, we again see that the state’s performance is at best tied with a number of other states at somewhere around the middle to below the middle level of performance.
To summarize, even if we only consider “All Student” scores, the NAEP results cannot possibly place Kentucky’s education level anywhere near the level of the true top five states (regular readers also will know I have spent a lot of time over the years explaining that comparing states only with their NAEP “All Student” scores can be misleading, as this past blog points out. That could create another problem for the analysis approach used in It Pays).To summarize, the NAEP Grade 8 Math results, which It Pays says are the focus of its attempts to quantify education levels of the adult workforce in each state, absolutely cannot support a high ranking for Kentucky.
College degree adjustments won’t help
It Pays and a companion report I received from the lead It Pays author say the NAEP scores are adjusted for the proportion of each state’s adults that hold a college degree.
That still can’t work for Kentucky. The proportion of Kentucky’s adults who hold a college degree ranks very low compared to the rest of the nation. An October 15, 2012 article in the Wall Street Journal ranks the state as the fourth lowest for the percentage of the population that holds a Bachelor’s Degree or higher as of 2011, the same end year used by It Pays for its final year of NAEP data.
To summarize, Kentucky’s population has a notably lower than average proportion of college graduates and the state would get less than an average boost in the adult education level overall in the It Pays analysis as a result. The state’s relatively low ranking for the direct NAEP Grade 8 Math Score analysis isn’t going to be dramatically boosted by college degree adjustments.
Foreign migration unlikely to help the It Pays picture for Kentucky
It pays makes interesting assumptions about the education levels of foreign immigrants to each state, making an assumption for every country but Mexico that adult immigrants to the US on average score way up at the top in international math tests at the 90th percentile score for their former country. Aside from the dubious nature of that assertion given large numbers of refugees that have come to our country recently, Kentucky has very few foreign born residents who could dramatically alter the state’s education ranking, as this graph from Index Mundi shows:
Foreign Born Residents by State
Whether the state has an unusually high or low education level among its non-US born residents doesn’t matter much when there are so few of them to move the scores around in any event.
In closing, I have a number of other concerns about It Pays’ use of test scores, but the point for now is something seems to be very wrong with the It Pays’ analysis if the bottom line judgement about the average education of Kentucky’s workforce says the Bluegrass State is the fifth best educated in the United States. Given Kentucky’s low NAEP Math scores and a low proportion of college degree holders, that just isn’t right. Any methodology that would show such an obviously erroneous situation has to have a flaw, somewhere.
Also, folks would be well-advised not to hastily jump on the It Pays report, though that process seems to have already started.
Finally, to be clear, I do think that increasing the education level of Kentucky’s citizens can reap economic benefits. I’m just not ready to grab at specific numbers for what those benefits might actually turn out to be. And, I don’t think looking at mostly eighth grade math education scores is going to give us high accuracy answers.