Tag: Campbell’s Law

Today in collegiate dystopia: when the metric is mistaken for the goal

February 10, 2022 • 9:30 am

by GREG MAYER

[This is a long post. Although focused on Wisconsin and my campus of the University of Wisconsin in particular, I think the issues are general ones in higher education. Here’s the TLDR: Higher education administrators, responding to the demands placed upon them by government and industry, have fixated on a flawed measure of educational success–the six-year graduation rate. They will use any means necessary to increase that metric. In doing so they have mistaken cause and effect, and are devaluing the credentials which they seek so ardently to award.]

The neoliberal consensus in higher education, endorsed in America by Democrats and Republicans alike, holds that higher education is, in its essence, a monetary transaction involving students and the university. The two parties differ only over a question that has become a perennial debate in higher education circles: are students the customers of higher education, or are they the product? Democrats contend that students are customers buying job training, and that the purpose of higher education is to get the populace good jobs; Republicans view students as products to be ordered and purchased by industry, so that the role of higher education is to discern the needs of industry and to train workers to fulfill these needs.

University administrators have embraced the consensus with enthusiasm, and have striven to remold their institutions so as to further its aims. Whether their keenness arises from conviction or an instinct for self-preservation is debatable, but the effects are indisputable.

However, in seeking to please their financial masters, university administrators have fallen into a familiar trap: they have mistaken a metric for the goal.

The metric they are in thrall to is the six-year graduation rate. This metric is used by both government and private agencies as the single best measure by which to evaluate an institution of higher education. Administrators thus have their eyes set firmly on raising their institution’s six-year graduation rate, and anything that would interfere with that–say, graduation requirements— are barriers to “student success” that must be overcome.

A recent report by the National Student Clearinghouse on six-year graduation rates for the incoming class of 2014 has found that Wisconsin is doing well by this measure. (That this group is a card-carrying member of the neoliberal consensus is announced on their homepage: they “Serve the K-20 to Workforce Continuum”.)  [Click on all screenshots to access the links.]My campus, the University of Wisconsin-Parkside, however, had the lowest six-year graduation rate of any campus in the University of Wisconsin system.Asked for comment on this, the administration said that graduation rates had improved a lot (they had), that raising them further was the university’s top priority (seems as if it is), and that student “demographics” explained it (I don’t know about that).

What’s the matter with six-year graduation rates? Traditionally, in the jargon of higher education, graduation rate is regarded as an “input measure”, rather than an “output measure”; that is, it measures the selectivity of the admissions process, and not the “value added” by the education within the institution. To use an analogy, if you bring your Ferrari to a car wash, it’s going to come out looking pretty good, no matter how indifferent the quality of the wash.

There are other problems with the six-year graduation rate–as usually defined, it misses incoming transfer students who successfully graduate, and students who transfer out and finish elsewhere– but these bugs could be remedied without addressing the fundamental drawback.

What leads to high six-year graduation rates? High graduation rates are characteristic of schools such as the Ivies, state flagship schools, and the service academies. You thus might be tempted to believe that since “good” schools have high graduation rates, then raising the graduation rate will make a school better. But that’s mistaking a correlation for a cause. High graduation rates are the result of 1. selective admissions, and 2. large input of resources. But public branch campuses (like mine) and most private colleges cannot be very selective in their admissions, nor can they command the necessary resources.

Why does UW-Parkside have the lowest six-year graduation rate in the UW System? I don’t know why it’s that way now (i.e., for the entering class of 2014). But I do have a strong idea about why it had the lowest graduation rate for the entering class of 2006, because eight years ago there was another kerfluffle about the campus’s low graduation rate. At that time, a local reporter asked me about why that was the case, and I was able to gather some data on graduation rates across the 13 UW System campuses, and show that the major reason was lack of academic preparation of incoming students, as measured by ACT scores. (The ACT, unlike the SAT, attempts to measure achievement rather than aptitude.)

This figure tells the story.

Across the 13 campuses, the average ACT score of the incoming class is a strong predictor of class graduation rate. (For the statistically adept, R² is 78%.) Parkside’s graduation rate is pretty much exactly what you would expect given the ACT scores of its incoming freshmen. The flagship campus (Madison) and Parkside have the highest and lowest graduation rates and ACT scores, respectively, but both fall within the expected relationship. (The only significant departure from the relationship is UW-Superior’s lower than expected graduation rate.) It was Parkside’s mission of providing access to the less well prepared that led to lower graduation rates.

The “demographic” explanation of Parkside’s low graduation rate invokes, among other factors, the presence of a high proportion of what are termed “under-represented minorities”. Parkside did, and does, have the highest proportion of such students in the UW System, but including this variable did not significantly improve the relationship between ACT score and graduation rate.

How can six-year graduation rates be raised? Parkside could perhaps raise its graduation rate by admitting freshmen with higher ACT scores. So why don’t we do this? Because part of our mission is to provide access and opportunity to students who are not as well prepared, due to a variety of educational, social, familial, and personal factors. Some of these students succeed splendidly, and we don’t want to deny them the opportunity to become successful. The City University of New York (and its institutional precursors) provided the opportunity for advancement for the largely immigrant communities of New York City through much of the 20th century, producing some of the great scientists, scholars, artists, and authors of that century. I don’t claim we are incubating a cultural renaissance here, but we hope we are helping another generation and population of students to succeed as individuals and contribute to society.

The flip side of success is failure. You can’t give opportunity to almost everyone, and not have some not make it. You could probably raise the graduation rate some with a massive influx of resources directed toward the least prepared students, but that’s not going to happen. So if you want to maintain opportunity, but also have everyone graduate, you need to redefine what it takes to graduate.

At a meeting on campus a few years ago, I learned that few incoming students needed remedial math classes anymore; formerly a considerable fraction had needed them. Impressed, I asked what accounted for the substantial improvement in mathematics preparation of our incoming students. I was distressed to learn that there was no improvement; “remedial” had been redefined downward. As Freddie deBoer drily wrote about increases in high school graduation rates, “There is no underlying trend in educational data that would suggest that this vast improvement is underwritten by actual student learning gains.”

So what’s the problem? By treating graduation rate as the goal, higher education administrators, government, and industry are mistaking the metric for the goal. The metric can be gamed. Thus, anything that impedes “student success” is a barrier that must be eliminated. They embrace the neoliberal consensus, but eventually their “products” will not be valued by industry. Again as Freddie deBoer put it, “Sooner or Later, Ability Rules“.

Higher education administrators are a fairly nomad class– they move on to other positions at other institutions, and thus do not always reap what they have sown. For government and industry, the attachment to the graduation metric is shortsighted– they are actually driving down the meaning of the degrees and credentials they demand that higher education give out.

What is to be done? Even within the neoliberal ideology there are contradictions that must be resolved. The goal of acquiring skills, whether for the benefit of the student or some eventual employer, is multidimensional, and probably cannot be encapsulated in a few metrics, let alone one. Higher education administrators must learn learn how to investigate causal relations among complex social phenomena, thus being able to distinguish causes, effects, and correlates. (It can be difficult!) Just as high graduation rates are an effect, in part, of selecting already successful students, it is almost certainly the case that good, successful, students take 15 or more credits a semester, and not, as a popular campaign in higher education insists, that taking 15 credits a semester will make a student good and successful. (One manifestation of this campaign– signs sternly warning students that “Time is money”– reminds me, in both a sad and funny way, of Rowdy Roddy Piper’s 1988 classic, “They Live“.) Students are not successful because they graduate; they graduate because they are successful.

I think we need to get rid of the neoliberal consensus. My own view is that the object of the educational system, taken as a whole, is to produce responsible citizens, and that the role of institutions of higher education specifically is the increase and diffusion of knowledge.

At the very time I was contacted eight years ago by the local reporter concerning the 2006 entering class graduation rates, I happened to be working with an excellent  student who was graduating that year– one of my top students, with sterling grades, a phenomenal hard worker, all while raising a child alone. Due to varied and (for our concerns here) unimportant circumstances, at one point in her academic career she moved elsewhere, took a few courses at another school there, and then came back; it took her seven years to complete her degree. By the popular neoliberal metrics of retention and graduation within six years at the same school, she was a double failure. But the reality is almost infinitely far from this: she is, in fact, an outstanding success, and a fine exemplar of my university doing its very best.

Higher education in general, and UW-Parkside in particular, should not– must not– give up on that part of its mission which provides access and opportunity to the less well prepared. But unless our understanding of that mission moves beyond the terms set out by the neoliberal consensus, we are condemned to contradiction, and failure to achieve the goals of that mission.

Today in collegiate dystopia: “Has College Gotten Too Easy?”

July 25, 2019 • 9:00 am

by Greg Mayer

The title of this second post in what I hope will be a continuing series comes from the title of a new article in The Atlantic by Joe Pinsker. In the article he discusses an unpublished paper by Jeffrey Denning and colleagues at Brigham Young University. It highlights one of the chief ills of American higher education: that the goal of the system is not education. As noted in my first post in this series, the neoliberal consensus in higher education holds that the goal of higher education is maximization of monetary value. For businesses and legislators, this means colleges should train workers that industry needs in order to maximize profits; for labor advocates, it means training workers for jobs that pay well.

Though it might seem that achievement of this goal can be measured in dollars earned (by businesses or workers), such economic data can be difficult to gather, and suffer from the fact that it can take many years for the payoff of an investment to be evident. In lieu of this, some measure of success in reaching this goal must be agreed upon, and businessmen, legislators, bureaucrats, and educational think tanks, with the ready assent of college administrators, have settled upon graduation rate as the metric by which to measure the results of higher education.

It is to these rates, and how and why they are changing, that Denning et al. address themselves. Noting that graduation rates have gone up recently, but that there are no demographic or academic reasons to readily explain this, they consider that maybe it’s just easier to graduate. Money quote from Pinsker:

If grades are improving but there’s no reason to think that students have become better students, an interesting possibility is raised: The unassuming, academic way Denning puts it in a recent paper (co-authored with his BYU colleague Eric Eide and Merrill Warnick, an incoming Stanford doctoral student) is that “standards for degree receipt” may have changed. A less measured way of saying what that implies: College may have gotten easier… altering what’s necessary to get a degree is “the lowest cost way to increase graduation rates.”

Now, on the face of it, a high graduation rate would seem to be a good thing, and it is; but when you make a metric the goal, then the metric can be gamed, which is what Denning et al. suggest is being done by colleges. This is a well known phenomenon, nicely summarized by Abhishek Chakraborty:

It has been established that when you measure effectiveness solely based on quantitative indicators, people involved have a high incentive to demonstrate less ethical behaviour, and most likely less effective results as well. This is called Campbell’s Law. [emphasis added]

Donald Campbell was an academic psychologist, who expressed his law this way:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. [paywalled]

Frederick Hess gives a nice example:

The first time I heard of Campbell’s Law was in a college class in public policy. The professor asked, “Can data ever cause problems? Can it ever hurt?” It seemed like a trick question. Pretty much in unison, the class uncertainly mumbled a version of, “I don’t think so.”

The professor then asked, “What if a police department decides to evaluate officers based on the number of traffic tickets they write? Could anything go wrong?” Someone observed that cops would try to write lots of tickets—including for people who might not deserve them.

The professor asked, “Okay, so what if they flip it? What if they reward cops who issue fewer tickets?” Well, duh. Police might turn a blind eye to real problems.

The instructor smiled and said, “See, you can think of lots of ways where data might hurt.”

Another term for essentially the same phenomenon is Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”

It seems to me that much of what passes for “reform” in higher education is an instantiation of Campbell’s and Goodhart’s Laws. Under pressure to show results responsive to the neoliberal consensus, measures, which might even have merit, are turned into the purpose of the system, and almost any method may be resorted to to achieve it. It is ironic that use of this measure may pervert even the neoliberal goal: if graduates are required to learn less and have fewer skills, will they really be able to maximize business’s profits, or their own salaries?