Evaluating the value and impact of university research


Evaluating the value and impact of university research

October 10, 2016

In early September, the UA-led OSIRIS-REx mission launched from Cape Canaveral. Years in the making, and with a budget of over $800 million, the project is a major point of pride for the University of Arizona. For one thing, attracting a grant of this size is a major accomplishment by the University, as well as the local and state communities, which benefit from high-paying jobs and other opportunities that come with the grant. However, the ultimate outcomes of the mission – whether it performs as expected, what we learn from it, and what that new knowledge enables – remain in question. 

OSIRIS-REx launches atop an Atlas V rocket on September 8, 2016

Therein lies one of the major complications for university research, particularly in an era of major pressures on federal, state, and other research funding: Calls for accountability, more oversight, and other perspectives skeptical of the benefits of big science, or even (relatively) less funded research in fields like the humanities and social sciences, are common, and explaining why this work matters and why we should support it is crucial. For instance, OSIRIS-REx is a basic science mission, meaning that there are no specific application outcomes or benefits defined from the outset, as might be the case with a funded clinical trial for a new pharmaceutical treatment. Of course, as I wrote in December, basic research contributes to applied innovations in often unpredicted and wonderful ways, and the same may be the case with OSIRIS-REx. However, even with this mission we often focus on the easily measured points of success and the impact they imply for our local and state community: the size of the grant that I mentioned above, the number of partners brought together, and other quantifiable means of evaluating both scope and potential impact.

But these are not the things that excite us the most about OSIRIS-REx and other projects like it. These easily measured outcomes are important (in part) because of the potential that they represent. We are excited about the OSIRIS-REx mission becauseof the uncertainty aroundits scientific, technical, and other less obvious outcomes, and the scale and ambition of the mission mean that those outcomes have the potential to be grand in some unanticipated and serendipitous ways. In other words, easily quantifiable indicators like the amount of funding help to point to the less obvious outcomes that are nonetheless incredibly important. We need to be able to share and encourage these less obvious outcomes and benefits for OSIRIS-REx and for other projects because research, at its core, is a form of exploration and a means to discovery. The most transformational benefits can come from the fact that research is often conducted without a specific end in mind other than answering a set of research questions that drive us to discovery.

Like many other areas and activities of the academy, assessing research success presents significant challenges. Travis T. York, Charles Gibson, and Susan Rankin point out in an academic paper from Practical Assessment, Research & Evaluation how this works. While their argument focuses largely on student success, the discussion raises important questions for research evaluation as well, particularly our tendency to rely on what statisticians often call proxy measurements. A critically important tool, proxy measurements are indicators that somehow correlate to but are not equivalent to whatever phenomenon needs to be described. For instance, they are often strong indicators of economic impact, one of the UA’s key contributions to our community and an important means of justifying the scope and expense of university research for the general public and community leaders. However, proxy measurements are not sufficient for understanding the impact of university research locally or globally because they miss what Michael Spence called “behind the scenes” impact. Thus, there is a real need to both be able to describe and advocate for the longer term and other less obvious benefits of university research and to encourage work that is geared towards those ends.

In this post, I will discuss the concept of research universities as anchor institutions, which a recent report from the Lincoln Project defined as “hubs of research and innovation [and] cultural institutions,” that “tend to remain in their location and support their immediate community,” which helps to “drive prosperity” (pg. 6). This concept provides one way to better share the impact of university research more comprehensively, which I will illustrate in this post. In a follow-up post, I will discuss how universities can encourage such work through research support and evaluation. With these discussions, I hope to illuminate why we need to be able to advocate more clearly for the benefits of research, and how we can talk with elected officials and other leaders about research impact and why it matters. 

Proxy measurements, necessary but not sufficient

What are we missing if we rely only on proxy measurements? One answer, as I wrote in December, is that many times the longer term and hard to measure aspects of research lead to impacts in our daily lives that we take for granted or for which we do not know the origins. A few examples:

  • Arizona’s economy depends in part on the state agricultural and ranching industries. However, in many cases the impact of these industries – from Yuma’s national importance in winter vegetables to the success of the Shamrock Farms dairy company – would be lessened without the work of university faculty and other researchers in the UA College of Agriculture and Life Sciences.  
  • Tucson’s largest private employer, Raytheon Missile Systems, often hires UA grads, many from the College of Engineering. As I discussed in April, the links between research and teaching at the UA are crucial to our impact, and without the College of Engineering’s strong research portfolio and record of innovation, its graduates may not be as ready for the workforce and Raytheon may not be as successful in Southern Arizona or as committed to our community. 
  • The University of Arizona’s Steward Observatory got its start in 1918 from a 1916 bequest from Lavinia Steward.  The first Steward Observatory Director, Andrew Ellicott Douglass, then went on to start the field of dendrochronology at the University as an outgrowth of his study of the Sun. The University of Arizona proved to be an attraction for other observatories with Kitt Peak being chosen as the site for the first national observatory, and ultimately other groups such as the Planetary Sciences Institute choosing to locate in Tucson.  Because astronomers need both precision and innovative optics, the UA’s world-premier programs in Optical Sciences were founded at the university. Optical Science has flourished, beginning as the University of Arizona Optical Sciences Center in 1964 and becoming the College of Optical Sciences in 2005. This expansion has stimulated the growth of what has become a multimillion dollar optics business that has led to the Tucson area being dubbed “Optics Valley.”
  • Many educational services and resources hosted by the UA are open to the public, like the Arizona State Museum (ASM). These services provide educational opportunities for everyone in our home region, and they are an integral part of creating cultural memory and the shared history of Arizona and the greater U.S. Southwest. Through this cultural history, museums and universities also help bring businesses and corporations to Tucson because companies that are looking to relocate often want culturally rich communities for their employees. For example, Caterpillar Inc. recently announced the company was moving to Tucson; a move that is expected to generate over $600 million for our local economy, and Tom Bluth, Vice President of the company, named “an attractive quality of life for both families and young graduates” as one of the reasons for the move.

These four examples help illustrate why the idea of the anchor institution is useful in thinking about the impact of the UA and other land-grant universities in a geographically defined community/region (Southern Arizona, for instance). The concept is also helpful in a more abstract way when we consider the impact of university research on the development of knowledge and capability. As places that have been unique in their ability to invest in long-term research without assured outcomes, universities have anchored the advancement of human understanding, technology, art, and many other fields in ways that other organizations cannot. 

What is an anchor institution?

So how to understand the impact of an anchor institution? If proxy measurements (metrics, etc.) are necessary but insufficient, what do we do? How do we evaluate success or impact when it’s hard to see? 

On one hand, an anchor institution on the scale of a major research university can be a unique resource to create opportunities in the community of which it is a part. The Arizona State Museum is again a good illustration. Built on a cornerstone of anthropological research beginning more than a century ago and with scholars who are leaders in the field of Southwest anthropology, research at the Arizona State Museum (ASM) cuts across numerous fields and involves many disciplines. When we talk about the ASM, we can easily point to metrics as an indication of the museum’s history and the scope of its collections, and the metrics are impressive: the facility houses more than 3 million objects, including more than 300,000 catalogued archaeological artifacts, 40,000 ethnographic artifacts, 500,000 photographic negatives and original prints, 90,000 volumes of rare and hard-to-find titles, 6,000 maps, 1,500 linear feet of archival documents, and more than 1,000 sound recordings. These numbers reflect the volume of the ASM’s collection, but on their own they do not convey the cultural importance and shared history that the Arizona State Museum helps to maintain.

Because it is an archive of the work of anthropologists and archaeologists working in the U.S. Southwest for over a century, the Arizona State Museum is obviously significant within those two fields. But, if we are trying to incorporate metrics in a broader view of research impact, why do the millions of objects housed at the ASM matter for the rest of the University, Southern Arizona, and the other communities we serve here in our home state and around the country? What does the museum anchor?

One answer comes with a shift in understanding of what an archive is. Rather than a repository of the past, we should also think of museums as places of ongoing learning and cultural vitality. Things go into archives, but they can also come out of them to be studied and shed new light on the world around us. For instance, a recent story on UA News shared ways that curators at many UA museums (the UA Museum of Art, Center for Creative Photography, and others, in addition to the ASM) are collaborating with UA instructors to teach students about the inner workings of a museum, as well as their work in the community. In this example, the ASM and other UA museums help to anchor the professional community of curators and students at the UA, giving students the opportunity to think like an archivist or curator. The museum’s archives are also a site of continual learning about the history and cultures of Southern Arizona and the U.S. Southwest.

For instance, many Tucsonans know that the region has been continually inhabited for at least 4,000 years, and learning more about the people who inhabited this place before us can change the way we see it and the way we live in it. A good illustration of why this matters comes from the press coverage of the recent good news that Tucson has been selected as a UNESCO World City of Gastronomy. The designation gathered a great deal of attention nationally, including an article in The New York Times, which began by mentioning Tucson’s climate in comparison to food deserts, “those urban neighborhoods where finding healthful food is nearly impossible.” While the article goes on to praise the local food heritage, it is worth noting, as The Arizona Daily Star did, that the Times’ surprise is indicative of broader ignorance about things beyond cacti and heat that make our home distinctive. In this context, the work of the Arizona State Museum, and other sources of knowledge about the history and vitality of the Southwest (though it may look different than other parts of the country), is so important. By seeing its collections as opportunities for learning rather than repositories of the past, we can realize the full potential of the Arizona State Museum, particularly as it brings new opportunities to the University and our community like the “Pieces of the Puzzle” travelling exhibition presented by Archaeology Southwest, which provides new insights into one of the Southwest’s earliest cultures, the Hohokam. By presenting the long cultural history of our region, this and other continuing research conducted or shared by the ASM can help contextualize our current experience (and knowledge) of the U.S. Southwest, reminding Arizonans and others of the significance of the long habitation of Tucson and surrounding areas. It can be a means of shifting the common conception of the Southwest (and other arid parts of the world) as barren wastelands where the richness of gastronomic and other cultural heritage surprises outside observers.

This approach characterizes the ASM and other UA museums as something other than a passive resource, making them agents of continued cultural memory and expression. In the case of the Arizona State Museum, this current impact is due to its history. However, with budgetary constraints at the federal, state, and local level – including within universities – reducing investment in both research and its use as a point of community engagement, we have to ask what potential future resources like the Arizona State Museum are not being created for lack of foresight about what they will or could be in the future.

One of the critical challenges of 21st century universities then, is to sustain this kind of impact despite the shifts in funding and other forms of support for public higher education. The form of engagement practiced by the ASM and other museums on the UA campus suggests another way of seeing and sustaining an anchor institution’s impact in its home region, one in which the resources built up over generations of research and scholarship allow the university to be a partner in the region’s ongoing cultural vitality. If the museum’s engagement with Southern Arizona is a form of service, another wonderful example of research impact at the UA helps to demonstrate how the University is not a single point anchoring the rest of the community but one among many spots of creativity and innovation that come together to shape the home that so many of us love. This example comes from the work of the Southwest Folklife Alliance (SFA), led by Executive Director and UA faculty member Maribel Alvarez. The SFA is an affiliate organization of the University of Arizona – an important and relatively rare distinction that indicates the importance the relationship holds for the University and the SFA’s unique contributions to the UA, Southern Arizona, and our entire region. Like the Arizona State Museum, the SFA demonstrates how an anchor institution can be an active agent of cultural vitality (rather than a passive resource) while also avoiding the pitfall of becoming an overbearing presence that too-directly shapes the work of others. The history of the SFA shows us why that distinction is so important and how it can emerge.

The Southwest Folklife Alliance and Tucson Meet Yourself

Like the Arizona State Museum, the SFA is built in part on historic research strengths of the UA. In the words of Dr. Alvarez, the SFA is “the most recent manifestation of a long trajectory of engagement between the UA and the vernacular culture of the region.” As a formal organization, it began as the UA Folklore Committee, inaugurated in 1943 by Dr. Frances Gillmor, an anthropologist who taught in the UA Department of English. The Committee started as an interdisciplinary effort that integrated the work of UA faculty with community members to grow the University Library’s Folklore Archives. This beginning is crucial to understand its impact; as the Committee grew through efforts of faculty members in Anthropology, English, and the UA’s Southwest Center, it also drew on developments in folklore studies, in particular a shift from an emphasis on folklore to what is now called folklife, a more holistic and participatory concept of what the SFA calls “living community traditions” where scholars understand their object of study to be continuously developing cultural communities and traditions. Important in fields including anthropology, Native American studies, folklore, cultural studies, and many others, this understanding of cultural tradition (and the study of it) changes the relationship of the UA to the local community in Southern Arizona. For instance, measuring the success of something like the SFA in terms of numbers or even detailed stories might capture the renowned nature of the institution itself, but it would fail to take into account the way in which the UA’s participation in the broader cultural community of Tucson and Southern Arizona has had a transformative effect on how we understand cultural traditions in the aforementioned fields. 

Today, the SFA’s work builds on these beginnings together in partnership with the UA so that both institutions can reach even further heights with what can be accomplished. Dr. Alvarez is the Public Folklorist at the UA’s Southwest Center, and with her leadership and the partnership of the College of Social and Behavioral Sciences, the SFA (an independent 501(c)(3) organization) became a “formal affiliate” of the UA, an important designation that only a few organizations have. This status allows Dr. Alvarez’s appointment as Public Folklorist and UA faculty member to include her role as Executive Director of the SFA and for the organization to be housed in the UA’s downtown Tucson location in the historic Roy Place Building.

This relationship formalizes years of partnership in practice shaped by many, including Dr. Jim Griffith, the first director of the Southwest Folklore Center when it was established in 1979. The early partnership was punctuated by Dr. Griffith’s insistence that his appointment at the UA include time for his work on Tucson Meet Yourself, the city and region’s annual folklife festival. This precedent helped pave the way for Dr. Alvarez’s work to transform Tucson Meet Yourself from a single event into a broader nonprofit organization that encourages, promotes, and helps to sustain the work of local heritage artists, communities, and their traditions.

Just as I mentioned in July, global partnerships and collaboration enable us to create impact and succeed despite the occasionally enormous scale of scientific inquiry. Partnerships like the one the UA has with the SFA are significant because of the way these relationships enable potential future scholarship and create lasting impact on our community that might otherwise not have occurred as individual units. Programs like the South Tucson Project, for instance, demonstrate the teaching and student research that helps create the culture of inquiry that is at the heart of the UA’s mission (and that of other universities). Equally impressive is the involvement of UA students in the Tucson Meet Yourself festival, perhaps most notably through the work of the Compost Cats, who staff Green Stations that reduce the amount of waste going from the festival to area landfills by 44%.   

The Challenge of Measuring What Might Not Have Been 

Town sign for the fictional Bedford Falls of It’s a Wonderful Life 

The impact I am trying to describe can be hard to measure precisely because it is so widespread and because oftentimes trying to see it is like measuring the impact of something’s absence. Similar to the movie, It’s a Wonderful Life, tracing the effects of one person’s life on everything he or she has influenced, either directly or indirectly, can prove nearly impossible – unless you have the aid of a guardian angel to show you the world without you in it. The work done at the SFA affects multiple fields of study even as it helps sustain local heritage communities and their traditions. Like the ASM it would be impossible to measure where the positive impact begins and ends. 

This understanding of behind the scenes impact can help better appreciate many areas of research and scholarship at the University, from those with obvious tangible benefits (like the optics and space sciences areas I mentioned earlier) to those that are sometimes further from public attention. Universities like the UA need to help ensure that the outcomes of research are shared and that its impact is well-understood and encouraged in our local, state, and national communities. These efforts can form through the kind of partnership that the UA has with the SFA, where the University leverages its assets for the benefit of the community or through better communication and outreach. For instance, the Arizona State Museum, the UA Museum of Art and the Center for Creative Photography are all part of the UA Office of Research and Discovery (ORD), which reinforces their critical role in the UA’s research enterprise. The ORD has recently expanded its community outreach efforts, particularly with campus museums, and is working to help expand the UA’s partnerships with Tucson and Southern Arizona artists by better sharing the resources and opportunities available through the University. A similar goal drives the public events like lectures and readings held at the UA Poetry Center, or the College of Social and Behavioral Sciences Downtown Lecture Series and the College of Science Lecture Series. All of these avenues of community engagement share new knowledge and are meant to encourage members of our local and state communities taking roles as active contributors to the vital work of research and discovery, as well as in the application of its insights. These examples and the others I have shared in this post remind us of the ways in which a university extends far beyond the achievements of the individual faculty member or student, instead reaching across many communities to develop even more avenues of inquiry. As I mentioned above, it is difficult to measure behind the scenes impact precisely because impact never stops at just one place – it leads to newer advancements and newer discoveries that lead to even more innovation.

The challenge for universities and the communities we serve is to stay open to the many ways the research and inquiry inform our lives. While proxy measurements are important tools, relying on them too heavily can lead to potentially negative impact if they diminish our frame of reference for why research matters, how it proceeds, how to support it, or even how we make connections that enable our range of public impact. I hope that in this post I have begun to suggest some ways in which we can see and understand the long-term and behind the scenes impact of university and community research. In the second part of this discussion I will return to this topic to explore ways that universities (as institutions) can further encourage and reward work of this kind.