Digital Scholarship and the Tenure Process as an Indicator of Change in Universities

In this paper, the author aims to demonstrate that a practical barometer of how universities are dealing with the changes wrought by a digital, networked world can be found in the manner in which their reward and tenure processes recognise digital scholarship. The use of new technologies by academics to perform research, construct knowledge, disseminate ideas, engage students in learning and conduct a wide range of scholarly activities generates a number of issues for established reward and tenure systems, which can be seen as a representative microcosm of the issues facing universities more generally. Con este artículo el autor quiere demostrar que el reconocimiento a la producción académica digital en los procesos de recompensa y titularidad académica es un barómetro útil para saber cómo las universidades abordan los cambios introducidos por el mundo digital y en red. El uso de las nuevas tecnologías en la investigación, en la construcción de conocimiento, en la difusión de las ideas, en los procesos para que el alumnado participe en el aprendizaje y en una amplia gama de actividades académicas da lugar a una serie de problemas para los sistemas de recompensa y titularidad académica establecidos, que pueden entenderse como un microcosmos representativo de los problemas a los que deben enfrentarse las universidades desde un punto de vista más general.


Digital scholarship
The term 'digital scholarship' can be viewed as a convenient shorthand to contrast with traditional, 'analogue' forms of scholarship. However, Weller (2011) suggests that 'digital' is only one aspect of a trilogy, the convergence of which makes for significant change. It is the combination of digital content with a global network and open approaches that is significant in higher education, proposing a definition of "someone who employs digital, networked and open approaches to demonstrate specialism. " There are different interpretations as to the scope of digital scholarship; an information science perspective emphasises the curation and collection of digital resources, whereas a digital humanities perspective uses it in a broader sense to cover a range of scholarly activities afforded by new technologies. It is this more wide-ranging interpretation which is intended in this paper.

As the American Council of Learned Societies Commission on Cyberinfrastructure for the
Humanities & Social Sciences observes, there are multiple interpretations of digital scholarship: "In recent practice, 'digital scholarship' has meant several related things: 1. Building a digital collection of information for further study and analysis 2. Creating appropriate tools for collection-building 3. Creating appropriate tools for the analysis and study of collections 4. Using digital collections and analytical tools to generate new intellectual products 5. Creating authoring tools for these new intellectual products, either in traditional form or in digital form" (http://cnx.org/content/m14163/latest/)

La producción académica digital y el proceso de obtención de la titularidad académica como indicador del cambio en las universidades Resumen
Con este artículo el autor quiere demostrar que el reconocimiento a la producción académica digital en los procesos de recompensa y titularidad académica es un barómetro útil para saber cómo las universidades abordan los cambios introducidos por el mundo digital y en red. El uso de las nuevas tecnologías en la investigación, en la construcción de conocimiento, en la difusión de las ideas, en los procesos para que el alumnado participe en el aprendizaje y en una amplia gama de actividades académicas da lugar a una serie de problemas para los sistemas de recompensa y titularidad académica establecidos, que pueden entenderse como un microcosmos representativo de los problemas a los que deben enfrentarse las universidades desde un punto de vista más general.
Perhaps more fruitful is to consider an example of a particular technology-based approach in order to demonstrate the issues that digital scholarship raises. Blogging is one of the more wellestablished approaches that would be included under the banner of digital scholarship, and so acts as a microstudy of all the issues in digital scholarship, although almost any of the new Internet technologies would suffice. Firstly, it has the digital, networked and open approach central to its use -these are not attributes that have been grafted onto it. So we see bloggers linking to each other, operating open comments, using 'open' services such as YouTube and Flickr to embed content to make their posts multi-media. Moreover, such services are democratic and easy to set up.
Blogs are also the epitome of the type of technology that can lead to rapid innovation. They can be free to set up, are easy to use and, because they are at the user's control, they represent a liberated form for expression. There is no word limit or publication schedule for a blog; the same blog may mix posts about politics, detailed subject analysis, sport and personal life. Blogs can remain unread or have thousands of subscribers.
It is this freedom of expression and open approach that is both their appeal and their problem for scholarship. The questions one might ask of blogs in relation to academic practice are true of all digital scholarship: 1. Do they represent 'proper scholarship' (however that might be defined)?
2. Are they central or peripheral to practice?
3. Are they applicable to all domains? 4. Are they more applicable for some scholarly functions than others e.g., teaching? 5. How is quality recognised? 6. Do they complement or replace existing channels? 7. Should they be rewarded through official routes such as tenure?
8. Should bloggers use institutional systems or separate out their blogging and formal identities? 9. What is their impact in academic communities?
It is the consideration of such issues that is at the heart of the dilemma facing many universities when they seek to engage with the digital culture. In the next section, the existing reward and tenure process will be examined, and then how this is brought into conflict with digital scholarship because of the types of issues listed above. Lastly, some of the approaches universities are adopting to deal with this are examined.

The tenure process
Promotion and tenure is usually judged on a combination of three factors: research, teaching and service or management. Some universities expand on these to include factors such as contribution to society and academic esteem, but these three represent the main categories. These are supposedly weighted equally, often with candidates required to demonstrate outstanding achievement in at least two of the three. It is often rumoured that there is an unspoken rule that research is regarded as Digital Scholarship and the Tenure Process… Martin Weller, 2012FUOC, 2012 CC CC RUSC VOL. 9 No 2 | Universitat Oberta de Catalunya | Barcelona, July 2012 | ISSN 1698-580X more significant. As Harley et al. (2010) summarise it, "advancement in research universities is often described as a 'three-legged stool, ' with a 'research' leg that is far more important. " In putting together a case for promotion, an academic then needs to provide evidence to support their case in these three areas (although not all three may be represented equally). For teaching, this is usually straightforward -a list of courses that have been taught (perhaps with student ratings).
Service can equate to work on committees or to management responsibility, but can also be a little more nebulous, like making the case for external work with a professional body for example. Research is the most difficult to accurately represent, particularly to a committee whose members are unlikely to be experts in the subject area of the individual, and thus will require explanation and clarification on the nature of that individual's contribution to the field. Across a university with many different niche subject areas, this generates a task of considerable complexity. Whereas teaching will usually conform to an understood and agreed curriculum, and service is predominantly represented by university committees, research is precisely the area of a scholar's activity where they are at their most individual and most specialised. It is the area that is thus most difficult for a general committee to assess. There is thus something of a conundrum around research in the promotion process -it is the most highly regarded of the three strands, and yet the most difficult to judge. It is this complexity in quantifying research combined with its significance that sits at the heart of many of the issues relating to digital scholarship and tenure.

The digital scholarship barriers
Before examining some of the approaches institutions have taken to recognising and rewarding digital scholarship, it is worth considering the barriers and obstacles that many perceive in its recognition.
In a comprehensive study on scholarly communication, Harley et al. (ibid.) found that the strong lock-in with the published journal article and monograph was the overriding factor in consideration for promotion, commenting "enthusiasm for the development and adoption of technology should not be conflated with the hard reality of tenure and promotion requirements in highly competitive and complex professional environments. Experiments in new genres of scholarship and dissemination are occurring in every field, but they are taking place within the context of relatively conservative value and reward systems that have the practice of peer review at their core. " The first, and fundamental, barrier is the recognition of digital scholarship as activity that is worthy of appreciation. This is distinct from concerns around how best to represent and measure it. Cheverie et al. (2009) argue that there is a strong bias towards print, or traditional, publication: "While this community talks about 'publication' , the language used implies that digital scholarship is of significantly lesser value, and word of mouth to younger colleagues discourages digital scholarship in the hiring, tenure and promotion process. " More significantly, the resistance to recognising digital scholarship reflects a more intractable problem -their benefits are often experiential in nature, so users have to engage with these technologies over a prolonged period to appreciate their value and the nature of interactions. Given Digital Scholarship and the Tenure Process… Martin Weller, 2012FUOC, 2012 CC CC RUSC VOL. 9 No 2 | Universitat Oberta de Catalunya | Barcelona, July 2012 | ISSN 1698-580X that many senior managers and professors in universities are not people who are disposed towards using these tools, then there is a lack of understanding about them at the level which is required to implement significant change in the institution. The membership of promotion committees is most likely to be drawn from senior academics, who have largely been successful with the traditional model of scholarship. Although these academics will have a wealth of experience, they come from a background that may have a limited understanding of the new forms of scholarly practice that utilise different media and technologies.
But, there does seem to be a move in many universities to recognise digital scholarship to some extent. This starts with the reasonably uncontroversial recognition that online journals have a similar standing to print ones, particularly when many major publishers are converting many existing titles to online only. Schonfield and Housewright (2010) report that there is a general move to online journals with most academics now content to see this shift happen, away from print.
In the arts, there has been a tradition of recognising a portfolio of work when considering promotion, and this has inevitably led to the inclusion of digital artefacts. In the sciences, other components have been recognised prior to more recent developments, including software and data.
A willingness to recognise new types of output and activity brings into focus the next significant barrier, which is how to measure or recognise quality in these widely varied formats. In order to overcome the problem highlighted above of dealing with complexity in research, evaluators have relied upon metrics such as the impact factors of journals. The peer-review process that leads to publication combined with a journal's impact factor acts as a quality filter, thus removing the necessity for the promotion committees to assess the quality of the outputs themselves. Journals have quality rankings, and therefore publication in any journal of sufficient standing is an indication of quality. As Waters (2000) puts it, "to a considerable degree people in departments stopped assessing for themselves the value of a candidate as a scholar and started waiting for the presses to decide. " Peer review is at the core of this practice and is seen as fundamental. Harley et al. stress that "The degree to which peer review, despite its perceived shortcomings, is considered to be an important filter of academic quality, cannot be overstated. " This highlights the problem with recognising new types of output and activity. The power of many of the new forms of communication lies in the democratisation of the publishing process. They have removed the filter that the tenure process has come to rely on so heavily. Without this filter in place, promotion committees are back in the position of having to find a means of assessing the quality of an individual's research activity in a field they know little about. This is now confounded, as it may be in a format they know little about too.

Recognising digital scholarship
Many universities have begun to acknowledge both a need to recognise digital scholarship, and also been implemented, some more radical than others. The approaches are summarised below and then examined in more detail: • Recreating the existing model Publishers charge between $500 and $3,000 for this model and, as Waltham (2009) reports, take-up has been limited, with 73% of publishers reporting 5% or less adoption of this model. This is hardly surprising, and highlights one of the problems with attempting to recreate current practice. Simply recreating the existing model, however, often fails to adequately address many of the issues raised at the start of this paper.

Digital equivalents
An improvement on this is to seek digital equivalents for the types of evidence currently accepted in promotion cases. In making a case for excellence in one of the three main promotion criteria, the scholar is required to provide evidence. For example, a good track record in peer-review publication is seen as indicative of effective research as judged by the individual's peers, of impact upon their subject area and of effective scholarly communication. The publication record can be seen as a proxy for these scholarly activities, but is often interpreted as the artefact itself, rather than a representation.
If each of the accepted pieces of evidence are examined for what they are seen to represent, then it may be possible to find equivalents in an open, digital networked context that demonstrate the same qualities. For example, a keynote talk at a conference is often cited as a valid piece of evidence of esteem for an individual seeking promotion. The reasons are twofold: Reputation -it demonstrates that they have gained significant standing in their field to be asked regularly to give a keynote talk at a conference; Impact -if they are giving the keynote, then everyone at the conference hears it, and they can therefore claim a significant impact in their subject area.
The important element, then, is not the keynote itself, but what it signifies. What might a digital equivalent of this be, which meets the two criteria above? For example, if someone gives a talk and converts this to a slidecast of that presentation (a slideshow with synchronised audio), a certain number of views might equate to impact, often with numbers greater than those present at a live Digital Scholarship and the Tenure Process… performance. And if the presentation is retweeted, linked to, embedded, and shared in different means, then this might give an indication of reputation.
It would be overly simplistic to provide straightforward translations along the lines of 500 views + 5 embeds = 1 keynote, but by focusing on the existing criteria and considering what it is they are meant to demonstrate, it is then possible to consider online equivalents.
The New Media Department at the University of Maine have taken a similar approach in suggesting a number of "alternative recognition measures" (Blais, Ippolito & Smith, 2007): • Invited / edited publications -if an individual is invited to publish in an online journal, then that is an indication of reputation.
• Live conferences -they suggest raising the profile of the conference (both face-to-face and virtual) to a par with peer-review publication, particularly in fast moving subjects.
• Citations -using Google and databases to find a better measure of citations and impact • Download / visitor counts -downloads of articles or visits to an academic site can be seen as equivalent to citations.
• Impact in online discussions -forums, discussion lists and blogs are "the proving grounds of new media discourse" with significant impact and a high degree of scrutiny and peer evaluation.
• Impact in the real world -this might be in the form of newspaper references, but they also argue that Google search returns can be a measure of real-world impact.
• Net-native recognition metrics -online communities can have their own measures of value, and these represent a more appropriate measure than one imposed upon the contributor from outside.
• Reference letters -they suggest reference letters which may counteract some of the difficulty with traditional recognition systems.
The faculty of the Humanities at the University of Nebraska-Lincoln have similarly developed a set of specific equivalents for recognition, including links to the scholar's research, peer review of digital research sites and technical innovation (http://cdrh.unl.edu/articles/promotion_and_tenure.php).

Digital scholarship guidelines
An approach being adopted by a number of universities is to produce general guidelines which set out broad criteria for assessing the quality of scholarly activity. These can include a catch-all term to accommodate new forms of outputs. For example, the Open University promotion guidelines state that "other appropriate outputs from scholarship can be taken into account including a demonstrable • Delineate and communicate responsibilities. When candidates wish to have work with digital media considered, then the expectations and responsibilities connected with such work and the recognition given to it should be clearly delineated and communicated to them at the point of employment.
• Engage qualified reviewers. Faculty members who work with digital media should have their work evaluated by persons knowledgeable about the use of these media in the candidate's field. At times this may be possible only by engaging qualified reviewers from other institutions.
• Review work in the medium in which it was produced. Since scholarly work is sometimes designed for presentation in a specific medium, evaluative bodies should review faculty members' work in the medium in which it was produced. For example, web-based projects should be viewed online, not in printed form.
• Seek interdisciplinary advice. If faculty members have used technology to collaborate with colleagues from other disciplines on the same campus or on different campuses, departments and institutions should seek the assistance of experts in those other disciplines to assess and evaluate such interdisciplinary work.
• Stay informed about accessibility issues. Search, reappointment, promotion, and tenure committees have a responsibility to comply with federal regulations and to become and remain informed of technological innovations that permit persons with disabilities to conduct research and carry out other professional responsibilities effectively.
Some of these will seem straightforward, like reviewing work in the medium in which it was produced for example, but even such a small step may come up against opposition when there is a strictly regulated promotion process which has been designed to suit the needs of print outputs.

Metrics
One approach to overcoming, or at least easing, the complexity of judging individual cases is the use of metrics or statistical calculations to measure impact or influence. This has been an area of increasing interest even with traditional publications. This measure of impact is often represented by a statistical measure such as the 'h-index' , which is based upon bibliometric calculations of citations using a specific set of publisher databases. This measure seeks to identify references to one publication within another giving "an estimate of the importance, significance, and broad impact of a scientist's cumulative research contributions" (Hirsch, 2005). Promising though this may sound, it is a system that can be cheated or gamed (Falagas & Alexiou, 2008), for instance by authors referencing previous papers or between groups, and so a continual cycle of detecting such behaviours and then eliminating them is entered into, rather akin to the battle fought between computer virus makers and anti-virus software.
There are at least three further degrees of separation from this walled garden approach to citations. The first is to use data outside of a proprietary database as a measure of an article's impact.
This 'webometrics' approach was identified early on as offering potential to get richer information Digital Scholarship and the Tenure Process… about the use of an article, by analysing the links to an article, downloads from a server and citations across the web (e.g., Marek & Valauskas, 2002). Cronin et al. (1998) argue that this data could "give substance to modes of influence which have historically been backgrounded in narratives of science. " The next step is to broaden this webometrics approach to include the more social, Web 2.0 tools.
This covers references to articles in social networks such as Twitter, blogs, social bookmarking tools such as CiteULike and recommendation tools such as Digg (Patterson, 2009). This recognises that a good deal of academic discourse now takes place outside of the formal journal and there is a wealth of data that can add to the overall representation of an article's influence.
The ease of participation, which is a key characteristic of these tools, also makes them even more subject to potential gaming. As Priem and Hemminger (2010) report, there are services which can attempt to increase the references from services such as Digg to a site (or article) for a fee. But they are reasonably optimistic that gaming can be controlled, proposing that "one particular virtue of an approach examining multiple social media ecosystems is that data from different sources could be cross-calibrated, exposing suspicious patterns invisible in single source. " A more radical move away from the citation work that has been conducted so far is to extend metrics to outputs beyond the academic article. A digital scholar is likely to have a distributed online identity, all of which can be seen to represent factors such as reputation, impact, influence and productivity. Establishing a digital scholar footprint across these services is problematic because people will use different tools, so the standard unit of the scholarly article is lacking. Nevertheless, a representation of scholarly activity could be established by analysing data from a number of sites, such as the individual's blog, Twitter, Slideshare and YouTube accounts, and then also using the webometrics approach to analyse the references to these outputs from elsewhere. A number of existing tools seeks to perform this function for blogs. For example, PostRank tracks the conversation around blog posts, including comments, Twitter links and Delicious bookmarks. These metrics are not without their problems, and achieving a robust measure is still some way off, but there is a wealth of data now available which can add to the overall case an individual makes.

Peer review
The issue of gaming is even more prevalent with metrics, and this is confounded by the mix of personal and professional outputs that are evident in many of these tools. This is likely to increase the need for the effective use of peer assessment in evaluating work. When the filter of peer-review publication is removed, or lowered in significance, then arguably the significance of peer review in the tenure process increases. It will be necessary to determine that the output and activity is indeed scholarly (after all, one could have a popular blog on a subject which had no relevance to the academic case). It is also a response to the increased complexity of judging digital scholarship cases.
The MLA guidelines above recommend using external experts to perform this peer review for tenure committees that may be unfamiliar with both the subject matter and the format.
Others have taken this approach further, soliciting commendations from their wider online network (e.g., Becker, 2009). There is obviously an issue around objectivity with this approach, but as in the types of evidence produced would allow recognition of not just outputs, but also the type of network behaviour that is crucial to effective digital scholarship. Smith Rumsey (2010) suggests that "perhaps there should be different units of micro-credit depending on the type of contribution, from curating content to sustaining the social network to editing and managing the entire communication enterprise of a collaborative scholarly blogging operation. "

Alternative methods
All of the approaches above can be viewed as modifications of the existing practices, which have largely been determined by the practicalities necessitated by a print medium. Many of the attempts to gain recognition for digital scholarship seem to be focused around making it behave like traditional scholarship. For example, webometric data for journal article analysis still foregrounds the peerreviewed article as the main form of evidence.
Bending new technology to fit existing practice is a common reaction, partly because we are unaware of its potential. Stephen Heppell (2001) declares that "we continually make the error of subjugating technology to our present practice rather than allowing it to free us from the tyranny of past mistakes. " Arguably, this is the case with current methods for recognising digital scholarship.
Promotion committees can play a significant role in this, not only by recognising new forms of scholarship, but also by positively encouraging them, either through guidelines or specific projects.
For example, a committee might seek to develop the sort of Web 2.0 metrics mentioned above or to encourage alternatives to the peer-review model. In analysing the peer-review process, Fitzpatrick (2010) makes a strong case that we need to move beyond merely seeking equivalence measures: "What I am absolutely not arguing is that we need to ensure that peer-reviewed journals online are considered of equivalent value to peer-reviewed journals in print; in fact, I believe that such an equation is instead part of the problem I am addressing. Imposing traditional methods of peer review on digital publishing might help a transition to digital publishing in the short term, enabling more traditionally minded scholars to see electronic and print scholarship as equivalent in value; but it will hobble us in the long term, as we employ outdated methods in a public space that operates under radically different systems of authorization. " Digital Scholarship and the Tenure Process…

Conclusion
Recognising and rewarding digital scholarship has a significance beyond the promotion of individuals.
For universities, as they seek to manage change to a digital, networked society, it acts as a strong indicator and vehicle for change.
There are two main reasons for prioritising the recognition of digital scholarship. The first is the message it sends to individuals within the university. Because they operate in an open, digital, networked manner, digital scholars are often well known in their institution (for example, many of their colleagues will read their blogs). If a well-known digital scholar struggles to get their work recognised, then it sends a message to the rest of the university that this is not the type of activity that is likely to be rewarded, with a subsequent decline in its uptake. The reverse happens if that digital scholar is rewarded; it sends the positive message that academics should engage in this type of activity.
The second reason for recognising digital scholarship is to encourage institutional innovation.
For example, universities are beginning to explore the use of Facebook to support students, or the use of blogs to disseminate research findings to the public, or new models of course development based on third-party content and crowdsourcing. There are very real benefits to the institution from these approaches, such as reaching new audiences, increasing the university profile without advertising, increasing student retention through improved peer support, lowering the costs of course production, developing new research methodology, etc. But it is difficult to realise any of these institutional approaches to new media if the university does not have a solid base of digital scholarship experience to draw upon. Having a range of digital scholarship experience amongst the faculty will be the key resource in realising the change required for many universities, and an appropriate reward and tenure process acts as a means of facilitating and encouraging this. This is not to underestimate the complexity of the task however. The already difficult task of assessing research and scholarly activity in highly specialised fields is only going to be made more difficult by introducing digital scholarship. Previously, there has been an agreed set of evidence that could be seen as acting as a proxy for excellence in research. Not only does this list need to be expanded to include digital scholarship outputs, but it may be that no such definitive list can be provided anymore.
There are a number of ways in which promotion committees can begin to address digital scholarship. What they may be leading to is a more portfolio-based approach, perhaps more akin to that found in the arts. Anderson (2009) suggests that the sciences have an advantage in recognising digital scholarship because they are more ready to adopt new technology, but it may be that the arts, with their more individual assessment models, are well disposed towards incorporating different forms of output. Such a portfolio-based approach is likely to draw on a range of tools and pieces of evidence. These may include a range of digital outputs, metrics demonstrating impact, commendations from the community and recognised experts, and an overarching narrative making the case for the work as a whole.
It is worth emphasising that monetary reward and promotion are not the sole, or even main, driver for most scholarly activity. The reasons why scholars engage in research, disseminate their Digital Scholarship and the Tenure Process… Martin Weller, 2012FUOC, 2012 CC CC RUSC VOL. 9 No 2 | Universitat Oberta de Catalunya | Barcelona, July 2012 | ISSN 1698-580X findings and teach on courses are varied, but are primarily driven by intellectual curiosity. It is not, therefore, the suggestion of this paper that digital scholars should pursue any of the digital, networked and open approaches because they can lead to tenure. Rather, the purpose is to argue that if these approaches are achieving scholarly functions via a different means, that they should be recognised as such, and the tenure process acts as something of a proxy for this recognition. To ignore the context in which scholars operate within their institutions would be to disadvantage new practices compared with established ones.