New article: Users, technologies, organisations – towards a cultural history of Web archiving

This article is now published, in Web 25. Histories from 25 Years of the World Wide Web, edited by Niels Brügger. It is published by Peter Lang, in hardback, paperback and ebook formats. A postprint version is available to download here.

From the Introduction:

If 2015 marked the elapse of 25 years since the birth of the web, 2016 marked the 20th anniversary of web archiving: of systematic attempts to preserve web content and make it accessible to scholars and the public. As such, the time is ripe to make an initial assessment of the history of the movement, and the patterns into which it has already fallen. This chapter represents the first attempt to document the subject at length. It concentrates on what might be termed the cultural history of the movement. It does not address the question of how web archiving has been carried out, but why, by whom, and on whose behalf.

Historians have for long known that, in order to interpret archival materials properly, it is first necessary to understand how that archive came into being. Why is a particular object to be found, and not another? What does the archive seek to document, and whose interests does it serve? The last very few years has seen a very welcome growth in interest in the archived web among scholars. However, that interest is not yet accompanied by the necessary familiarity with how the archived web came into being, and to be thus familiar is arguably even more important in this context than for traditional paper-based archives. Older distinctions with which historians are familiar — between published document, ‘grey literature’ and institutional records — have become blurred, as have those between personal and institutional publication. As a result, it has become less clear where the responsibility for preserving which types of content lies among the established institutions in the library and archives field. In addition, the archived web resource is unlike the live version from which it was derived in subtle and complex ways that do not apply to print publications or to manuscripts. If this chapter serves to orient users as to some of the questions they should be asking of their sources, and of the institutions that provide them, it will have achieved its aim.

It falls into the following sections: The Internet Archive / National libraries / The corporate record / Research-driven archiving / Activist archiving / Users and the future

Reflections on Web Archiving Week 2017

Once in a while, the unplanned turns out to be as good if not better than the planned. It had not been the intention that the annual Web Archiving Conference of the IIPC should be combined with the second conference of ReSAW (the Research Infrastructure for the Study of Archived Web Materials). However, they came together in London last week, with intriguing results.

One of the great pleasures of the event is the diversity of both speakers and delegates: the institutions represented by the IIPC were there in strength, but also present were the largest assemblage of researchers I have yet seen. These include not only people from computer science and related fields – a group that has been engaged in this space for a while – but also an enlarged contingent of scholars of media and communications and several of the humanities disciplines. At the Archives Unleashed datathon on Monday and Tuesday there was a particularly creative meeting of scholars, technologists and archivists – the crucial nexus of relationships for making successful tools and services. The whole week was marked (for me) by a refreshing openness to the perspectives of others, a frankness about difference, and a collegiality without hierarchy which (if it can be sustained) bodes very well for the future.

If I compare this discussions last week with those in this community perhaps three or four years ago, a number of differences stand out. As I’ve tried to show in my short history of Web archiving, direct engagement between archiving institutions and researchers came relatively late in that twenty year history, and even four years ago there was still a sense that researcher engagement was still only very exploratory. We now seem to have reached the stage where substantial attention is being paid to understanding the needs of users as a preliminary step to developing new tools and services (of which there were also many exciting examples). Here I’m bound to mention the research study that I (as Webster Research and Consulting) carried out for the Parliamentary Archives, which Chris Fryer and I presented, but I also have in mind papers on citation practice (Nyvang et al), the research data management issues involved (Zierau and Jurik), and what users need to know about the materials they use (ie. what to do about descriptive metadata), a theme taken up variously by Bingham, Dooley et al, Maemura et al. The variety of different use cases both discussed in the abstract and demonstrated in concrete reminded me of how varied the user base for web archives is (or could be) and how much we need as fine-grained an understanding of those different users as possible. As Ben Steinberg of Harvard noted ‘How we [ie. the providers of services] think archives should or could be used may not be as pertinent as we imagine…’

Another theme for researchers that surfaced several times at the first ReSAW conference in Aarhus two years ago was the need to understand the offline as context for the online. In Aarhus the particular point was about the need for oral history and for analysis of print and manuscript sources to understand how web materials make it online to begin with, and the theme was taken up last week by Federico Nanni and (in passing) by Gareth Millward and Richard Deswarte. There were also reminders here that a full history of the Web will need to take account of the history of computing more generally (Baker and Geiringer), the interaction between the Web proper and other content delivered online, notably social media (Castex, Schafer et al, Day Thomson), as well as the wider social and intellectual context in which the Web is embedded (Schroeder, and my own paper on the religious language of the Web) .

What of the future? Delegates who followed the same tracks as me may have come away with a sense of the diversity of analytic approaches to the study of the Web, and impressed with the depth at which scholars are now seeking to understand the methodological challenges they face. The aim, however, must be to build on this reflection to a point at which the Web archive becomes simply one type of scholarly source amongst many in the production of substantive scholarly insight on history, sociology or literature as Gareth Millward noted. I look forward to the day when I can go to mainstream historical conferences and hear contemporary history written using the archived Web.

There is also, I think, a challenge to the community at large in navigating a path through the diversity of new technical development and analytical need on display here, to decide which elements best serve users in particular situations, and so should brought forward and made part of ‘business as usual’ operations. Some will be incorporated by web archives themselves, others maintained by communities of interested scholars, others probably commercialised. The IIPC has a part to play here, while remembering that a significant part of this new thinking is taking place outside the membership. At least one person on Twitter thought a combined conference like this was worth repeating, and it would certainly be a way of developing the listening process between archives, users and developers that is required.

Finally: I celebrated the diversity of the conference when viewed in terms of professional background, but in another sense there is still much to do in terms of geography. I counted some 17 or 18 nationalities represented here, a joyous thing in a fragmenting world, but nonetheless overwhelmingly from Europe and north America. The archiving and study of the Web, a global medium, still remains dominated by certain countries.

My thanks are due to all those involved in organising such an excellent event: Jane Winters as host at the School of Advanced Study (University of London), and Olga Holownia of the IIPC and my former colleagues at the British Library which also contributed most significantly. It was my pleasure to be a part of both the IIPC and the ReSAW programme committee, and to hear such a fine set of papers.

 

Religion, law and national identity in the archived Web: new article

I’m delighted to say that an article of mine has appeared this week in a new collection of essays, edited by Niels Brügger and Ralph Schroeder: The Web as History (London: UCL Press, 2017, ISBN: 9781911307563).

My article is ‘Religious discourse in the archived web: Rowan Williams, Archbishop of Canterbury, and the sharia law controversy of 2008’ (pp. 190-203). It examines the controversy over a public lecture given by the archbishop on the interaction of civil and religious law, but from a new angle: the imprint the controversy left in the archive of the UK web. It makes particular use of British Library data documenting the link structure of the .uk country code top level domain for the period 1996-2010.

The whole thing is available as an Open Access PDF, but here’s my conclusion.

It is a brave historian who attempts to interpret the very recent past, as opposed to merely documenting it. As with most aspects of very recent history, the full significance of Rowan Williams’ lecture about sharia law will only become clear as the passage of time grants the historian a sufficiently long perspective from which to view it. An exhaustive qualitative examination of both the published record, and memoirs and private papers that are as yet inaccessible (not least the papers of the archbishop himself, not due to be released until 2038) will be needed to place the episode in its fullest context. Without these, we cannot yet know how changes in patterns of communication that are observable in the archived web were motivated, or how opinions expressed online related to broader patterns of social and intellectual change. However, even if it is difficult to explain changing patterns of religious discourse on the web, we may nonetheless document those changes.

First, the sharia law episode prompted a step-change in the levels of attention paid to the domain of the archbishop of Canterbury, as evidenced by the incidence of inbound links, and also a broadening of the types of hosts that contained those links. Second, a comparison of the inbound links to the Canterbury domain to that of the archbishop of York suggests that the historic privilege given to the views of Canterbury over those of York was extended onto the web. Regardless of their actual status in relation to each other within the Church of England, the media and the public at large seemed only to pay attention to Canterbury. Finally, a qualitative examination of the site of the British National Party shows that at least one organization, with a very particular concern with the place of Islam in British life, certainly took new account of the person of the archbishop as a result of the 2008 controversy.

This chapter has also sought to use the episode as a means of demonstrating both the potential for historians to utilize the archived web to address older questions in a new way, and some of the particular issues of method that web archives present. At one level, the methodological complications presented here – understanding the meaning of a link from one resource to another, say – are peculiar to the archived web and must be understood anew. As with all other born- digital sources, there is work to be done amongst historians in understanding these issues of method, and in acquiring the skills needed to handle data at scale. At the same time, it is part of the historian’s stock- in- trade to assess the provenance of a body of sources, its completeness and the contexts in which those sources were transmitted and received. The task at hand is in fact the application of older critical methods to a new kind of source: a challenge which historians have confronted and overcome before.

This chapter has also tried to show some of the potential available to historians, should they accept the challenge. In the study of public controversy, the archived web allows the detection of changing communication patterns at scale that would be impossible using a traditional qualitative method. It also enables the detection of attention being paid online in places where a scholar would not think to look. More generally, the chapter has attempted to outline an approach that combines quantitative readings of the links in web archives with qualitative examination of particular subsets of resources. When dealing with a new superabundance of historical sources, a combination of distant and close reading will be required to understand the archived web.

Lessons from cross-border religion in the Irish web sphere

[UPDATED: 21 June 2018. This book is now in production and should appear before the end of 2018.]

I’m delighted to announce that I have a chapter accepted (subject to peer review) in a forthcoming book of essays on national web domains, The Historical Web and Digital Humanities: the Case of National Web domains. It is edited by Niels Brügger & Ditte Laursen and will be published by Routledge.

Here’s the abstract.

Understanding the limitations of the ccTLD as a proxy for the national web: lessons from cross-border religion in the Northern Irish web sphere

The web archiving community has known for a long while that the country-code top level domain (.uk, .ie) only ever represents a subset (although a very substantial one) of a national web sphere. Every national sphere (when defined as content published within a national jurisdiction) includes very substantial amounts of content that resides within the various geographically non-specific domains, such as .com or .org. However, the current state of knowledge is such that little is known with any certainty about the content that ‘lives’ outside the ccTLD, and what factors determine the choices made by webmasters as to the domain registrations they choose.

The situation is particularly complicated in the island of Ireland, since two political units (the UK and the Republic of Ireland) and two ccTLDs (.uk and .ie) share (as it were) a land border. This chapter takes the case of the Christian churches in Ireland (north and south) as a case study in the mapping (or lack of it) between the nation and the ccTLD. The churches in Ireland are organised on an all-Ireland basis: a reflection of their origins that pre-date the partition of Ireland into north and south.

The chapter makes two distinct but related points. It investigates the degree to which the differing historic attitudes of Protestant and Roman Catholic churches in Northern Ireland to national identity are reflected in patterns of domain registration. Based on data for 2015 and 2016, it shows that Roman Catholic congregations were more likely to register domains outside the .uk ccTLD. However, there was no corresponding prioritisation of registration within .uk for the several Protestant denominations. If organisations which might be expected to register their web estate within a particular national domain do not in fact do so, it suggests that the ‘gravitational pull’ of the ccTLD is weak. Focussing on the Baptist churches in particular, the chapter also shows that the network of links between the individual Baptist church congregations on both sides of the border between 1996 and 2010 was both tightly focussed around the churches in Northern Ireland, and also highly localised within one part of the province, whilst at the same time being spread across four TLDs. While offline patterns of numeric strength and geographic concentration are reflected online, in this case at least, they map only very loosely to the ccTLD. As such, it would seem that the ccTLD on its own is a weak proxy for the national web.

What do we need to know about the archived web?

A theme that emerged for me in the IIPC web archiving conference in Reykjavik last week was metadata, and specifically: precisely which metadata do users of web archives need in order to understand the material they are using?

At one level, a precise answer to this will only come from sustained and detailed engagement with users themselves; research which I would very much hope that the IIPC would see as part of its role to stimulate, organise and indeed fund. But that takes time, and at present, most users understand the nature of the web archiving process only rather vaguely. As a result, I suspect that without the right kind of engagement, scholars are likely (as Matthew Weber noted) to default to ‘we need everything’, or if asked directly ‘what metadata do you need?’ may well answer ‘well, what do you have, and what would it tell me?’

During my own paper I referred to the issue, and was asked by a member of the audience if I could say what such enhanced metadata provision might look like. What I offer here is the first draft of an answer: a five-part scheme of kinds of metadata and documentation that may be needed (or at least, that I myself would need). I could hardly imagine this would meet every user requirement; but it’s a start.

1. Institutional
At the very broadest level, users need to know something of the history of the collecting organisation, and how web archiving has become part of its mission and purpose. I hope to provide a overview of aspects of this on a world scale in this forthcoming article on the recent history of web archiving.

2. Domain or broad crawl
Periodic archiving of a whole national domain under legal deposit provisions now offers the prospect of the kind of aggregate analysis that takes us way beyond single-resource views in Wayback. But it becomes absolutely vital to know certain things at a crawl level. How was territoriality determined – by ccTLD, domain registration, Geo-IP lookup, curatorial decision? The way the national web sphere is defined fundamentally shapes the way in which we can analyse it. How big was the crawl in relation to previous years? How many domains are new, and how many have disappeared? What’s the policy on robots.txt (by default) ? How deep was the crawl scope (by default)? Was there a data cap per host? Some of this will already be articulated in internal documents, some will need some additional data analysis; but it all goes to the heart of how we might read the national web sphere as a whole.

3. Curated collection level
Many web archives have extensive curated collections on particular themes or events. These are a great means of showcasing the value of web archives to the public and to those who hold the pursestrings. But if not transparently documented they present some difficulties to the user trying to interpret them, as the process introduced a level of human judgment to add to the more technical decisions that I outlined above. In order to evaluate the collection as a whole, scholars really do need to know the selection criteria, and at a more detailed level than is often provided right now. In particular, in cases where permissions were requested for sites but not received, being able to access the whole list of sites selected rather than just those that were successfully archived would help a great deal in understanding the way in which a collection was made.

4. Host/domain level
This is the level at which a great deal of effort is expended to create metadata that looks very much like a traditional catalogue record: subject keywords, free-text descriptions and the like. For me, it would be important to know when the first attempt to crawl a host was, and the most recent, and whether there were 404 responses received for crawl attempts at any time in between. Was this host capped (or uncapped) at the discretion of a curator differentially to the policy for a crawl as a whole? Similarly, was the crawl scoping different, or the policy on robots.txt? If the crawl incorporates a GeoIP check, what was the result? Which other domains has it redirected to, and which redirect to it, and which times?

5. Individual resource level
Finally, there are some useful things to know about individual resources. As at the host level, information about the date of the first and last attempts to crawl, and about intervening 404s, would tell the user useful things about what we might call the career of a resource. If the resource changes, what is the profile of that: for instance, how has the file size changed over time? Were there other captures which were rejected, perhaps on a QA basis, and if so, when?

Much if not quite all of this could be based on data which is widely collected already (in policy documents, or curator tools, crawl logs or CDX) or could be with some adjustment. It presents some very significant GUI design challenges in how best to deliver these data to users. Some might be better delivered as datasets for download or via an API. What I hope to have provided, though, is a first sketch of an agenda for what the next generation of access services might disclose, that is not a default to ‘everything’ and is feasible given the tools in use.