Pick a corpus, any corpus

Sun, 13 Mar 2011 01:53:59 +0000

A few weeks ago I participated in a brainstorming session exploring the kinds of academic research projects the WikiLeaks archives might generate. Beyond the substantive specifics of the leaked cables, the media coverage of Cablegate, and their impact on geopoltics, a central concern we recognised is the challenge of transforming torrents of qualitative data into narratives, arguments, and evidence . The impact that technology is having on what’s knowable and how we go about knowing is a theme I have been chewing on for years – one that goes well beyond journalism, and cuts across the social sciences, law, education, etc. There is an urgency to this problem since the tools and techniques involved in these analyses are unevenly distributed. High-end corporate law firms, marketing agencies, and political parties are all embracing new approaches to making sense of petabytes. Unfortunately, impact law firms, social scientists, and journalists often don’t even know these tools exist, never mind how to use them. Part of what I call the organizational digital divide. During our brainstorming I formulated a new twist on a possible research agenda. I realized how daunting it has become to evaluate and calibrate the emerging suites of digital instruments. There are many digital tools emerging that can be used to analyze large troves of data, but it is difficult to determine what each tool is best at, and if it does its job well. One good way to benchmark our digital instruments is to select a standard corpus, and spend lots of time researching and studying that corpus until the corpus is fairly well understood. Similar to the role that the Brown Corpus played in computational linguistics, data miners need a training ground we can test, hone, and sharpen our digital implements. If we bring a new tool to bear on a well understood archive, we can evaluate its performance relative to our prior understanding. Currently Wikipedia serves as the de-facto benchmark for many digital tools, though, since its a moving target, it is probably not the best choice for calibration. In many respects the selection of this kind of corpus can be arbitrary, though it needs to be adequately sophisticated, and we might as well pick something that is meaningful and interesting. The Wikileaks documents are an excellent contender for training the next generation digital instruments and data miners. The AP is hard at work on new approaches for visualizing the Iraq War logs, and just last week there was a meetup for hacks and hackers working on the wikileaks documents Data Science & Data Journalism . It is easy to see how Knight funded projects like DocumentCloud converge on this problem as well. Ultimately, I think these efforts should move in the direction of interactive storytelling, not merely an passive extraction of meaning. We need tools that enable collaborative meaning-making around conceptual space similar to what Ushahidi has done for geographic space.

Faith's Transmission

Tue, 23 Jun 2009 23:17:14 +0000

Well, its been 2 months since I participated in MIT’s Media in Transition (MiT6), but the event is still vividly fresh in my mind. The conference was really amazing. It attracted a really diverse mix of theorists and practitioners, academics and professionals, and folks from many walks of life. This conference I tried to go to talks where I “didn’t belong” - hoping to learn from disciplines I don’t regularly encounter. It was a great strategy, as I often gravitate towards talks that I know something about, wanting to hear the presenter’s take on it, but venturing beyond my usual horizons was much more fun. Aram Sinnreich and I presented a paper on Strategic Agency in an Age of Limitless Information (abstract, slides), and I am really happy with how things turned out. Hopefully, we’ll work on polishing this paper up to submit to a journal soon, though I don’t really know where we should submit yet. The videos for the main plenary events are now up and I am looking forward to clipping the little hand grenades I remember throwing during Q&A. This panel on Archives and History (my question starts @ 1:35:15) wasn’t the only conversation about archiving, but it was fairly representative of the perspectives. It’s too bad MIT World does not provide me with a mechanism to address a point of time in their videos (like our recently liberated VITAL tool allows), so you’ll have to advance the playhead manually to hear me out. It’s basically a riff on - Why Archive? - The beauty of the Sand Mandala and the effort required to actually delete something…. The conversations were very similar to some that we had back in May ‘07 at the Open Content conference, but not I think I can finally articulate what’s been bugging me about these conversations. With the help of Ben and John Durham Peters (we shared a bus ride to/from the conf), I realized that archiving can be thought about as a transmission, for anyone, into the future. I also realized that ordinarily, when we look to the past, we use history to help us understand ourselves better. The presumption that future generations will actually care about us for our own sake, strikes me as narcissistic (narcissism and new media has surfaced on this blog before). I imagine they will want to use the messages that we send them to help themselves, understand themselves better. So, to archive purposefully the question becomes - how can we best help the future? To the archivists who claim we don’t have any idea what questions the future will be asking, so we better save it all - I think I know what the future will be trying to understand about us. They will likely be trying to figure out what on earth was distracting us while we let the planet die! We were busy devoting our resources to saving every last copy of American Idol and Big Brother while Gia screamed in agony for help. So, how can we increase the signal-to-noise ration of the messages we send into the future? Without somehow reducing the message to the critically problematic golden record on the voyager spaceship, or its successors? I guess the Long Now Foundation is thinking along these lines, and I have always envied David Vakoch’s job title (Director of Interstellar Message Composition)… The conference helped me realize that Vakoch and the Long Now have a really similar task - but I don’t know how many archivists conceive of their task as Intergenerational Message Composition. Perhaps we need to spend even more time curating? Indicating in our archives why we think they were worth saving? And what’s the most important message we can send into the future? Not like it matters much longer, as I really do believe we are embarking on The End of Forgetting (see our conf paper for more details). Shifting frames for a moment, what if the ancients had a really important message to send us? Their Theory of Everything, or the equivalent of E=MC^2. How would they have attempted to transmit it? When I discussed these ideas w/ my friend Rasmus he recommended I start up a consulting firm specializing in Future Relations. ;-)

Temporal-Transmission on Alchemical Musings

Pick a corpus, any corpus

Faith's Transmission