Museums and the audience comments paradox

I was at the Imperial War Museum for an advisory board meeting for the Social Interpretation project recently, and had a chance to reflect on my experiences with previous audience participation projects.  As Claire Ross summarised it, the Social Interpretation project is asking: does applying social media models to collections successfully increase engagement and reach?  And what forms of moderation work in that environment – can the audience be trusted to behave appropriately?

One topic for discussion yesterday was whether the museum should do some 'gardening' on the comments.  Participation rates are relatively high but some of the comments are nonsense ('asdf'), repetitive (thousands of variants of 'Cool' or 'sad') or off-topic ('I like the museum') – a pattern probably common to many museum 'have your say' kiosks.  Gardening could involve 'pruning' out comments that were not directly relevant to the question asked in the interactive, or finding ways to surface the interesting comments.  While there are models available in other sectors (e.g. newspapers), I'm excited by the possibility that the Social Interpretation project might have a chance to address this issue for museums.

A big design challenge for high-traffic 'have your say' interactives is providing a quality experience for the audience who is reading comments – they shouldn't have to wade through screens of repeated, vacuous or rude comments to find the gems – while appropriately respecting the contribution and personal engagement of the person who left the comment.

In the spirit of 'have your say', what do you think the solution might be?  What have you tried (successfully or not) in your own projects, or seen working well elsewhere?

Update: the Social Interpretation have posted I iz in ur xhibition trolling ur comments:

"One of the most discussed issues was about what we have termed ‘gardening comments’ but to put it bluntly it’s more a case of should we be ‘curating the visitor voice’ in order to improve the visitor experience? It’s a difficult question to deal with… 

We are at the stage where we really do want to respect the commenter, but also want to give other readers a high value experience. It’s a question of how we do that, and will it significantly change the project?"

If you found this post, you might also be interested in Notes from 'The Shape of Things: New and emerging technology-enabled models of participation through VGC'.

Update, March 2014: I've just been reading a journal article on 'Normative Influences on Thoughtful Online Participation'. The authors set out to test this hypothesis:

'Individuals exposed to highly thoughtful behavior from others will be more thoughtful in their own online comment contributions than individuals exposed to behavior exhibiting a low degree of thoughtfulness.' 

Thoughtful comments were defined by the number of words, how many seconds it took to write them, and how much of the content was relevant to the issue discussed in the original post. And the results? 'We found significant effects of social norm on all three measures related to participants’ commenting behavior. Relative to the low thoughtfulness condition, participants in the high thoughtfulness condition contributed longer comments, spent more time writing them, and presented more issue-relevant thoughts.' To me, this suggests that it's worth finding ways to highlight the more thoughtful comments (and keeping pulling out those 'asdf' weeds) in an interactive as this may encourage other thoughtful comments in turn.

Reference: Sukumaran, Abhay, Stephanie Vezich, Melanie McHugh, and Clifford Nass. “Normative Influences on Thoughtful Online Participation.” In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, 3401–10. Vancouver, BC, Canada: ACM, 2011. http://dl.acm.org/citation.cfm?id=1979450.

15 thoughts on “Museums and the audience comments paradox”

  1. Could even simple and prosaic comments such as 'cool' or 'nice' could have their own value if you get sufficient audience numbers? Perhaps they are analogous to the ubiquitous 'like' button?

    Would it be possible to consider the idea of using well-tested text-mining methods to comments? Would sentiment analysis on comments gain at least an idea (given the numbers) on feelings or conjecture surrounding objects? Perhaps it would be interesting to apply "distant reading" principles to the comments section – although YouTube has it's fair share of (mostly) listless comments, it's scale poses some interesting possibilities of exploring sentiment and using machine learning techniques to predict whether comments are useful or not :

    http://dl.acm.org/citation.cfm?id=1772781

    Comments that are identified as potentially pointless or non-discriminant could be earmarked for moderation – the issue of discerning useful comments is problematic at the readership level for an individual object, entity or post. But what about scaling the subject of analysis to a group of objects or even an entire collection or institution – asking the questions – "what does *everyone* think?" – "what's the collective sentiment?"

  2. The Science Museum's Antenna gallery interactives and website included a 'like' button to give people an outlet for that 'this is cool' feeling (random example: http://antenna.sciencemuseum.org.uk/deep-sea-Alvin) and to potentially provide data for interfaces that presented content based on the number of interactions with it.

    If we did aggregate comments together (maybe showing the comment text with the names of the last five people who said it), does that give people the satisfaction of seeing their comment go live? And does it respect their intention in posting it? I'd love to see research into why people leave comments on gallery interactives, particularly of the 'cool' or 'I like this object' variety.

    The potential for data-mining comments for collective sentiment is interesting. I wonder if any museums have already done that?

  3. One of the problems with moderation, whether human or algorithmic, is the delay. People tend to post comments after they read some comments. Once you hit the submit button, you'd really like to see your comment on display immediately. There's really no way to offer a satisfying comment experience without risking the quality of the reading experience. Website operators got used to this idea a long time ago (which explains why so many mainstream websites are filled with terrible comments and why so many other websites just turned commenting off).

    I like the idea of a "like" button, but it seems so limited. I'm not sure what kind of useful information you could gather from that aside from discovering which exhibits are the most popular (which we can already figure out most times).

    Perhaps you could use a multiple choice question of some sorts. Have a question that is directly relevant to the content on display and make each answer meaningful in some way. You could have a display at a history museum, for instance, and present a set of differing interpretations of a historic object or event tied to your display. Ask the audience which of those interpretations they think is most likely the "true" one. Display the aggregate results in the gallery and on your website.

    Then your question serves multiple purposes. You've educated the audience by showing them that history is speculative, open to differing interpretations for different reasons, and you've shown them what some of those interpretations (for a particular object) are. And you've collected data about public sentiment that you can put to use for future displays and share with others for sociologic study. You could use the data gathered to develop more programming, either responding to public demand or addressing a perceived need.

    There are ethical considerations to think about in many cases, but if we're going to give our audience just a button to press, let's attach some meaning to it beyond just "I like this."

  4. Over on Twitter, Harriet Deacon (@the_archive) made a case for simple comments:
    "Maybe reading simple comments like 'great' make visitors feel relaxed? … Edit out offensive ones but over-curation of visitor comments might limit further engagement by raising the erudition bar".

    I think the answer depends on scale – maybe it's ok to see five comments saying 'cool' but what if you have to scroll through 50 of them to see something more interesting?

    Matt – there was discussion yesterday about showing the most recent comments mixed in with comments that had been picked by staff or voted up by visitors. The 'like' button worked quite well for contemporary science stories that didn't have an object attached, but we also had polls and comments to provide other ways of interacting with the post. It's a high-traffic gallery so in a way it saved being overwhelmed with comment variations on 'I like this'.

    Pre- vs post-moderation is a big issue – IIRC, we had post-moderation for online comments on Antenna, and pre-moderation for in-gallery comments. Not ideal, but with up to 8000 school kids intent on teasing each other passing through the galleries each day, it was necessary. I didn't see the same level of school yard humour in the IWM gallery, which was interesting in itself. They're going entirely post-moderated in gallery and their results will be really useful for other museums.

  5. "I think the answer depends on scale – maybe it's ok to see five comments saying 'cool' but what if you have to scroll through 50 of them to see something more interesting?"

    Surely if 49 people wrote "cool", and a 50th person still felt the urge to make the same comment, then they are seeing some value in contributing that comment, and reading the same?

  6. Separating out the value in leaving the comment and the value in repeating that comment on page after page is the crux of the matter, isn't it?

    I suspect that the people reading the comments don't always overlap with the people leaving comments, though the proportions would depend on the core audiences for any given museum.

    If there's any value in knowing that 50 or 5000 people thought it was cool (which as Matt pointed out, is a whole other question), I personally don't want to have to scroll past the same comment 50 or 5000 times to understand how many people have said it. As Tim suggested, there's a role for clever visualisations in presenting the gist of content.

    To put my question another way, how can museums increase the signal-to-noise ratio so that the rarer, more contemplative, factual, heart-felt or personal experiences can shine?

  7. I find it strange that museums are still keen to invest in 'have your say' kiosks – paper comment books had exactly the same problem and it isn't as if the different medium makes them any better.

    I know there's an argument that by having these kiosks (or comment books) that it gives the impression that the museum 'cares' or is 'participatory' – but it really is just illusory.

    There's a lot to be said for raising the bar for participation just high enough to reduce the volume of low value comments – either by question design, kiosk design and authentication, or by requiring a serious commitment from the visitor, and, of course, scaffolding.

    I'm always reminded of the StoryCorps project that has been running for years that Local Projects designed. The heavy requirement – a 45 minute sit down interview – combined with good scaffolding as to why you'd want to commit the time, yielded and continues to yield fantastic results.

    That's at one end of the scale obviously, but the other end of the scale shouldn't be zero-commitment 'virtual comment books'.

  8. I think the QRator team, in their presentation at Museums and the Web, put forward a good example of visitors questions that work fine: ask direct questions instead of the open-ended "have your say"/"comment here". Asking, for instance, "should we display or conserve our objects" will provoke more interesting, thought-through ansers from the audience, and still won't stop them from giving general comments.

  9. Thanks for commenting, Aron and Seb!

    I've been thinking about the issue of general comments being thrown in with careful responses to the specific questions posed… How much does an appropriate level of gardening depend on the model we're expecting to see?

    I'd expect the Letters page of a newspaper to be carefully considered in terms of balance, timeliness and how interesting the comments are. Unfortunately the same isn't true for comments 'below the line' on an online newspaper, where trolling, one-upmanship and irrelevance are sadly closer to the norm… but which is an appropriate model for a museum interactive?

    And if you absolutely had to pick, whose experience should be prioritised – the readers, or the commenters?

  10. A few days ago Clay Shirky posted 'Pay attention to what Nick Denton is doing with comments'
    http://www.niemanlab.org/2012/06/pay-attention-to-what-nick-denton-is-doing-with-comments/

    "Most news sites have come to treat comments as little more than a necessary evil, a kind of padded room where the third estate can vent, largely at will, and tolerated mainly as a way of generating pageviews. This exhausted consensus makes what Gawker is doing so important. …re-designed Gawker to serve the people reading the comments, rather than the people writing them." [my emphasis]

    And: "The new design dispenses with the tyranny of time order. … On most systems, the most prominent comments are posted either by the most obsessed users (when comments are posted oldest first) or the drive-bys (newest first). …there is, by design, no way for regular participants … to use either volume or aggression to maximize attention."

  11. More on dealing with comments: 'On a smaller site, on niche content, with active community management, where people aren’t being dicks for the sake of being dicks, you can have a considered discussion and good community interaction. Getting that to scale to the level of a high traffic general news site seems a Sisyphean task.' From 'Commenting about newspaper website comments. Again. 'http://martinbelam.com/2013/newspaper-website-comments-mediabriefing/

    And via that, after @charltonbrooker said "Enabling reader comments is one of the worst thing to have happened to newspapers" some other Guardian writers discussed the issue – see https://twitter.com/HadleyFreeman/status/299429630589403136

    This recent discussion has made me realise that too much emphasis on a small group of commentators means your newspaper could become a forum with articles attached at the top of new threads. (And I say this as someone who's spent a lot of time on forums in the past).

    And back to issues particularly relevant to museums: 'Online comments hurt science understanding, study finds' http://www.jsonline.com/features/health/online-comments-hurt-science-understanding-study-finds-ib88cor-185610641.html

  12. I've just re-read Peter Samis' 2008 Museums and the Web paper (Who Has The Responsibility For Saying What We See? Mashing up Museum, Artist, and Visitor Voices, On-site and On-line), where he discusses 'stereotypical effusions' like 'cooooooool!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!' and asks:

    'What are we to do with this? With ourselves? Are we just hopelessly uptight, or should we do our visitors and readers a service by curbing such unbridled expressions of enthusiasm, especially when they bring no added or original insight to the table?'

  13. How about a way for people (both staff and visitors) to "recommend" particularly insightful comments and an algorithm that presents the "most recommended" comments first? Surely that's been done already?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.