Tag: evaluation

Interview about museum metadata games and a pretty picture

I haven't had a chance to follow up Design constraints and research questions: museum metadata games with a post about the design process for the museum metadata games I've made for my dissertation project (because, stupidly, I slipped on black ice and damaged my wrist), so in the meantime here's a link to an interview Seb Chan did with me for the Fresh+New blog, Interview with Mia Ridge on museum metadata games, and a Wordle of the tags added so far.

There have been nearly 700 turns on the games so far, which have collectively added about 30 facts (Donald’s detective puzzle) and just over 3,700 tags (Dora’s lost data).

Some of the 1,582 unique tags added so far

Design constraints and research questions: museum metadata games

Back in June I posted parts of my dissertation project outline in 'Game mechanics for social good: a case study on interaction models for crowdsourcing museum collections enhancement'. Since then, I've been getting on with researching, designing, building and evaluating museum metadata games (in my copious spare time after work, in a year when we launched three major galleries).

I'm planning to blog bits of my dissertation as I write it up so there'll be more posts over the next month, but for now I wanted to contextualise the two games I'm evaluating at the moment.  In the next post I'll talk about the changes I made after the first solid round of evaluation.

Casual games
The two games, nicknamed 'Dora' and 'Donald' are designed as casual games – something you can pick up and play for five minutes at a time.  Design goals included: an instantly playable game that provides stress relief, supports a competitive spirit (but not necessarily against other people), inherently rewarding experience, simple game play and puts 'fun before do-gooding'.  The games were designed around a specific research-based persona ('Meet Janet', pdf link) – hopefully it's exactly right for some people who are close to the persona in various ways, and quite fun for a wider group.  It won't suit everyone, not least because definitions of 'fun' and expectations around 'games' can be deeply individual.

Design constraints
The games are also designed to test ideas about the types of objects and records that can be used successfully, and the types of content people would be able to contribute about the less charismatic and emotionally accessible reaches of science, technology and social history collections – this means that some of the objects I've used are quite technical, not all the images are great and small variations on object records are repeated (risking 'not another bloody telescope').  While this might match the reality of museum catalogues, would it still allow for a fun game?

The realities of a project I was building in my free time and my lack of graphic design and illustration skills also provided constraints – it had to be browser-based, it couldn't rely on a critical mass of concurrent players to validate actions or content, it had to help the player dive straight into playing and overcome any fears about creating content about museum objects, and it had to use objects ingested through available museum APIs (I selected broad subjects for testing but didn't individually select any objects).

I then added a few extra constraints by deciding to build it as a WordPress plugin – I wanted to take advantage of the CMS-like framework for user logins, navigation and page layout, and I wanted the code I wrote to be usable by others without too much programming overhead.  I'll need to tidy up the code at the end, but once that's done you should be able to install it on any hosted WordPress installation.  I'm making a related plugin to help you populate the database with objects (also part of an experiment in the effectiveness of letting people choose their own subject areas or terms to select playable objects).  I'll talk more about how I worked with those constraints and how they informed the changes I made after evaluation in a later post.

Different games for different purposes
I've been thinking about a museum metadata game typology, which not only considers different types of fun, but also design constraints like:

  • the type and state of the collection (e.g. art works, technical/specialist and social history objects; photographs and other media vs objects; reference collections vs selected highlights; 'tombstone' vs general vs interpretative records)
  • the type of data sought including information curators could add if they had infinite time (detail on the significance of the object, links to other subjects, people, events, objects, collections, etc); information that can be extrapolated from the existing catalogue record; things curators couldn't know (personal history, experiential accounts about the design, manufacture, use, disposal etc of objects); emotional responses; external specialist knowledge; amateur/hobbyist specialist knowledge; synonyms in every day language; terms in other spoken languages

I've also been playing with the idea of linking different game types to different 'life stages' of museum collection metadata.  For example, some games could help a museum work out which of its catalogued items seem more interesting to the public, others help gather tags, create links between items or encourage players to research objects and record new information or links about them, and others still could work well for validating data created in earlier games.  The data I gather through evaluating the games I've designed will help test this model.

So, all that said, if you'd like to play (and help with my evaluation), the two games are:

Donald's detective puzzle – find a fact about an object
Dora's lost data – a simpler tagging game

One reason why (science) museums rock

When I started work at the Science Museum a few years ago, after five years with a social history museum and archaeology service, I was curious about how the additional tasks of communicating scientific principles, contemporary science news and the history of science and technology would affect interpretation, collections and exhibitions. With that in mind, I did some research about science museums and came across an article about the impact of the Palais de la Découverte (a science museum in Paris) translated for me as:

In a recent work entitled 'Comment devient-on scientifique?' (How does one become a scientist?) published by Editions EDP, Florence Guichard indicates the results of a survey undertaken in the Ile-de-France: 60% of scientists over 30 and 40% of scientists under 30 claim, without prompting, that the Palais de la Découverte triggered their vocation. Pierre Gilles of Gennes, the winner of the Nobel prize for Physics is one of those 'lovers' of thePalais de la Découverte who was still visiting the place a few years before his death in 2007.

[Update – coincidentally, the day after posting, I came across another reference to the impact of science museums on children's interest in science:] The evolution of the science museum:

When did scientists first become interested in science? A 1998 survey of 1400 scientists, conducted by the Roper Starch organization for the Bayer Foundation and NSF, reported that a respected adult, such as a parent, was the biggest factor in stimulating childhood interest in science. […] a variety of informalactivities had an effect. […] 76 percent said science museum visits

That's pretty amazing.  But ok, for people who aren't scientists, why does science matter?  Well, for a start, the UK needs to keep up with the rest of the world, and there are global problems that we need scientists to help solve.  At about the same time, President Obama stated in his inaugural address that he would "restore science to its rightful place". Ed Yong, among others, answered the question, so 'what is science's rightful place?':

…underneath all of the detail lie some basic principles that science is built upon and these, I feel, ought to be more mainstream than they perhaps are. We should be strive to be unceasing in our curiosity, rational in our explanations and accurate in our communication. We should value inquiry and the power of evidence to change opinions. We should be unflinching in our search for understanding and the desire to test the world around us.

There is no question in my mind that these tenets should act as guides to our lives (albeit not exclusively; they are necessary, rather than sufficient). This is the greatest contribution of science to society. It acts as a stimulant that keeps us from sleepwalking through a wonderland. It is a cloth that wipes away superstition and myths to reveal an ever-closer approximation of the truth. It is a mental prophylactic that shields our minds from the folly of confirmation bias or the lure of unrepresentative anecdotes.

Tell people about the latest discoveries and many will ask what the significance is to their lives. In some cases, there's no way to answer that – they either appreciate it or they don't. But the very question misses an important point. The actual results may not be relevant but the principles that underlie them most definitely are, and they are omnipresent. Curiosity. Investigation. Communication. What could be more human or more pertinent to our casual existence?

The UK curriculum (key stage 1, key stage 3) aims to teach the value of scientific enquiry and perhaps fire a lifelong 'curiosity about phenomena in the world around them', and I suspect science museums make that teaching just a bit easier.  I know that the majority of people I talk to can still remember their school visit to science museums, but I haven't yet asked them what effect it might have had on their lives – does anyone know of any research?

Slides and talk from 'Cosmic Collections' paper

This is a lazy post, a straight copy and paste of my presentation notes (my excuse is that I'm eight days behind on everything at work and uni after being grounded in the US by volcanic ash). Anyway, I hope you enjoy it or that it's useful in some way.

Cosmic Collections: creating a big bang?

View more presentations from Mia .

Slide 1 (solar rays – Cosmic Collections):

The Cosmic Collections project was based on a simple idea – what if we gave people the ability to make their own collection website? The Science Museum was planning an exhibition on astronomy and culture, to be called ‘Cosmos & Culture’. We had limited time and resources to produce a site to support the exhibition and we risked creating ‘just another exhibition microsite’. So what if we provided access to the machine-readable exhibition content that was already being gathered internally, and threw it open to the public to make websites with it?  And what if we motivated them to enter by offering competition prizes?  Competition participants could win a prize and kudos, and museum audiences might get a much more interesting, innovative site.
The idea was a good match for museum mission, exhibition content, technical context, hopefully audience – but was that enough?
Slide 2 (satellite dish):
Questions…
If we built an API, would anyone use it?
Can you really crowdsource the creation of collections interfaces?
The project gave me a chance to investigate some specific questions.  At the time, there were lots of calls from some quarters for museums to produce APIs for each project, but would anyone actually use a museum API?  The competition might help us understand whether or how we should invest in APIs and machine-readable data.
We can never build interfaces to meet the needs of every type of audience.  One of the promises of machine-readable data is that anyone can make something with your data, allowing people with particular needs to create something that supports their own requirements or combines their data with ours – but would anyone actually do it?
Slide 3 (map mashup):
Mashups combine data from one or more sources and/or data and visualisation tools such as maps or timelines.
I'm going to get the geek stuff out of the way and quickly define mashups and APIs…
Mashups are computer applications that take existing information from known sources and present it to the viewer in a new way. Here’s a mashup of content edits from Wikipedia with a map showing the location of the edit.
Slide 4 (APIs)
APIs (Application Programming Interfaces) are a way for one machine to talk to another: ‘Hi Bob, I’d like a list of objects from you, and hey, Alice, could you draw me a timeline to put the objects on?’
APIs tell a computer, 'if you go here, you will get that information, presented like this, and you can do that with it'.
A way of providing re-usable content to the public, other museums and other departments within our museum – we created a shared backend for web and gallery interactives.
I think of APIs as user interfaces for developers and wanted to design a good experience for developers with the same care you would for end users*.  I hoped that feedback from the competition could be used to improve the beta API
* we didn’t succeed in the first go but it’s something to aim for post-beta
Slide 5: (what if nobody came?)
AKA 'the fears and how to deal with them'
Acknowledge those fears
Plan for the worst case scenario
Take a deep breath and do it anyway
And on the next slides, the results.  If I was replicating the real experience, you’d have several nerve-biting months while you waited for the museum to lumber into gear, planned the launch event, publicised the project in the participant communities… Then waited for results to come in. But let’s skip that bit…
Slide 6: (Ryan Ludwig's http://www.serostar.com/cosmic/)
The results – our judges declared a winner and a runner-up, these are screenshots – this is the second prize winning entry.
People came to the party. Yay! I'd like to thank all the participants, whether they submitted a final entry or not. It wouldn't have worked without them.
Slide 7: (Natalie and Simon's http://cosmos.natimon.com/)
This is a screenshot from the winning site – it made the best use of the API and was designed to lure the visitor in and keep drawing them through the site.
(We didn’t get subject specialists scratching their own itch – maybe they don’t need to share their work, maybe we didn’t reach them. Would like to reach researchers, let them know we have resources to be used, also that they can help us/our audiences by sharing their work)
Slide 8: (astrolabe – what did we learn?)
People need (more) help to participate in a geektastic project like this
The dynamics of a competition are tricky
Mashups are shaped by the data provided – you get out what you put in
Can we help people bring their own content to a future mashup?
Slide 9: (evaluation)
I did a small survey to evaluate the project… Turns out the project was excellent outreach into the developer community. People were really excited about being invited to play with our data.  My favourite quote: "The very idea of the competition was awesome"
Slide 10: (paper sheet)
Also positive coverage in technical press. So in conclusion?
Slide 11: (Tim Berners-Lee):
“The thing people are amazed about with the web is that, when you put something online, you don’t know who is going to use it—but it does get used.”
There are a lot of opportunities and excitement around putting machine-readable data online…
Slide 12: Tim Berners-Lee 2:
But:  It doesn’t happen automatically; It’s not a magic bullet
But people won't find and use your APIs without some encouragement. You need to support your API users. People outside the museum bring new ideas but there's still a big role for people who really understand the data and audiences to help make it a quality experience…
Slide 13 (space):
What next?
Using the feedback to focus and improve collection-wide API
Adding other forms of machine-readable data
Connecting with data from your collections?
I've been thinking about how to improve APIs – offer subject authorities with links to collections, embed markup in the collections pages to help search engines understand our data…
I want more! The more of us with machine-readable data available for re-use, the better the cross-collections searches, the region or specialism-wide mashups… I'd love to be able to put together a mashup showing all the cultural heritage content about my suburb; all the Boucher self-portraits; all the inventions that helped make the Space Shuttle work…
Slide 14: (thank you)
If you're interested in possibilities of machine-readable data and access to your collections, join in the conversation on the museum API wiki or follow along on twitter or on blogs.  Join in at http://museum-api.pbworks.com/
More at https://openobjects.org.uk/ or @mia_out

Image credits include:
http://antwrp.gsfc.nasa.gov/apod/ap100415.html
http://antwrp.gsfc.nasa.gov/apod/ap100414.html
http://antwrp.gsfc.nasa.gov/apod/ap100409.html
http://antwrp.gsfc.nasa.gov/apod/ap100209.html
http://antwrp.gsfc.nasa.gov/apod/ap100315.html
http://www.sciencemuseum.org.uk/Centenary/Home/Icons/Pilot_ACE_Computer.aspx

Mash the state

Cosmic Collections – the results are in. And can you help us ask the right questions?

For various reasons, the announcement of the winners of our mashup competition has been a bit low key – but we're working on a site that combines the best bits of the winners, and we'll make a bit more of a song and dance about it when that's ready.

I'd like to take the opportunity to personally thank the winners – Simon Willison and Natalie Down in first place, and Ryan Ludwig as runner-up – and equally importantly, those who took part but didn't win; those who had a play and gave us some feedback; those who helped spread the word, and those who cheered along the way.

I have a cheeky final request for your time.  I would normally do a few interviews to get an idea of useful questions for a survey, but it's not been possible lately. I particularly want to get a sense of the right questions to ask in an evaluation because it's been such a tricky project to explain and 'market', and I'm far too close to it to have any perspective.  So if you'd like to help us understand what questions to ask in evaluation, please take our short survey http://www.surveymonkey.com/s/5ZNSCQ6 – or leave a comment here or on the Cosmic Collections wiki.  I'm writing a paper on it at the moment, so hopefully other museums (and also the Science Museum itself) will get to learn from our experiences.

And again – my thanks to those who've already taken the survey – it's been immensely useful, and I really appreciate your honesty and time.

A sort of private joy? User-generated content and museums

I came across this lovely perspective on the content visitors create with museums:

We didn't start out asking people to leave their work, but it always happened. Now, we build it into the consideration of the activities that will be offered in the space. It isn't really like the formal artist-displaying-work model that is in evidence throughout the museum…the work is typically anonymous and individual pieces aren't highlighted.

When you walk into the space during the last month or so of an exhibition you experience the visitor-created artwork as a single, room-sized installation first, and only later do you focus on individual pieces. I think it is closer in some ways to the urge behind street art…the sort of private joy to be had from making something great and then leaving it behind for others to discover. I sometimes see visitors coming back to find something that they left behind a month or two before, not to reclaim it, just to see where it is now.

From the post 'Show your work' at the Indianapolis Museum of Art Blog.

It's talking about work created during physical visits to a museum rather than virtual visits but I think I noticed it because there's been a spurt of discussion about user-generated content and visitor participation on museum websites on the MCG mailing list following a workshop at the London Museums Hub on 'Understanding collections use and online access across the London Hub'.

A study presented at the event found an apparent lack of interest from museum website visitors in user-generated content, but in discussion at the event it appeared that the findings might have been different if the questions had been asked differently (with examples of some possible outcomes from UGC, perhaps 'would you like museum collections to be more discoverable because other visitors had tagged them with everyday words you'd use' rather than 'would you like opportunities to comment or upload content'), or if the focus groups hadn't been recruited from people who were physically visiting a museum and who were therefore fairly traditional museum-goers.

This lead to some interesting discussion about the differing reactions to the idea of opinions from other visitors versus real-life stories from other visitors; and of the idea that sometimes the value in user-created content lives with the person who contributes rather than those who read their contributions lately. This last idea was also raised at the User-generated content session at Museums and the Web in Montreal, my notes are here. I think the role of authority and trust and the influence of the context (type of museum or collection, user goal) need to be teased out into a more sophisticated model for analysing user-generated content in the cultural heritage sector.

There's a lot of research into user-generated content, participation and social software going on in the UK at the moment, it'd be great if there was somewhere that results, and ideally the raw data too, could be shared. Perhaps the MCG site?

Notes from 'National Collections Online Feasibility Study' at MCG's Spring Conference

These are my notes from Bridget McKenzie's presentation, 'National Collections Online Feasibility Study' at the MCG Spring meeting. Bridget's slides are online: National Collections Online Feasibility Study'. There's some background to my notes about the conference in a previous post. Any of my comments are in [square brackets] below.

The partners in the National Collections Online Feasibility Study are the National Museum Director's Conference, the V&A, the National Museum of Science and Industry, the National Maritime Museum, and Culture 24 (aka the 24 Hour Museum).

The brief:
Is it possible to create a discovery facility that integrates national museum collections; provides seamless access to item-level collections; a base on which build learning resources and creative tools? And can the nationals collaborate successfully?

The enquiry:
What's the scope? What's useful to different partners? What can be learnt from past and current projects? How can it help people explore collections? How can it be delivered?

There's a workshop on May 9th, with some places left, and another on June 18th; reports at the end of May and July.

Community of enquiry… people from lots of different places.

What are they saying?
"Oh no, not another portal!"
"You need to go to where the eyeballs are" – they're at Google and social networking sites, not at portals (but maybe at a few museum brands too).

It has to be understood in the context of why people visit museums. We don't know enough about how people use (or want to use) cultural collections online.

There's some worry about collaborative projects taking visits from individual sites. [Insert usual shtick about the need to the online metrics for museums to change from raw numbers to something like engagement or reach, because this is an institutional concern that won't go away.]

"Don't reinvent the wheel, see how other projects shape up": there's a long list of other projects on slide 9!

It's still a job to understand the options, to think about they can be influenced and interoperate.

"We have to build the foundations first"
Needs: audience research – is there a market need for integrated collections?; establish clarity on copyright [yes!]; agreement on data standards; organisational change – communicate possibilities, web expertise within museums; focus on digitising stuff and getting it out there.

[re: the audience – my hunch is that most 'normal' people are 'museum agnostic' when they're looking for 'stuff' (and I mean 'stuff', not 'collections') – they just want to find 18th century pictures of dogs, or Charles and Di wedding memorabilia; this is different to someone looking for a 'branded' narrative, event or curated experience with a particular museum.]

"Let's just do small stuff"
Need to enable experiment, follow the Powerhouse example; create a sandbox; try multiple approaches – microformats, APIs, etc. [Woo!]

Does a critical mass of experimentation mean chaos or would answers emerge from it?

What does this mean?
Lots of options; questions about leadership; use the foundations already there – don't build something big; need an market- or audience-led approach; sector leadership need to value and understand emerging technology.

Museum technology project repository launched

MCN have announced the launch of MuseTech Central, a project registry where museum technologies can 'share information about technology-related museum projects'. It sounds like a fabulous way to connect people and share the knowledge gained during project planning and implementations processes, hopefully saving other museum geeks some resources (and grey hairs) along the way.

I'd love to see something like that for user evaluation reports, so that institutions with similar audiences or collections could compare the results of different approaches, or organisations with limited resources could learn from previous projects.

New models for measuring the effectiveness of websites

From an article called Evidence-based website management, some thoughts from Gerry McGovern on measuring the effectiveness of websites:

We can, with increasing precision, know what content gets people to act and what content doesn't. The length of time people spend on the page is just a basic measure. Here are some others:

  • How many links have there been to the content. This is the ultimate measure as the link is the gold standard of the Web.
  • Where did the customer go once they read the content? Did they have a positive or negative reaction?
  • How has the content been rated by customers?
  • Has the content been blogged about? Did it get a conversation going?

New culture secretary says era of "targets" is over

I wonder what effect this will have on our online metrics and evaluation.

James Purnell, 37, the newly-appointed secretary of State for Culture, made his first substantive speech calling for a change in statistical "targets" in the arts: "Targets were probably necessary in 1997 [when Labour came to power], to force a change of direction in some parts of the arts world. But now, we risk idolising them." He has appointed Sir Brian McMaster, a member of Arts Council England, to advise on "how we can remove crude targets".

Article from The Art Newspaper