Get thee to a wiki – the great API challenge in action

Help us work on an informal, lightweight way of devising shared data, API standards for museum and cultural heritage organisations – museum-api.pbwiki.com is open for business.

You could provide examples of APIs you've used or produced, share your experience as a consumer of web services, tell us about your collections.

Commenting on other people's queries and content is an easy way to get started.  I'd particularly love to hear from curators and collections managers – we should be working together to enable greater access to collections.  If you check it out and none of it makes any sense – be brave and say so!  We should be able to explain what we're doing clearly, or we're not doing it right.

Some background: as announced on the nascent museumdev blog, the Science Museum is looking at releasing an API soon – it'll be project-specific to start with, but we're creating it with the intention of using that as an iterative testing and learning process to design an API for wider use. We could re-invent the wheel, but we'd rather make it easy for people to use what they've learnt using other APIs and other museum collections – the easiest way to do that is to work with other museums and developers. The Science Museum's initial public-facing collections API will be used for a 'mashup competition' based on object metadata from our 'cosmos and culture' gallery.

Speaking of museumdev, I started it as somewhere where I could ask questions, point people to discussions, a home for collections of links and stuff in development.  It's also got random technical bits like 'Tip of the Day: saving web.config as Unicode' because I figure I might as well share my mistakes^H^H^H^H^H^H^H^H learning experiences in the hope that someone, somewhere, benefits.

'The 50 Most Important Women in Science'

More inspiration for Ada Lovelace Day 2009 from Discover magazine's 2002 list of The 50 Most Important Women in Science. Not everyone listed is directly involved with technology, but it's worth checking them out anyway, because as the article points out, '[i]f just one of these women had gotten fed up and quit—as many do—the history of science would have been impoverished':

Three percent of tenured professors of physics in this country are women. Nonetheless, a woman physicist stopped light in her lab at Harvard. Another woman runs the linear accelerator at Stanford. A woman discovered the first evidence for dark matter. A woman found the top quark. The list doesn't stop there, but the point is clear.

Three years ago, Discover started a project to look into the question of how women fare in science. We knew there were large numbers of female researchers doing remarkable work, and we asked associate editor Kathy A. Svitil to talk to them. The result of her investigation is a selection of 50 of the most extraordinary women across all the sciences. Their achievements are detailed in the pages that follow.

To read their stories is to understand how important it is that the barriers facing women in science be broken down as quickly and entirely as possible. If just one of these women had gotten fed up and quit—as many do—the history of science would have been impoverished. Even the women who have stuck with it, even those who have succeeded spectacularly, still report that being a woman in this intensely male world is, at best, challenging and, at worst, downright disheartening.

It will take goodwill and hard work to make science a good choice for a woman, but it is an effort at which we cannot afford to fail. The next Einstein or the next Pasteur may be alive right now—and she might be thinking it's not worth the hassle.

An easy candidate for Ada Lovelace Day – Barbara Liskov, winner of the 2008 Turing Award

How cool is she? ACM Turing Award Goes to Creator of Influential Innovations in Computer Software Design

NEW YORK, March 10, 2009 – ACM, the Association for Computing Machinery, has named Barbara Liskov of the Massachusetts Institute of Technology (MIT) the winner of the 2008 ACM A.M. Turing Award.  The award cites Liskov for her foundational innovations to designing and building the pervasive computer system designs that power daily life.  Her achievements in programming language design have made software more reliable and easier to maintain.  They are now the basis of every important programming language since 1975, including Ada, C++, Java, and C#.  The Turing Award, widely considered the "Nobel Prize in Computing," is named for the British mathematician Alan M. Turing.  The award carries a $250,000 prize, with financial support provided by Intel Corporation and Google Inc.

The first woman to be awarded a Ph.D. from a Computer Science department (in 1968 from Stanford University), Liskov revolutionized the programming field with groundbreaking research that underpins virtually every modern computer application for both consumers and businesses. Her contributions have led to fundamental changes in building the computer software programs that form the infrastructure of our information-based society. Her legacy has made software systems more accessible, reliable, and secure 24/7.

Spotting QR tags in the real world

One of the prototypes made for dev8D has been adapted so it can 'splash a big QR code onto the screen' so people can conferences can take a shot of it and click straight through to the URL – no typing.  Super cool!

I'm excited by Semapedia, a project "which uses QR Code nodes to connect Wikipedia articles with their relevant place in physical space". You can browse locations that have been tagged on a map or on Flickr. I get excited by things like this because it makes 'outside the walls of the museum' projects seem much more feasible.

The ZKM (Centre for Art and Media, Karlsruhe) are exploring mobile tagging for their 20th anniversay: "[w]ith this new tag solution, you can communicate with the museum and use it as a platform also outside of opening hours, i.e., not bound to a certain time, and without being physically present in the museum, i.e., not bound to a certain place." The site is in German so it's difficult to work out exactly what you get online. Thanks to Jennifer Trant for the tip-off.

Two notes on QR tags out in the wild – Seb's recent post linked to 'a guerilla art installation at [Melbourne's] Federation Square' which is ace on so many levels.  I love their ethos.

The image is a photo I took today – a band have put up a QR tag outside a London tube station.  It takes you to a page that links to a downloadable track and their iTunes and MySpace pages.

Tragically, I've even started using QR codes in the office – I often use my phone to test sites outside our network, and I've printed out a sheet of QR codes for sites I check often, to save typing in URLs on my phone keyboard.

Agile development presentation, dev8D

These are my very rough notes on Grahame Klyne's talk on Agile Development as JISC's dev8D event in February. Grahame works with bioinformatics and the semantic web at the zoology department of Oxford University. He is interested in how people can make useful things out of semantic web technologies.

Any mistakes are mine, obviously, and any comments or corrections are welcome. (I should warn that they're still rough, even though it's taken me a month to get them online.) My notes in [square brackets] below.

He started by asking people in the room to introduce themselves, which was a nice idea as most people hadn't met.

Agile and their project
Agile seemed to be particularly appropriate for development in a research context, where the final outcomes are necessarily uncertain. [I'm particularly interested in how they managed to build Agile development into the JISC project proposal, as reflected in 'A call for agile museum projects (a lunchtime manifesto)'. Even getting agile working in a university environment seems like an impressive achievement to me, though universities tend to be more up-to-date with development models than museums.]

They had real users, and wanted to apply semantic web technologies to help them do their research.

At the start of the project, all the developers went on week-long training course on agile development, which was really important for the smooth running of the project as they all came out with a common view on how they might go forward. Everyone worked with 'how can we make this work best for us' view of agile development.

Agile – what's it about?
Agile values individuals and interactions over processes and tools. It values responding to change over following a plan – this is particularly interesting when writing proposals for funders like JISC.

From a personal perspective, the key principles became: what we do is (end) user led; spend a lot of time communicating; build on existing working system code (i.e. value code over documentation); develop incrementally. It's not in all the textbooks but they found it's important – they 'retrospect'.

User-led: you need a real user as a starting point. Not sure how to advise someone working on a project without real users [I didn't ask, but why would you be doing a project without a real users?]. It's far easier to elicit requirements from user if you have a working system.

Build and maintain trust in the team. [Important.]

Building on working code: start with something simple that works. Build in small steps – it's easy to say, but the discipline of forcing yourself to take tiny steps is quite hard. The temptation is to cram in a bit more functionality and think you'll sort it out later. When you get used to the discipline of small steps, it gets so much easier to maintain flow of development.

Minimise periods of time when you don't have working system.

Don't sacrifice quality.

Always look for easy ways of implementing what you need to do now. Don't bring in more complex solution because you think you might need it later.

Retrospection – 'the one indispensable practice of agile'? As a team, take time regularly to review and adjust.

More random points: Test-lead development is often assoc with agile development. Think of test cases as specification – it's also a useful mindset for standards development groups. Test cases are particularly useful when working with external code bases or applications – even more so than when working with your own code. [There was quite a bit of discussion about this, I think whether it made sense to you depended on your experiences commissioning or reviewing possible applications for purchase for institutional use. I can think of occasions when it would have been a very useful approach for dealing with vendor oversell – it sounds like a sensible way to test the fit of third-party applications for your situation.]

Keep refactoring separate from the addition of new functionality.

Card planning: for e.g. user stories, tasks. It's a good solution in an environment with very strong characters, it allows everyone to be confident that their input was being noted, which means they don't hijack the session to make sure their points have been heard… the team can then review and decide which are most important in next small block of work.

Outcomes: progress had been steady and continuous rather than meteoric. What seems like a slow pace at the time actually gets you quite far. It produces a sense of continuous progress.

Unexpected architectural choices – had particular view about how web interactions were going to work in the project, e.g. choice between server side or client side JavaScript – he was sceptical, but knew he could change his mind later, could follow one path and change if necessary. But actually resulted in architectural choices that would never have made upfront but which were best for the situation.

Discussion
Never refactor until you have to. Don't make stuff you *might* need, make it when you need it.

A call for agile museum projects (a lunchtime manifesto)

Yet another conversation on twitter about the NMOLP/Creative Spaces project lead to a discussion of the long lead times for digital projects in the cultural heritage sector. I've worked on projects that were specced and goals agreed with funders five years before delivery, and two years before any technical or user-focussed specification or development began, and I wouldn't be surprised if something similar happened with NMOLP.

Five years is obviously a *very* long time in internet time, though it's a blink of an eye for a museum. So how do we work with that institutional reality? We need to figure out agile, adaptable project structures that funders and bid writers can also understand and buy into…

The initial project bid must be written to allow for implementation decisions that take into account the current context, and ideally a major goal of the bid writing process should be finding points where existing infrastructure could be re-used. The first step for any new project should be a proper study of the needs of current and potential users in the context of the stated goals of the project. All schema, infrastructure and interface design decisions should have a link to one or more of those goals. Projects should built around usability goals, not object counts or interface milestones set in stone three years earlier.

Taking institutional parameters into account is of course necessary, but letting them drive the decision making process leads to sub-optimal projects, so projects should have the ability to point out where institutional constraints are a risk for the project. Constraints might be cultural, technical, political or collections-related – we're good at talking about the technical and resourcing constraints, but while we all acknowledge the cultural and political constraints it often happens behind closed doors and usually not in a way that explicitly helps the project succeed.

And since this is my lunchtime dream world, I would like plain old digitisation to be considered sexy without the need to promise funders more infrastructure they can show their grandkids.

We also need to work out project models that will get buy-in from contractors and 3rd party suppliers. As Dan Zambonini said, "Usability goals' sounds like an incredibly difficult thing to quantify' so existing models like Agile/sprint-esque 'user stories' might be easier to manage.

We, as developers, need to create a culture in which 'failing intelligently' is rewarded. I think most of us believe in 'failing faster to succeed sooner', at least to some extent, but we need to think carefully about the discourse around public discussions of project weaknesses or failures if we want this to be a reality. My notes from Clay Shirky's ICA talk earlier this year say that the members of the Invisible College (a society which aimed to 'acquire knowledge through experimental investigation') "went after alchemists for failing to be informative when they were wrong" – " it was ok to be wrong but they wanted them to think about and share what went wrong". They had ideas about how results should be written up and shared for maximum benefit. I think we should too.

I think the MCG and Collections Trust could both have a role to play in advocating more agile models to those who write and fund project bids. Each museum also has a responsibility to make sure projects it puts forward (whether singly or in a partnership) have been reality checked by its own web or digital specialists as well as other consultants, but we should also look to projects and developers (regardless of sector) that have managed to figure out agile project structures that funders and bid writers can also understand and buy into.

So – a blatant call for self-promotion – if you've worked on projects that could provide a good example, written about your dream project structures, know people or projects that'd make a good case study – let me know in the comments.

Thanks also to Mike, Giv and Mike, Daniel Evans (and the MCG plus people who listened to me rant at dev8D in general) for the conversations that triggered this post.


If you liked this post, you may also be interested in Confluence on digital channels; technologists and organisational change? (29 September 2012) and Museums and iterative agility: do your ideas get oxygen? (21 November 2010).

Wellcome Library blog – a quick review

I originally wrote this a while ago, and a whole bunch of new content has been added by what seems like a range of authors, so it's worth checking out.

The Wellcome Library has a quite lovely blog. I like their 'item of the month', the way they're addressing common questions 'where do things come from', the list of latest aquisitions (though it's about as human readable as I feared it might be), a 'call for testing' when they've got newly digitised records up… it's a good example of transparency and the provision of access in practice. It feels a little as if you had a friend who worked there who sent on little tidbits they came across during the work. 

The site says it has (the uber-annoying) Snap Shots but it didn't seem to actually be interfering with my browsing experience when I checked it out today.

There's a Flickr stream too (though they haven't yet nabbed a name, so it's at the not-so-snappy http://www.flickr.com/photos/26127598@N04/) and it hasn't been updated since what looks like a big batch upload in May 2008. Some of the images are lovely, check out the human cancer cells, or neurons in the brain or historically important – such as the first DNA fingerprint.

They've just added Charles Babbage, I wonder if they have anything on Ada Lovelace they could highlight for Ada Lovelace Day.

Institutions, authority, community and social media

A very interesting example from the library sector – the CEO of CILIP (the Chartered Institute of Library and Information Professionals) posted about twitter, and caused a minor outbreak of fury. A little while later he posted what I think is an honest reflection on and acknowledgement of the issues raised, and of the changes institutions face in the era of social media – 'social networking [is] changing the dynamic of institutionalised professionalism'. It's also a good demonstration of the idea that making mistakes in public doesn't mean the end of the world, and might even cause positive changes.

In a post titled, Yes, let's try that! the CEO responds to criticisms of his original post (below):

I went on to make an observation (that there's a widening gap between the culture of the institution and the culture of the network) and ask a question: How can we best combine the authority of our Institute and the democracy of our network?

CILIP (like many organisations) is conflicted between authority and community – or (to put it in a way which chimes more with this discussion) between systems and conversations.

So let me try to explain my thinking – and show why I think the discussion about using social media is also a discussion about the future for professionalism.

We can't simply (as some comments have suggested) ignore the issue of authority. After all, we're a profession which prides itself on authenticating information as well as providing access to information – "authority control" is a skill we practice. And any profession worthy of the name has to have systems in place to authenticate and accredit professional practice. The problem (and that sense of frustration and irritation) arises when an organisation's systems and a community's conversations get out of kilter with each other – when the gap appears to widen between the organisation and the community, between the institute and the network, between "us" and "them".

For context, the original post, All of a twitter, that kicked off the debate started:

There's some twittering at present about whether CILIP has (or should have) any "official" presence on various lists or micro blog sites.

The simple answer, of course, is no. In terms of "official" activity, cyber life is just like real like – if it happens in a CILIP-sanctioned space, it's official; if it happens down the pub or in someone else's space, it isn't.

It's interesting seeing how the library sector is grappling with these issues, particularly in a week when the 'creative spaces' beta launch has caused such a stir.

Our dev8D 'Lazy Lecturer' prototype

In the interests of transparency, I thought I'd put the submission for the 'Lazy Lecturer' prototype I worked on with Ian Ibbotson and Pete Sefton for the JISC dev8D 'developer decathlon' online.

I really should blog more about the event – both lessons I learnt from the content and event structure, but also the experience of being surrounded by actual (higher education, mostly open source/LAMP) geeks. But hey, this will do in the meantime.

Happy developers + happy museums = happy punters (my JISC dev8D talk)

This is a rough transcript of my lightning talk 'Happy developers, happy museums' at JISC's dev8D 'developer happiness' days last week. The slides are downloadable or embedded below. The reason I'm posting this is because I'd still love to hear comments, ideas, suggestions, particularly from developers outside the museum sector – there's a contact form on my website, or leave a comment here.

"In this talk I want to show you where museums are in terms of data and hear from you on how we can be more useful.

If you're interested in updates I use my blog to [crap on a bit, ahem] talk about development at work, and also to call for comment on various ideas and prototypes. I'm interested in making the architecture and development process transparent, in being responsive to not only traditional museum visitors as end users, but also to developers. If you think of APIs as a UI for developers, we want ours to be both usable and useful.

I really like museums, I've worked in three museums (or families of museums) now over ten years. I think they can do really good things. Museums should be about delight, serendipity and answers that provoke more questions.

A recent book, 'How does one become a scientist? : survey on the birth of a Vocation' states that '60% of scientists over 30 and 40% of scientists under 30 note claim, without prompting, that the Palais de la Découverte [a science museum in Paris] triggered their vocation'.

Museums can really have an impact on how people think about the world, how they think about the possibilities of their lives. I think museums also have a big responsibility – we should be curating collections for current and future audiences, but also trying to provide access to the collections that aren't on display. We should be committed to accessibility, transparency, curation, respecting and enabling expertise.

So today I'm here because we want to share our stuff – we are already – but we want to share better.

We do a lot of audience research and know a lot about some of our users, including our specialist users, but we don't know so much about how people might use our data, it's a relatively new thing for us. We're used to saying 'here are objects in a case, interpretation in label', we're not used to saying 'here's unmediated access, access through the back door'.

Some of the challenges for museums: technology isn't that much of a challenge for us on the whole, except that there are pockets of excellence, people doing amazing things on small budgets with limited resources, but there are also a lot of old-fashioned monolithic project designs with big overheads that take a long time to deliver. Lots of people mean well but don't know what's possible – I want to spread the news about lightweight, more manageable and responsive ways of developing things that make sense and deliver results.

We have a lot of data, but a lot of it's crap. Some of what we have is wrong. Some of it was written 100 years ago, so it doesn't match how we'd describe things now.

We face big institutional challenges. Some curators – (though it does depend on the museum) – fear loss of control, fear intellectual vandalism, that mistakes in user-generated content published on museum sites will cause people to lose trust in museums. We have fears of getting the IT wrong (because for a while we did). Funding and metrics are a big issue – we are paid by how many people come through our door or come to our websites. If we're doing a mashup, how do we measure the usage of that? Are we going to cost our organisations money if we can't measure visits and charge back to the government? [This is particularly an issue for free museums in the UK, an interesting by-product of funding structures.]

Copyright is a huge issue. We might not even own an object that appears in our collections, we might not own the rights to the image of our object, or to the reproductions of an image. We might not have asked for copyright clearance at the time when an object was donated, and the cost of tracing it might be too high, so we can't use that object online. Until we come up with a reliable model that reduces the risk to an institution of saying 'copyright unknown', we're stuck.

The following are some ways I can think of for dealing with these challenges…
Limited resources – we can't build an interface to meet every need for every user, but we can provide the content that they'd use. Some of the semantic web talks here have discussed a 'thin layer' of application over data, and that's kind of where we want to go as well.

Real examples to reduce institutional fear and to provide real examples of working agile projects. [I didn't mean strictly 'agile' methodology but generally projects that deliver early and often and can respond to the changing technical and social environment]

Finding ways for the sector to reward intelligent failure. Some museums will never ever admit to making a mistake. I've heard over the past few days that universities can be the same. Projects that are hyped up suddenly aren't mentioned, and presumably it's failed, but no-one [from the project] ever talks about why so we don't learn from those mistakes. 'Fail faster, succeed sooner'.
I'd like to hear suggestions from you on how we could deal with those challenges.

What are museums known for? Big buildings, full of stuff; experts; we make visitors come to us; we're known for being fun; or for being boring.

Museum websites traditionally appear to be about where we are, when we're open, what's on, is there a cafe on site. Which is useful, but we can do a lot more.

Traditionally we've done pretty exhibition microsites, which are nice – they provide an experience of the exhibition before or after your visit. They're quite marketing-led, they don't necessarily provide an equivalent experience and they don't really let you engage with the content beyond the fact that you're viewing it.

We're doing lots of collections online projects, some of these have ended up being silos – sometimes to the extent if we want to get data out of them, we have to screen-scrape our own data. These sites often aren't as pretty, they don't always have the same design and usability budgets (if any).

I think we should stick to what we're really good at – understanding the data (collections), understanding how to mediate it, how to interpret it, how to select things that are appropriate for publication, and maybe open it up to other people to do the shiny pretty things. [Sounds almost like I'm advocating doing myself out of a job!]

So we have lots of objects, images, lots of metadata; our collections databases also include people, events, dates, places, businesses and organisations, lots of qualified information around things like dates, they're not necessarily simple fields but that means they can convey a lot more meaning. I've included that because people don't always realise we have information beyond objects and object metadata. This slide [11 below] is an example of one of the challenges – this box of objects might not be catalogued as individual instruments, it might just be catalogued as a 'box of stuff', which doesn't help you find the interesting objects in the box. Lots of good stuff is hidden in this way.

We're slowly getting there. We're opening up access. We're using APIs internally to share data between gallery interactives and the web, we're releasing them as data points, we're using them to provide direct access to collections. At the moment it still tends to be quite mediated access, so you're getting a lot of interpretation and a fewer number of objects because of the resources required to create really nice records and the information around them.

'Read access' is relatively easy, 'write access' is harder because that's when we hit those institutional issues around authority, authorship. Some curators are vaguely horrified that they might have to listen to what the public have to say and actually take some of it back into their collections databases. But they also have to understand that they can't know everything about their collections, and there are some specialist users who will know everything there is to know about a particular widget on a particular kind of train. We'd like to capture that knowledge. [London Transport Museum have had a good go at that.]

Some random URLs of cool stuff happening in museums [http://dashboard.imamuseum.org/, http://www.powerhousemuseum.com/collection/database/menu.php, http://www.brooklynmuseum.org/opencollection/collections/, http://objectwiki.sciencemuseum.org.uk/] – it's still very much in small pockets, it's still difficult for museum staff to convince people to take what seems like a leap of faith and try these non-traditional things out.

We're taking our content to where people hang out. We're exploring things like Flickr Commons, asking people to tag and comment. Some museums have been updating collections records with information added by the public as a result. People are geo-tagging photos for us, which means you can do 'then and now' mashups without a big metadata enhancement budget.

I'd like to see an end to silos. We are kinda getting there but there's not a serious commitment to the idea that we need to let things go, that we need to make sure that collections online shareable, that they're interoperable, that they can mesh with other things.

Particularly for an education audience, we want to help researchers help themselves, to help developers help others. What else do we have that people might find useful?

What we can do depends on who you are. I could hope that things like enquiry-based learning, mashups, linked data, semantic web technologies, cross-collections searches, faceted browsing to make complex searches easy would be useful, that the concept of museums as a place where information lives – a happy home for metadata mapped around objects and authority records – are useful for people here but I wouldn't want to put words into your mouths.

There's a lot we can do with the technology, but if we're investing resources we need to make sure that they're useful. I can try things in my own time because it's fun, but if we're going to spend limited resources on interfaces for developers then we need to that it's actually going to help some group of people out there.

The philosophy that I'm working with is 'we've got really cool things, but we can have even cooler things if we can share what we have with everyone else'. "The coolest thing to do with your data will be thought of by someone else". [This quote turns out to be on the event t-shirts, via CRIG!] So that said… any ideas, comments, suggestions?"

And that, thankfully, is where I stopped blathering on. I'll summarise the discussion and post back when I've checked that people are ok with me blogging their comments.

[If the slide show below has a brown face on a black background, it's the right one – slideshare's embed seems to have had a hiccup. If it's not that, try viewing it online directly.]

[My slide images include the Easter Egg museum in Kolomyya, Ukraine and 'Laughter in Odd Places' event at the Museum of London.]

This is a quick dump of some of the text from an interview I did at the event, cos I managed to cover some stuff I didn't quite articulate in my talk:

[On challenges for museums:] We need to change institutional priorities to acknowledge the size of the online audience and the different levels of engagement that are possible with the online experience. Having talked to people here, museums also need to do a bit of a sell job in letting people know that we've changed and we're not just great big imposing buildings full of stuff.

[What are the most exciting developments in the museum sector, online?] For digital collections, going outside the walls of the museum using geo-location to place objects in their original context is amazing. It means you can overlay the streets of the city with past events and lives. Outsourcing curation and negotiating new models of expertise is exciting. Overcoming the fear of the digital surrogate as a competitor for museum visits and understanding that everything we do builds audiences, whether digital or physical.