Blackout Testing

Yesterday we performed our second round of blackout testing. Now we can’t take any credit for coming up with the idea. Twitter carried out their first blackout test in March 2013 to help application developers understand the impact the retirement would have on their applications a their users.

Meanwhile back in the Mendeley Ranch we were scratching our heads and wondering how to gracefully transition multiple clients off of our OAPI onto the new API so we decided to follow suit.

Mendeley is currently transitioning all of its existing applications onto our new API. Our desktop was using an XML/RPC API which was the first incarnation of an API in Mendeley. It was a beginning and worked well enough to get users, and to raise funding.

Then the Mendeley Open API (often referred to as the OAPI) was written with the idea of opening up data to third party developers and in fitting with the Mendeley ethos, of making research more accessible. Of course we used it internally to build some of our own applications such as the iOS client.

Now the desktop client has been released using the new API it is time to turn our attention to retiring the OAPI. One API down, one to go. This time the migration is more difficult. For example, last night we had over 132 active clients of which a large proportion are still using the OAPI.

Why do we need to retire the OAPI? This warrants a blog post of it’s own but we have a creaking MySQL database that is struggling to scale how we would like it. Retiring the OAPI is a big piece of this jigsaw.  That’s all I’ll say for now.

We initially carried out our first blackout test back in May when we were moving from OAuth1 to OAuth2.  At the time this involved configuring our proxy to return a HTTP 410 Gone for all requests to OAuth1 for a period of 1 hour.  A similar exercise was performed yesterday for all calls to the OAPI. While we endeavor to email all of our clients with sufficient notice, there is no guarantee that the end developer receives the email or let alone reads it. As my colleague Callum Anderson stated in the linked blogged post, these tests act as a call to action for some folks that may have missed our announcements.

So what have we learned from our tests?

Well the first round of blackout testing highlighted that our iOS client has a rogue call to an OAuth1 endpoint that we didn’t know about. Also,  yesterday we found out that one of our monitoring tools, Sensu, wasn’t working as expected.

No great surprises from our point of view but still some lessons learned. As yet I have to hear from some of our third party developers on how they coped with the blackout tests. I’ll report back on any interesting lessons if/should I hear about them.

On a final note, we created a Mendeley Blackout Test playlist on Spotify. Can you spot the theme?

This entry was posted in blackout tests and tagged on by .

About Joyce-Stack - Developer Outreach

Joyce Stack completed a BSc. (Hons) Computer Science from The Open University while working in a startup in Co. Cork, Ireland. She moved to London in 2005 to join one of the City’s leading exponents of agile techniques at that time to work as a Java Developer. She is now working in the Developer Outreach role in Mendeley where her responsibilities are to meet and educate developers about the Mendeley API, attend conferences, meetups and hackathons and basically be the face of the API. She is passionate about providing an excellent developer experience and APIs. She likes biking, swimming and yoga but she hates potatoes despite being Irish.