Mobile TestingEstablishing An Open Device Lab

Advertisement

Managing a personal device lab can be quite hard with an ever expanding number of devices. It’s not only expensive, but also bad for our environment. Think of a situation where every Web developer would purchase a large pile of gadgets and keep adding new ones as they are launched — this wouldn’t make much sense. Thankfully, there are better ways to handle the problem.

During the spring of 2012, Jeremy Keith wrote on his website1 that anybody is welcome to visit their device lab2 at Clearleft’s office in Brighton, UK and use their devices. What they didn’t expect, though, was that in response many local developers offered to add their own devices to the collection as well. Two weeks later Jeremy Keith and Remy Sharp also presented this idea at the Mobilism conference in Amsterdam and the people attending loved it. A few months after Mobilism, similar labs started popping up in places like Amsterdam, Berlin, London and Malmö.

Following this example, we decided to bring our own devices to the Kisko Labs3 office and do the same thing in Helsinki. We now have an open device lab4 where anyone can come, start using the devices and contribute by lending their old devices. Three weeks after establishing this we already have three new devices from the local community: a Nokia N900 running Maemo, a Nokia Lumia 800 running Windows Mobile and an HTC Desire HD running Android. I have also been in contact with the device manufacturers and so far at least Nokia5, Palm6 and Sony7 have promised to help us out with the lab.

Helsinki Open Device Lab8
Helsinki Open Device Lab, devicelab.fi9.

I encourage everyone to do the same thing in the city they live in as well. This article will cover most of the things needed to be considered when doing that. It will also work as a guide and give practical tips — things like location, how to get devices, what devices to get and what software to use. At the end of the article is also a list of all the current open device labs around the world.

Location

This is probably the hardest part as it should be relatively open, easily accessible, but also a safe location to prevent burglars. I wouldn’t be too worried though as that’s a risk you need to take with any office space that’s filled with desktop computers and other gadgets. Just make sure that the space has proper locks and an alarm system in place.

Mozilla’s workspace in London
Mozilla’s workspace in London. Image by @mozillaeu10.

Finding a location was never a problem for us as we had a space at the office which wasn’t used that often, but I understand that it might be hard to find one unless you already work in a location like that. However, I imagine that the best places to start asking are local Web community meet-ups, conferences and Twitter. Find out about local co-working spaces too, as those could be the best candidates to host these kind of projects. I also asked Shaun Dunne11 (who established the London Device Lab12) if he had any advice on how to find a space, and this is what he answered:

“I was in the Reasons to be Appy13 conference during April when Christian Heilmann14 mentioned Mozilla’s new London office and how it had a co-op space with free WiFi for anyone to use. I spoke to him after his talk and mentioned what I was trying to do and he said that he thought Mozilla would be more than happy to host it. He said I should speak to @cyberdees15 who runs things in the office. We exchanged a few emails and I went there a couple of times and it was born from there.”

Devices

You don’t need a huge collection of devices to establish a lab — it can initially be small and grow once other developers start contributing their devices. Some device manufacturers are also willing to help the community by providing test devices for initiatives like this, so you should definitely contact them too.

There are basically four main areas that need to be covered in the lab. These are: feature phones, smartphones with low level support, smartphones and tablets. Later on you might also want to consider adding a Smart TV and other more exotic devices like game consoles16. Additionally, the lab should also efficiently cover various screen sizes17 between 240 x 320 and 1280 x 800 pixels, as well as some high-DPI variants.

PhoneGap’s Device Wall18
PhoneGap’s Device Wall19. Photo © 2012 Adobe Systems Inc.

David Blooman20 from BBC recently shared21 the process that they use while testing on mobile devices, and the minimum set of devices to get the job done. This list is a slightly modified version of their minimum test device stack. For now it is a good starting point for anyone who is thinking about setting up an open device lab:

  • iOS 5 — iPhone 4
  • iOS 6 — iPad 3 Retina
  • Android 2.2 — HTC desire
  • Android 2.3 — Huawei U8650
  • Android 2.3 — Kindle Fire (Silk browser)
  • Android 3.X — Motorola Xoom
  • Android 4.X — Samsung Galaxy Tab 2
  • Blackberry OS 5 — Curve 8900
  • Blackberry OS 6 — Bold 9700
  • Windows Phone 7.5 — Lumia 800
  • Symbian S40 — Nokia 2700
  • Symbian S60 — Nokia N95
  • Symbian Belle — Nokia 500

Remember: You will want to end up with a collection that represents the audience and overall market share in your own location, which might be different from what I have listed here. Stephanie Rieger, who is a co-owner of Yiibu22, has written an excellent article explaining Strategies for choosing test devices4923 (so be sure to read that to find out more about the subject). Dave Olsen has also written an article about how to build a device lab24, where he explains how and why they decided to get certain devices.

“If I had to start from nothing, I would start with the phone in my pocket. After that, I would take the usual suspects — Android 2.3, iOS5, etc., and make sure to have the more popular phones in place, but not go overboard. One of each to begin with, and then more varieties as time goes on. In a good way, everyone’s device lab should be different, as every market is going to have variations. There is no golden list of devices.”

— David Blooman

How To Contact Device Manufacturers

  • You will need to have a space and a website (a single-page website is fine) for the lab before asking for any devices. Otherwise, it may be hard to look convincing.
  • Twitter and email seem to be the best way. Look for developer relations accounts like this25 from Twitter and send emails to their developer related addresses. You can find the addresses from the developer websites, which usually reside in an URL formed like this: developer.manufacturer.com.
  • Ask people on Twitter if they know someone who works at one of the companies you are trying to contact. It may be an easier way to begin communicating with the right people.
  • When sending emails, explain carefully what the project is about, what you need and why you need it. It’s good to keep it short and get straight to the point.
  • Remember that you are sending emails to other human beings, not to some random corporations.
  • If you don’t get any answer from them within two weeks try to contact another person from that manufacturer.
  • Last but not least: Don’t be afraid to ask for help. I contacted several people about their test devices and where they got them from. Shaun Dunne (from the London Device Lab) was also kind enough to provide his perspective on how to contact the manufacturers:

“I started hitting the developer relations people up on Twitter. BB and Palm got back to me via Twitter and an email conversation went on from there. Nokia had their developer relations person at Mobilism26, so I found out his email and sent him an email directly. It’s about being persistent, really, email is hard because it could end up in their spam or they can ignore it.”

“In a public forum like Twitter it’s harder for them not to engage. Don’t just go after the hardware manufacturers either, it’s worth speaking to the Android + WP7/8 people to see what they can do for you. If they want people to develop applications and websites that work on their devices, then it’s in their best interests to get those devices in developers’ hands. The best and easiest way to get the devices in their hands is through a community lab.”

Setting Up The Lab

You don’t need much in the beginning: a table, a few chargers and a Wi-Fi connection is all you need to get things up and running. If you have more than five devices, it may be a good idea to get a USB hub27 which can provide power to avoid cable clutter and stands28 where you can put the devices on. A few of the labs have also built their own stands and you can get some ideas from the resources listed below. Jeremy Keith also told me that they have all the devices running through a wall socket that’s on a timer which switches the power off in the evening and nighttime and back on again in the morning. That might be useful for saving some energy and also to keep the batteries healthy.

64 Digital’s device testing station29
64 Digital’s device testing station30. Photo © 2012 64Digital.

Later on, when there are many more devices in the lab, you may want to start considering getting a better wireless router which can handle all the devices. Andre Jay Meissner31 told me that Apple’s Airport Extreme can handle up to around 30 devices, but not much more (it claims to support 50!). SIM cards with data plans are also something which you might need once you start adding older devices that exclude Wi-Fi.

Software Tools To Get You Started

  • Adobe Shadow41:
    Probably one of the best tools for testing at the moment. It allows device pairing, synchronous browsing and remote inspection using Chrome extension42 on either Mac or Windows. To be able to use Adobe Shadow you will need to download and install the mobile client to all test devices. In addition, you will also need the Google Chrome extension to run Shadow on your laptop.

  • JS Bin43:
    JS Bin is a Web app specifically designed to help JavaScript and CSS folks test snippets of code, within some context, and debug the code collaboratively. You can use JS Bin together with the Adobe Shadow mentioned earlier.

  • Web Inspector Remote44:
    Weinre is a remote inspector tool for WebKit browsers. It has been included in the Adobe Shadow application, but you can also set up your own installation to be able to use it on platforms like WebOS and Blackberry (in addition to the iOS and Android platforms).

  • xip.io45
    xip.io is a domain name that provides wildcard DNS for any IP address. You can use these domains to access virtual hosts on your development Web server from devices on your local network (like iPads, iPhones, and other computers).

  • Showoff.io46, Localtunnel47, Proxylocal48:
    For sharing your localhost over the Web.

  • Mobile browsers:
    Remember to install various browsers like Opera Mobile, Opera Mini, Chrome and Firefox to all of your supported test devices.

“Be sure to track the OS versions found on your test devices, and think carefully each time you upgrade. Owning four BlackBerry devices with four different versions of the OS is infinitely more valuable than owning four with the same version.”

— “Strategies for choosing test devices4923” by Stephanie Rieger.

Adobe Shadow running on multiple devices
Adobe Shadow running on multiple devices. Photo © 2012 Adobe Systems Inc.

Maintenance

There are some running expenses — rent, Wi-Fi, personal time used — and you may initially need to spend a few hundred bucks to provide chargers, wires and stands for the devices. So it’s worth considering if a small monthly payment would be acceptable. As it also might not be possible to find a space which is completely open like the one in London50, it’s possible to have everything available by appointment too. This seems to be quite common practice and it allows you to use the same space for your own workshops as well, if needed.

In the beginning, when you are setting up the lab, I wouldn’t worry about all of this though. It’s possible that the lab will get popular and have lots of visitors and that someone might be using the devices when someone else comes in, but only time will tell. Shaun Dunne also said that they were discussing this very same problem in the beginning and decided finally that the lab should just be open. Jeremy Keith seems to think in a similar manner:

“When I started the device lab in Brighton, I didn’t worry about the paperwork. Instead of worrying about insurance, theft, liability and all those other worst-case scenarios, I decided to just do it and deal with the bad stuff if and when it arises. So far, so good.”

Closing Words

I believe in testing on real devices. Software emulators and simulators can be useful, but in the end they can only do that; simulate the experience (as Paul Robert Lloyd points out51). To make testing on real devices possible for everybody, we need open device labs. If your city doesn’t yet have such a lab, I would say go for it, establish one. Don’t worry about the amount of devices you start off with, you’ll be surprised about how much the community is willing to help.

Last but not least: just a few days ago, while writing this, Andre Jay Meissner contacted me about the possibility to set up LabUp!52, which would help people around the world in establishing nonprofit Open Device Labs. I think it’s a splendid idea and everyone who can help should join the movement53.

Open Device Labs Around The World

This list is by no means complete, but it should include all the labs which were established before this post was published. For the future you should bookmark the up-to-date list of all open device labs54 which Jay is collecting.

(jvb) (il)

Footnotes

  1. 1 http://adactio.com/journal/5433/
  2. 2 http://clearleft.com/testlab/
  3. 3 http://kiskolabs.com/
  4. 4 http://www.devicelab.fi
  5. 5 http://www.developer.nokia.com
  6. 6 http://www.hpwebos.com/us/
  7. 7 http://developer.sonymobile.com
  8. 8 http://www.devicelab.fi
  9. 9 http://www.devicelab.fi
  10. 10 http://www.flickr.com/photos/mozillaeu/
  11. 11 http://twitter.com/shaundunne/
  12. 12 http://shaundunne.github.com/deviceLibrary/
  13. 13 http://www.reasonstobeappy.com
  14. 14 http://christianheilmann.com
  15. 15 http://twitter.com/cyberdees
  16. 16 http://www.alistapart.com/articles/testing-websites-in-game-console-browsers/
  17. 17 http://viljamis.com/blog/2012/responsive-workflow/#device-diagram
  18. 18 http://phonegap.com/2012/03/29/phonegaps-new-device-wall/
  19. 19 http://phonegap.com/2012/03/29/phonegaps-new-device-wall/
  20. 20 http://twitter.com/dblooman
  21. 21 http://mobiletestingfordummies.tumblr.com/post/20056227958/testing
  22. 22 http://yiibu.com
  23. 23 http://stephanierieger.com/strategies-for-choosing-test-devices/
  24. 24 http://www.dmolsen.com/mobile-in-higher-ed/2012/06/26/how-to-build-a-device-lab-part-1/
  25. 25 http://twitter.com/shaundunne/oem-relations/members
  26. 26 http://mobilism.nl
  27. 27 http://www.amazon.co.uk/Trust-Port-Usb-2-0-Power/dp/B001UE6OC8
  28. 28 http://www.theironmill.co.uk
  29. 29 http://www.64digital.co.uk/blog/66/the-64-digital-super-dooper-device-testing-station
  30. 30 http://www.64digital.co.uk/blog/66/the-64-digital-super-dooper-device-testing-station
  31. 31 http://twitter.com/klick_ass
  32. 32 http://www.64digital.co.uk/blog/66/the-64-digital-super-dooper-device-testing-station
  33. 33 http://phonegap.com/2012/03/29/phonegaps-new-device-wall/
  34. 34 http://adactio.com/journal/5661/
  35. 35 http://www.flickr.com/photos/adactio/7801326422/in/photostream
  36. 36 http://techcrunch.com/2012/05/11/this-is-what-developing-for-android-looks-like/
  37. 37 http://www.devicelab.fi
  38. 38 http://www.dmolsen.com/mobile-in-higher-ed/2012/06/26/how-to-build-a-device-lab-part-1/
  39. 39 http://klick-ass.com/awesomeness/need-multidevice-stand-for-mobile-web-development/
  40. 40 http://www.theironmill.co.uk
  41. 41 http://labs.adobe.com/technologies/shadow/
  42. 42 http://support.google.com/chrome/bin/answer.py?hl=en&answer=154007
  43. 43 http://jsbin.com
  44. 44 http://mobiletestingfordummies.tumblr.com/post/27405408570/web-inspector-remote
  45. 45 http://xip.io
  46. 46 https://showoff.io
  47. 47 http://progrium.com/localtunnel/
  48. 48 http://proxylocal.com
  49. 49 http://stephanierieger.com/strategies-for-choosing-test-devices/
  50. 50 http://shaundunne.github.com/deviceLibrary/
  51. 51 http://www.netmagazine.com/tutorials/build-responsive-site-week-going-further-part-5
  52. 52 http://lab-up.org
  53. 53 http://lab-up.org
  54. 54 http://klick-ass.com/awesomeness/avoid-the-tamagotchis-a-list-of-open-device-labs/
  55. 55 http://clearleft.com/testlab/
  56. 56 http://shaundunne.github.com/deviceLibrary/
  57. 57 http://www.exeter-web.org/device-library
  58. 58 http://cole007.net/blog/111/setting-up-a-mobile-testing-suite
  59. 59 http://co-up.de/2012/07/31/open-device-lab.html
  60. 60 http://modl.se/
  61. 61 http://www.80beans.com/en/blog/2012/08/07/introducing-our-open-test-lab-for-mobile-device-testing
  62. 62 http://www.netvlies.nl/openbaar-testlab
  63. 63 http://www.devicelab.fi
  64. 64 http://fliptation.ch/#lab
  65. 65 http://mobileportland.com/device-lab
  66. 66 http://www.devicelabdc.com/
  67. 67 http://www.academy.mightyminnow.com/location/open-mobile-lab/
  68. 68 http://www.ajcomputers.com/news/introducing-open-test-lab-mobile-device-testing/
  69. 69 http://devicelab.co.za

↑ Back to topShare on Twitter

Viljami is the Lead Front-End Designer at Adtile. He is also the founder of Helsinki Device Lab which is established to help the web developer community of Helsinki to be able to test their responsive designs on a growing range of mobile devices. He’s passionate about responsive design, CSS, JavaScript and web typography, and both tweets and writes actively about these subject. Viljami has been designing web sites for over a decade.

Advertising
  1. 1

    Very nice idea! I’m kind of shocked that it don’t exist in Barcelona!

    0
  2. 2

    What can I say. It’s great to see that ideas like these are implemented in real world. Expect me to visit Helsinki Open Device Lab some day. Thanks for the article and setting up a device lab in Helsinki.

    It’s another story if site doesn’t work as expected in some devices.

    0
  3. 4

    Good idea but i believe doing such things via internet is still a lot easier. There are some websites that allow you to test how your site looks in different browsers. The same thing can be accomplished by setting up different devices and post screenshots online. No travel needed, fairly secure and quicker to use.

    Btw, you should change Windows Mobile to Windows Phone in the 3rd paragraph ;)

    0
    • 5

      In my experience, that kind of services just aren’t enough. They don’t tell you if your touch targets are too small for fingers, or if the changing context of use requires different approach. They also don’t tell you how the site actually responds to touch gestures and if the performance on actual device is good enough.

      Thank you for the correction, it seems that I’m referring to the ancient predecessor of Windows Phone right now there. :-)

      0
    • 6

      Testing online, on your monitor with no actual touch events, not latency, no processor lag, different pixel density… etc. There is no comparison to the real thing.

      0
  4. 7

    Wooohoo!! I live in Brighton!

    I shall be checking this out for sure!

    0
  5. 8

    Great, we are thinking about starting a ODL in Frankfurt/Germany. Lots to consider, but this article will come very useful, many thanks!!

    0
  6. 9

    Thanks for the eye-opener! We have a meeting tomorrow down here and I’m going to propose the build a device lab here too (Leeuwarden, Netherlands). Never thought about this, but it seems to me as really useful.

    0
  7. 10

    For city’s in the USA you should label the state its in as well – Portland could be in Maine, Oregon or a few other spots… I’m guessing its the OR one tho.

    0
  8. 12

    Thank you for taking the time to write this article. Great idea

    Some follow up questions:
    How do you handle security in the open lab?
    Any advice on your process in deciding which devices to upgrade
    Are you rooting to move versions up and down? (particularly android)

    I have been using keynotes deviceanywhere – advertised as real devices that can be seen and used remotely. Still like to hold the physical device

    0
    • 13

      “How do you handle security in the open lab?”

      We are pretty heavily armed against burglars. We handle security by using an alarm system with remote monitoring + video cameras and then there’s also the security company who’s monitoring the space 24/7 and if someone tries to break in the company instantly sends a guard to the location. We also have a huge safe at the office where the devices rest when not in use.

      Anyway, we had all these things listed above also before the lab were here, and I don’t really think that all this is necessary only for it as our current collection has a lot of elder/pretty used devices that don’t have a high selling value.

      There’s also a lot of people at our office during the day time, so I’m not really worried about someone coming in from the door and trying to steal the devices (and the door is locked anyway, so you can’t just walk in here without no one seeing).

      “Any advice on your process in deciding which devices to upgrade”

      My advice is to be careful with that. Sometimes there’s no way to go back to older version (with iOS you have to use SHSH blobs to be able to manually downgrade). As Stephanie Rieger said, owning four BlackBerry devices with four different versions of the OS is infinitely more valuable than owning four with the same version.

      You should base the decisions on the audience and overall market share in your own location (or the target area). Which OS versions are most widely used there right now?

      “Are you rooting to move versions up and down?”

      Not at the moment.

      0
  9. 14

    I heard Jeremy talk about this at #smashingconf and I instantly loved the idea. Hopefully, there will follow some more in Germany soon.

    You might want to add to the software section that Safari 6 (on Mac apparently, as Safari Windows was silently discountinued) in combination with Safari on iOS 6 has the ability to do remote debugging, too. The only downside compared to shadow: It only works when you connect your device thru USB.

    0
  10. 16

    Would love to know if there is a lab like this in Toronto, Canada.

    0
  11. 17

    And Adobe Shadow is history. The labs page redirects to http://html.adobe.com/edge/inspect/.

    0
    • 18

      Yes, the name has been changed to Adobe Edge Inspect.

      0
    • 19

      I wouldn’t say it’s “history”, as that’s basically the same tool. Just the name has changed and it has a price now. I still recommend it, if you can afford the price tag.

      0
  12. 20

    I love this Open Test Lab thing that is growing. I run a community for software testers (http://www.softwaretestingclub.com) and since seeing this I have been trying to get members to post what devices they own on their profile listing, I’m hoping over time it will become a useful way for people to get in touch with testers directly to perform specific tests on specific devices.

    I’d also love to see testing companies open up, am sure there are a few out there with many testing device resources.

    0
  13. 21

    Thanks a lot. I live in Turku near Helsinki. I need to visit this lab.

    0
    • 22

      Viljami Salminen

      October 2, 2012 3:16 am

      Welcome! Tweet me or send an email few days beforehand so I can make you a cup of coffee/tea too. :-)

      0
  14. 23

    Hi, I’m very interested in opening an ODL, unfortunately the situation of my country (Mx), and my ignorance on the topic of non-profit orgs tell me that doing that isn’t an option for doing it for a living, I would really like to invest in a lab, but by being open, how could I rise funds to make it work? Hope you guys could give me some ideas, which will be very appreciated.

    Thanks!

    0
  15. 24

    Absolutely great service for the developer’s community.

    I wish i had thought of this self-service device lab concept myself :)

    0
  16. 25

    If there is no device lab in your city, you can use Samsung’s Remote Test Lab for a huge range of Android and bada devices:
    http://developer.samsung.com/remoteTestLab.do

    0
  17. 27

    What an impressive lab!.

    We have nothing similar in South America, so… in the name of my company (Ibex Technologies), and Chile, I must say: Challenge Accepted!

    0
  18. 28

    Initiated by a chat with Jeremy Keith and his talk at Smashing Conference I never got rid of the idea in my head of establishing a device lab as well. Seeing the people in Frankfurt and Berlin starting one did not help me forgetting it. This are, NRW, also needs one. But where? That was the question. And with where I don’t mean the actual city. I mean the room or space? I mean, of course I can do this in my living room, but I guess my family would somehow not really like this idea. So finally thru contacting people whom I thought maybe interested in joining the exhibition at this years beyond tellerrand conference, I got to mention this idea to Oliver who is running the Garage Lab in Düsseldorf in the Grarage Bilk. And what should I say? He liked the idea, they have the space, I want to do it … so we are doing it. Within the next weeks after btconf, I we are going to plan and organize it. That means Düsseldorf will have an Open Device Lab soon as well! (Meanwhile you can of course already donate your old and/or devices you don’t use anymore ;) ) .

    Thats my little story about this 8)=

    0
  19. 29

    This is a great idea, but i agree with many that commented and said that internet would still be a a lot easier. The monthly costs to keep this up and running and the theft can be really messy. The concept is great and could succeed world wide, but I believe opening a few screen shots or tabs could result in a less mess world and money saver.

    0
  20. 30
  21. 31

    The link to the ODL in Washington, DC in the US is incorrect. SolidDC is not in charge of the DC Device Lab, they only designed the brand/site. The DCDL website is located at: http://dcdevicelab.com/ Thanks!

    0
  22. 32

    We are planning to start an Open Device Lab in Hong Kong.. Any advices?

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top