Transport for London

The Brief

Transport for London (TfL) spent 18 months working hard to redevelop its Journey Planner webapp. It is a high profile project: the Journey Planner website gets over 250 million visits a year and it provides a core service to its customers.

As a leading provider of transportation in the UK, TfL is rigorous in testing all new developments, but realised that conventional testing wouldn't include mobile device users. TfL asked us to conduct In the Wild Testing to ensure the Journey Planner webapp performed as well in real conditions, as it did in controlled, lab conditions.

What are the Issues?

Essentially, the issues are: variety of device, operating system, browser and connection. The data connection between device and webapp can be affected by variable strength Wi-Fi, cellular network effects (including contention signal strength and movement). Although many of these issues cannot be controlled by the webapp owner, how a webapp handles sessions, and careful consideration of data issues and design can help make a better mobile experience.

What was the Testing Approach?

We started by thinking like a user. This entails creating user scenarios that mimic real life – the Journey Planner webapp allows you to plan your journey, either using your input start and destination, or by geographic lookup of your device location.

To test the webapp, we mapped likely scenarios for Journey Planner users – for example, looking up local travel options from a station terminus, researching departure times in congested areas, or idly browsing on low bit public Wi-Fi.

After this, we enhanced these scenarios using the full range of latest and legacy mobile devices, operating systems and browsers and added in different forms of cellular networks (E, GPRS, 3G, 4G) as well as various qualities of Wi-Fi. Finally, we added in movement between network cells or Wi-Fi zones – which can also cause issues with complete or partial connection dropouts.

Testing these same scenarios under lab conditions gave us control results which we could then use as a clear baseline comparison to the real world results.

To get clear results, we sent testers out with a full range of mobile devices and recording tools to test the Journey Planner webapp.

The Result

As expected, the key issues were around data performance. Some of the pages weights were around 1MB. These either partially or completely failed to load in high contention areas or in areas with lower connection speeds. Interestingly, after failing to load, often the pages didn’t load when the device was moved back to a higher connection speed area and had to be refreshed altogether.

It became apparent how crowded areas affected the quality of the data connection – not because the signal was blocked but because the number of other users and their data requirements caused contention. This became a key issue as we had predicted a strong use of the Journey Planner webapp in crowded stations.

Other key issues were around the ‘Use My Location’ function, which failed or returned wrong results when data flow was switched between Wi-Fi and cellular networks.

More modern devices, which were able to use 4G, were able to seamlessly switch between cellular and Wi-Fi networks and gave a substantially better performance than older generation devices.

Surprisingly, there remain a lot of legacy phones and operating systems in the market. The Android statistics from Appbrain show a significant user base of Samsung Galaxy S2, when the latest model is Galaxy S5:

*showing share of total market

http://www.appbrain.com/stats/top-android-phones

Likewise for the Android versions:

*showing share of total market

(http://www.appbrain.com/stats/top-android-sdk-versions)

For Apple users, there is a resolute section with legacy iOS versions, but the majority upgrade to the latest version relatively quickly. (Data from Mixpanel):

*showing share of total iOS market

Even with Apple users, legacy phones make up the highest numbers:

*showing share of total iPhone market. Data from Mixpanel.

Although the published statistics tend to lag the actual trends, it's still important to consider the older devices and operating systems, while continuing to design for newer versions.

Following our testing, TfL had the data to implement updates that mitigated and solved the issues. TfL optimised the Journey Planner webapp to ensure the greatest number of users are able to access it effectively at all times and in all places.