Exploring what’s behind the real-time transport data in Nottingham city
For a number of a years now Nottingham has had real-time information available for the different public transport operators. A question that I had more recently is what are the data sources behind them? My reason for suddenly thinking about this is because at my employer we have recently gone live with over 30 digital signage screens across our different locations. One idea discussed was leveraging public transport information to display on these screens at intervals, aiming to be useful for those using public transport. The first big question around this was is this data even open and available? The answer: it’s complicated.
Removing specific operators in the city out of the picture for just a moment, within Nottingham city we have bus, tram and rail transport services. There is of course also East Midlands Airport, but you aren’t catching flights on a daily basis usually, so we’ll ignore air services in this scenario. Looking at this from purely the transport/service type, can real-time bus, tram or rail service information be obtained in some way outside of the specific operators themselves?
Open data/real-time API access availability
During my research I came a few open data/API access options that weren’t either operator specific or paid commercial services and met the open data criteria.
- Bus Open Data Service (BODS) — A GOV.UK service through the Department for Transport. Providing timetable, live location data and fare information for operators across the United Kingdom.
- National Rail Enquires — Real-time information for train/replacement rail services. This data comes from a system called Darwin. Using Computer Reservation System (CRS) codes, you can obtain real-time train service information for railway stations. For example, the CRS code for Nottingham Station is “NOT”.
- NextBuses API — Provided by a company called Traveline, this too is also open data for multiple operators within the city, using ATCO/NaPTAN codes for returning data by individual stop points.
ATCO/NaPTAN codes are available as a dataset through the National Public Transport Access Nodes (NaPTAN) dataset, this covers all operators in Great Britain, this is an important datapoint in relation to working with real-time data and stops.
The Find Open Bus Data service from GOV.UK is part of a legal obligation for operators to provide their data for use by third parties or members of the public.
The National Rail Enquiries APIs are under the Rail Data Marketplace which is also a GOV.UK service.
NextBuses is run by Traveline who are a not for profit company.
From investigating and obtaining access to these open data services, I can conclude that:
- Real-time bus location information is available through BODS for major operators (exception is tram/metro services) but would require blending with the timetable data to have real-time prediction data that is usable, a fairly large effort to do as a solo project!
- Rail services serving stations within Nottingham city are available with real-time data through National Rail.
- NextBuses does not have real-time information for Nottingham transport services for any operator from my testing, all SIRI-SM responses are aimed departure times at this time.
Alternative APIs/data access
Establishing that open data APIs exist but do not cover the real-time data aspect in all cases, we enter into more unofficial areas. Looking at the situation from an operator perspective again, we have several major operators in the city, I focussed on what I would call the major trio, Nottingham City Transport (NCT), trentbarton and Nottingham Express Transit (NET), these operators cover the majority of bus services and our tram service in the city respectively.
Nottingham City Transport (NCT)
From an open data perspective, NCT to their credit do provide various datasets and these are freely available through their website, this appears to be something established since 2019. The missing data is unfortunately real-time access however. An interesting discovery is that a company called Passenger Technology Group appears to be behind NCT and a lot of other transport providers and their open data portals across the United Kingdom. Here’s the same open data portal on their website.
Without too much effort, inspecting just the DNS requests made by the official mobile app of NCT, reveals that https://nctx.arcticapi.com is the source of the real-time data in the mobile app. Even more surprising, this API does not appear to require any authentication and you can openly query it just by using your browser, but here is visual view of the network requests being intercepted.
Here’s a few endpoints of interest which you can just run in a web browser:
- Get all network alerts/disruptions (This is the same data displayed on the service updates page on their official website) — https://nctx.arcticapi.com/network/disruptions
- Get stops by search term — https://nctx.arcticapi.com/network/stops?search=Forest
- Get real-time bus information from a specific stop — https://nctx.arcticapi.com/network/stops/3390FO07/visits
The ATCO code 3390FO07 is for Forest Recreation Ground. This can be obtained through this API directly, the NCT website by taking the “id” query parameter when using: https://www.nctx.co.uk/explore and selecting a stop or using the NaPTAN dataset.
There are various more endpoints available under this API, however whether you should use this API outside of the official NCT mobile app is another question entirely, it is technically “open” but not able to be considered public. It is however the real-time data specifically for NCT services and at least seems to be backed by solid open data principles.
There has been a few other developers who have been interested in obtaining the real-time transport data in Nottingham and one project I came across was from Simon Prickett who essentially modelled a JSON API around the NCT website data using scraping. It also turns out that Simon has also had conversations around the existence of this arcticapi.com API with some people in the know. Whether or not this API gets locked down at some point is unknown, but it was an interesting find to come across during my research!
Screen scraping is of course a solution when no “open” API exists (although clearly one does!) and the NCT website has a nice web path for stops e.g. https://www.nctx.co.uk/stops/3390FO07 however many websites (including NCT) use services like Cloudflare which has various technologies to analyse traffic and prevent this type of thing, you’ll likely trigger bot detection at some point and then your scraper has been defeated by a challenge/captcha, breaking whatever app is using it. Also any change made to the HTML of the website, will also cause issues.
Given Passenger Technology Group’s involvement with a lot of other operators, by using their open data portal, you can find different arcticapi.com endpoints for these providers. The customers featured which do not have “myTrip” under their name seem to have a corresponding API endpoint with their company name as defined by the open data portal page slug as the subdomain. e.g. for Blackpool Transport their handle is bts. There also appear to be staging environments by appending -staging to the subdomain e.g. nctx-staging.arcticapi.com.
For operators using “myTrip” the API endpoint appears to be: https://mytrip.arcticapi.com/network/vehicles?tenant=[operator]
trentbarton
The other major bus operator in Nottingham is trentbarton, covering Nottinghamshire, Derbyshire and Leicestershire. Unlike NCT, trentbarton has a couple of mobile applications for different purposes. There’s the mango app, which is for mobile tickets and then the hugo app, which has real-time data for stops including data on other operators, like NCT, which appears to be real-time as well.
The trentbarton website provides real-time data outside of the hugo app so I decided to stick with their website for some analysis for any API possibilities, given it is much easier to use Chrome Developer Tools than do TLS interception on mobile apps. I have as a separate project, reverse engineered the Android version of the hugo App and found a public API behind it, but I will not be disclosing the information directly, given decompiling and reverse engineering of proprietary source code was performed.
For the main website however, I spied this resource being loaded in Chrome Developer Tools which stood out as possibly interesting.
https://www.trentbarton.co.uk/Scripts/LiveTimings.jsInspecting this resource reveals an endpoint of: https://www.trentbarton.co.uk/RTILiveTimings.aspx is being used for live time functionality on their website.
As LiveTimings.js being a JavaScript resource and thus client-side, it is rather easy to see the type of requests that can be made and even the specific AJAX calls and parameters required when using this endpoint. It requires POST requests with specific URL parameters. Examples of valid requests to this endpoint:
- Retrieve all services — https://www.trentbarton.co.uk/RTILiveTimings.aspx?m=GetServices
- Get stops using a postcode search — https://www.trentbarton.co.uk/RTILiveTimings.aspx?m=GetStops&includePostcodeSearch=true&postcode=NG1 1NG
- Retrieve real-time information for a service at a specific stop — https://www.trentbarton.co.uk/RTILiveTimings.aspx?m=GetRTI&service=16&stop=16732
- Retrieve real-time information for a service at a specific stop with full data — https://www.trentbarton.co.uk/RTILiveTimings.aspx?m=GetRtiFull&service=16&stop=16732
I created a public collection through Postman which provides example API calls and requests.
I used the Rainbow One service as an example (because let’s face it, it’s a great name!). It’s service ID is 16, so in the API this is the service value to use, for the stop, these can be obtained via the GetStop query or NaPTAN dataset.
There’s a fair bit of data that you can access. Interestingly, despite providing the HTTP header of Accept with an application/json value, this API does not return an actual JSON response, instead it’s returned as text/html, but the response data is formatted as JSON, so you could easily decode it as JSON into an array.
While technically not an open data API, this is also a valid data source given their own website is using it! However again, this not an official open API and you run the risk of potentially getting rate limited or outright blocked if you abuse it. Their website does have protections against excessive requests, I know this because I’ve accidentally triggered it!
Nottingham Express Transit (NET)
NET has a slightly more interesting story, because I’ve previously come across someone else in the city who shared my interest in the real-time transport information several years ago. Martin Rothe discovered the existence of https://robinhood.arcticapi.com, this was found within the now deleted Robin Hood Network app. This API provided real-time tram information and like NCT’s API was also open and is operated by Passenger Technology Group. Comparing both APIs from memory they seemed very similar if not identical. Notice how I am talking in past tense, this was several years ago, the API endpoint has long since disappeared and now returns NXDOMAIN.
Fast forwarding to the present, as far as I’m aware there is no real-time information within any NET or what is now the Robin Hood Ticketing app now, but all the tram stop information displays are of course real-time, so there’s a data source out there somewhere! The NETGO! app has service status updates, but that is about as close as it gets to anything related to real-time tram data currently.
I put the NETGO! app through a proxy to examine the requests if there was anything interesting.
There’s some interesting Google related infrastructure information that the app appears to use Google Firebase and Firestore as a database backend and the discount page is coming from their main website: https://www.thetram.net/destinationdiscountnetgo. Aside from that, not much. The service updates part of the mobile app does not provide hints on where the data source for it is. I can assume, this data is inputted by NET staff and this mobile app is then pulling it from the Firestore database.
I did discover the website Nottingham Tram Live, which appears to have some form of API data source behind it which isn’t any of the one’s written about above. The site uses AJAX, so you can see the requests easily, for example, this is one of the API requests made for a specific stop (Forest Recreation Ground).
http://api.tramlive.co.uk/api/N17However viewing the JSON response, it is clear it’s just scheduled timetable information with “No Live Data” reported in the data. The JSON response data doesn’t match any API endpoint/specification I’ve found from other providers but I strongly suspect this site is proxying another API behind the scenes.
Hiding in the source appears to be another map for Sheffield, and a similar API layer specific for stops under Sheffield e.g. http://api.tramlive.co.uk/api/S10, so clearly the data source behind this site is wider than Nottingham as well.
The website itself identifies as “Powered by nextbus and National Rail Enquiries”. National Rail Enquiries is one of the open data APIs, however “NextBus” is a different data provider entirely and doesn’t cover UK operators. The source is more likely to be the Traveline National Dataset (TNDS). This is updated regularly, provided in the TransXChange format, although this site appears to be doing further work or conversion, given the original source is XML and the API behind it is JSON.
The site itself appears to be a unofficial project by an individual. Any stop with a train station does not load due to the site using a broken endpoint for rail services. For example trying to query “Nottingham Station”, causes a 500 error. The webserver throws an error with a verbose stack trace, which is probably a good reminder of why you need to be careful with stack traces in production/live sites. This exposes a few details which potentially make it vulnerable, but we can glean a few insights into what’s happening behind the scenes.
The stack trace thrown:
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~yabariarchery/15.415213526270860197/main.py", line 567, in get
statuspage = nrdicts[stop]['fetchobject'].get_result()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 615, in get_result
return self.__get_result_hook(self)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 461, in _get_fetch_result
'Deadline exceeded while waiting for HTTP response from URL: ' + url)
DeadlineExceededError: Deadline exceeded while waiting for HTTP response from URL: http://yabaritrain.apphb.com/departures/NOT/20?expand=true&accessToken=bfa11d10-6304-40d8-a57d-07d2313ef150The offending request:
http://yabaritrain.apphb.com/departures/NOT/20?expand=true&accessToken=bfa11d10-6304-40d8-a57d-07d2313ef150It is a good job that the endpoint is no longer around, as the site has leaked its access token and with that other potentially sensitive information. It suggests this site has not been updated in sometime. The apphb.com domain appears to be related to AppHarbor, which shut down in late 2022. This confirms to me that this is likely a project by an individual not associated with NET/Nottingham City Council.
Throwing a couple of bogus API calls, we can coerce this app into revealing slightly more of the logic behind it, here the Python logic is expecting an object attribute that doesn’t exist, because it’s not a valid stop value.
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/a76f273e9db3761c/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~yabariarchery/15.415213526270860197/main.py", line 164, in get
for mode in record.directions:
AttributeError: 'NoneType' object has no attribute 'directions'The valid stop values appear to be coming from this JSON data: http://nottingham.tramlive.co.uk/json/stopv2.json
My only clue as to the author or origin of this site is the name “Yabari”, this appears in a few places within the code and infrastructure of the site, including the home path of the app itself as seen from the Python stack trace, it is hosted as a Google App Engine web application. A bit more searching provides a Google Play Store link to a mobile app which reveals a developer contact email. The app is unlikely to work on any modern Android device version due to being old and not updated since 2016. Through an Android Emulator and unofficial APK installers, you can force it to be installed and run, but the mobile app is just using the same website APIs already covered.
Going to the source: Nottingham City Council
After all of my own research and findings, the best source remains Nottingham City Council themselves. To the council’s credit, they haven’t been shy about talking about the real-time transport information in the city and have posted several updates on transportnottingham.com about projects and on-going updates, the most recent one I could find was September 2022. On that update is an email realtime@nottinghamcity.gov.uk which I have contacted to try and get any further direct information. My main question is around the existence of a singular open data system which all major transport providers are surely feeding into for the real-time displays that are active in the city to operate effectively.
This project I started probably couldn’t have come at a worst time, given the recent news around the financial condition of our city council. There was a recent consultation around cost saving measures which included turning off all real-time displays to save money, which I think would be absolutely the wrong thing to do. Clearly Nottingham has invested a lot in real-time transport data infrastructure turning it off would essentially send us backwards as a city. Of course there are also other vital services under threat too, but removing access to real-time data in the city is certainly something I would not be in favour of, even more so now from my own research, the availability of this data being open has a long way to go, which what perhaps has driven me to explore this in case unofficial solutions were needed to provide this data going forward.
I hope I hear back from Nottingham City Council if anything just to get a bit more insight into how things are running and what the future might be for open data access across all operators.
Final thoughts
While clearly there’s technically access to real-time data for some Nottingham transport operators, these options are mostly operator specific or methods which cannot be classified as open or proper API layers in most cases and using them in any production use case would be highly risky or possibly in extreme cases, in violation of some form of service usage.
Reverse engineering or doing further TLS interception on mobile apps from providers is another possible avenue to explore. On Android this is possible, but with some caveats and can get complicated when requiring root to install trusted CA certificates to allow you to see TLS traffic. This technique was used to confirm some of the findings documented and can be explored further.
I hope the open data situation changes in the future, however I fear given recent financial conditions and pressures the city is facing, something like this will most likely be not a priority in the foreseeable future.
