Improving the TFL Open Data Feeds and APIs #2

So following up from my earlier post about the TFL Data APIs I don't think I was being fair towards the TFL dev team. The API that is currently provided, called TrackerNet, was made available in August 2011 and is now going on its 3rd year. The API is still quite horrendous by modern (and 2011) standards but granted is the first step publishing the vast network of disparate data-sources in use used within the TFL after its merger in 2000.

Little did I know that there is indeed an impressive brand new API (api.beta.tfl.gov.uk) powering the newly redesigned tfl.gov.uk website that was launched in late March 2014. This I discovered by accident when investigating the seeming information difference between station disruptions in the TrackerNet API and what was displayed on the TFL website.

Below are images showing the difference in information provided for the same tube station "Paddington" between the old TrackerNet API and what is shown on the TFL website itself.

TrackerNet results for Paddington station

TFL website information for Paddington station
(only the past paragraph is what is available in TrackerNet)

The new API

api.beta.tfl.gov.uk offers information about it seems every aspect of the TFL network and facilities. The developers seem to have taken the approach to developing the new TFL site as simply one presentation layer on top of this vast and rather well performing API.

The guidelines ask that developers that want to leverage the API sign up for a key and an app-id that should be submitted with every query. But all API calls seem to work just fine without supplying these values. 

One minor gripe that I have with the API is that it is perhaps overly verbose with average data result sizes close to 1MB in size (uncompressed) . Comparing the size of the boris bikes between the APIs:

Old API		New API
Compressed	Uncompressed	Compressed	Uncompressed
31.8KB	266KB	110KB	1.6MB

But this is a small price to pay today for a much richer and consolidated API experience.

Unfortunately the data published by the old API is not compatible with the new one and most notibly is the station and tube line names that are different. Luckily developers such as Matt Hinchliffe have already gathered some really useful support data to make the data and its use easier.

Now I guess I should update my bike and bus websites to leverage this new found API gold :)

Improving the TFL Open Data Feeds and APIs

This blog has moved to https://blog.sverrirs.com/2014/07/improving-tfl-open-data-feeds-and-apis.html

You will be automatically redirected.

Failover resiliency is indeed important

When building software that is somehow dependant on data provided by someone or something that is out of your direct control I find myself wondering how reliable that service actually is and what the impact is when it eventually goes offline (because they all will at some point...)

I experienced this today with my super simple cycle hire page I built a few weeks for my mobile. The page quickly gives me the status of cycle stations close to my home and work which helps me decide where to go to get home/to work. An example of the page is below:

My simple page relied on the rather excellent free service at http://api.bike-stats.co.uk/ offered by Sachin Handiekar to provide it with the latest docking station information.

But today his service was down and my handy little website showed me nothing
:(

Adding failover providers

After gambling on today's docking stations numbers I decided to hunt for alternative data providers that I could use in case bike-stats were unavailable again in the future. One such alternative service is another excellent one provided at http://borisapi.heroku.com/ by Adrian Short.

A simple way to provide this failover would be to simply call my preferred one first and if that failed to give me valid data I would use the alternative as a backup. Easy enough to hack in php as both sites offered a JSON feed:

function URLopen($url) 
{ 
	// Fake the browser type
	ini_set('user_agent','MSIE 4\.0b2;');
	$dh = fopen("$url",'rb'); 
	$result = stream_get_contents($dh);
	fclose($dh);
	return $result; 
} 

global $json, $isJsonValid, $jsonSource;

// Indicates if we successfully parsed the json data
$isJsonValid = false;

// First try the bike-stats stuff (this might be slow or offline)
$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$json = json_decode($rawjson, true);
if( $json != null )
{		
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}

// As a backup use the heroku json site	
if( !$isJsonValid )
{
	$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
	$json = json_decode($rawjson, true);
		
	if( $json != null )
	{			
		$isJsonValid = true;
		$jsonSource = "borisapi.heroku.com";
	}
}

Now with my super basic failover my problem became that these two providers, although scraping the same data, present the JSON data in slightly different formats.

Going back to my original page I decided to normalise the json data that was used and simply have my failover code perform a quick transformation of the data to my new normalised format. So my final improved data sourcing with failover resiliency and normalisation (wow quite big words there...) ended up looking like so (sorry for the indentation):

try {

$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){		
	$json = array();
	foreach( $jsont["dockStation"] as $station){
		$json[] = array(
		"id"=> $station["@ID"],
		"name" => $station["name"], 
		"bikes" => $station["bikesAvailable"], 
		"docks" => $station["emptySlots"],
		"lat" =>  $station["latitude"],
		"lon" =>  $station["longitude"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}
}catch (Exception $e){
	$isJsonValid = false;
	$json = null;
}

// As a backup use the heroku json site	
if( !$isJsonValid ){
try{
$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){
	$json = array();
	foreach( $jsont as $station){
		$json[] = array(
		"id"=> $station["id"],
		"name" => $station["name"], 
		"bikes" => $station["nb_bikes"], 
		"docks" => $station["nb_empty_docks"],
		"lat" =>  $station["lat"],
		"lon" =>  $station["long"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "borisapi.heroku.com";
}
}catch( Exception $e ){
	$isJsonValid = false;
	$json = null;
}}

It is ugly sure, but does a marvelous job getting me the information I need :)

Want your own map?

A little while ago I designed a simple customization page that anyone can use to create their own light-weight bike stats web page. These pages are best viewed on a mobile device (such as a phone).

Get your own right here :)

Happy cycling...

Learning about it makes it better somehow

Stuck on or waiting for the Tube in London?

Hearing the disembodied electronic woman's voice droning on about a "Signal failure"?

Yearning to know more?

GPS points of all bus stops in London

Hope I am not the only one this happens to.

Dream of building the greatest app ever, come up with a decent enough idea, research the available technology to build it and the data needed. Even gather said data and work on converting it to a suitable format. Then realise that it is 5am and you need to get up for work in the morning. 

Anyways, in hopes that I help someone else out there kickstart their project. Here is a handy way to download the entire TFL london bus network, every route, GPS coordinates of every stop right to your doorstep.

1. Download cURL here.

2. Run the following commands 

curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={1,10,100,101,102,103,104,105,106,107,108,109,11,110,111,112,113,114,115,116,117,118,119}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={12,120,121,122,123,124,125,126,127,128,129,13,130,131,132,133,134,135,136,137,138,139,14}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={140,141,142,143,144,145,146,147,148,149,15,150,151,152,153,154,155,156,157,158,159,16,160}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={161,162,163,164,165,166,167,168,169,17,170,171,172,173,174,175,176,177,178,179,18,180,181}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={182,183,184,185,186,187,188,189,19,190,191,192,193,194,195,196,197,198,199,2,20,200,201,202}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={203,204,205,206,207,208,209,21,210,211,212,213,214,215,216,217,219,22,220,221,222,223,224,225}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={226,227,228,229,23,230,231,232,233,234,235,236,237,238,24,240,241,242,243,244,245,246,247,248}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={249,25,250,251,252,253,254,255,256,257,258,259,26,260,261,262,263,264,265,266,267,268,269,27}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={270,271,272,273,274,275,276,277,279,28,280,281,282,283,284,285,286,287,288,289,29,290,291,292}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={293,294,295,296,297,298,299,3,30,300,302,303,305,307,308,309,31,312,313,314,315,316,317,318,319}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={32,320,321,322,323,325,326,327,328,329,33,330,331,332,333,336,337,339,34,340,341,343,344,345,346}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={347,349,35,350,352,353,354,355,356,357,358,359,36,360,362,363,364,365,366,367,368,37,370,371,372}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={375,376,377,379,38,380,381,382,383,384,385,386,387,388,389,39,390,391,393,394,395,396,397,398,399}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={4,40,401,403,404,405,406,407,41,410,411,412,413,414,415,417,418,419,42,422,423,424,425,427,428,43}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={430,432,434,436,44,440,444,45,450,452,453,455,46,460,462,463,464,465,466,467,468,469,47,470,472}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={473,474,476,48,481,482,484,485,486,487,488,49,490,491,492,493,496,498,499,5,50,507,51,52,521,53}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={54,549,55,56,57,58,59,6,60,601,602,603,606,607,608,609,61,611,612,613,616,617,62,621,624,625,626}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={627,628,629,63,632,634,635,639,64,640,641,642,643,646,647,648,649,65,650,651,652,653,654,655,656}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={658,66,660,661,664,665,667,669,67,670,671,672,673,674,675,678,679,68,681,683,685,686,687,688}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={689,69,690,691,692,696,697,698,699,7,70,71,72,73,74,75,76,77,78,79,8,80,81,82,83,85,86,87,88}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={89,9,90,91,917,92,93,931,94,941,95,953,958,96,965,969,97,972,98,99,A10,B11,B12,B13,B14,B15,B16}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={C1,C10,C11,C2,C3,D3,D6,D7,D8,E1,E10,E11,E2,E3,E5,E6,E7,E8,E9,EL1,EL2,G1,H1,H10,H11,H12,H13,H14}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={H17,H18,H19,H2,H20,H22,H25,H26,H28,H3,H32,H37,H9,H91,H98,K1,K2,K3,K4,N1,N11,N13,N133,N136,N137}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={N15,N155,N159,N16,N171,N18,N19,N2,N20,N207,N21,N22,N253,N26,N279,N28,N29,N3,N31,N343,N35,N38,N381}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={N41,N44,N47,N5,N52,N55,N550,N551,N63,N68,N7,N73,N74,N76,N8,N86,N87,N89,N9,N91,N97,N98,P12,P13,P4}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={P5,PR2,R1,R10,R11,R2,R3,R4,R5,R6,R68,R7,R70,R8,R9,RV1,S1,S3,S4,T31,T32,T33,U1,U10,U2,U3,U4,U5,U7}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={U9,UL1,UL2,W10,W11,W12,W13,W14,W15,W16,W19,W3,W4,W5,W6,W7,W8,W9,X26,X68}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"