When building software that is somehow dependant on data provided by someone or something that is out of your direct control I find myself wondering how reliable that service actually is and what the impact is when it eventually goes offline (because they all will at some point...)
I experienced this today with my super simple cycle hire page I built a few weeks for my mobile. The page quickly gives me the status of cycle stations close to my home and work which helps me decide where to go to get home/to work. An example of the page is below:
My simple page relied on the rather excellent free service at
http://api.bike-stats.co.uk/ offered by Sachin Handiekar to provide it with the latest docking station information.
But today his service was down and my handy little website showed me nothing
:(
Adding failover providers
After gambling on today's docking stations numbers I decided to hunt for alternative data providers that I could use in case bike-stats were unavailable again in the future. One such alternative service is another excellent one provided at
http://borisapi.heroku.com/ by Adrian Short.
A simple way to provide this failover would be to simply call my preferred one first and if that failed to give me valid data I would use the alternative as a backup. Easy enough to hack in php as both sites offered a JSON feed:
function URLopen($url)
{
// Fake the browser type
ini_set('user_agent','MSIE 4\.0b2;');
$dh = fopen("$url",'rb');
$result = stream_get_contents($dh);
fclose($dh);
return $result;
}
global $json, $isJsonValid, $jsonSource;
// Indicates if we successfully parsed the json data
$isJsonValid = false;
// First try the bike-stats stuff (this might be slow or offline)
$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$json = json_decode($rawjson, true);
if( $json != null )
{
$isJsonValid = true;
$jsonSource = "api.bike-stats.co.uk";
}
// As a backup use the heroku json site
if( !$isJsonValid )
{
$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
$json = json_decode($rawjson, true);
if( $json != null )
{
$isJsonValid = true;
$jsonSource = "borisapi.heroku.com";
}
}
Now with my super basic failover my problem became that these two providers, although scraping the same data, present the JSON data in slightly different formats.
Going back to my original page I decided to normalise the json data that was used and simply have my failover code perform a quick transformation of the data to my new normalised format. So my final improved data sourcing with failover resiliency and normalisation (wow quite big words there...) ended up looking like so (sorry for the indentation):
try {
$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){
$json = array();
foreach( $jsont["dockStation"] as $station){
$json[] = array(
"id"=> $station["@ID"],
"name" => $station["name"],
"bikes" => $station["bikesAvailable"],
"docks" => $station["emptySlots"],
"lat" => $station["latitude"],
"lon" => $station["longitude"]
);
}
$isJsonValid = true;
$jsonSource = "api.bike-stats.co.uk";
}
}catch (Exception $e){
$isJsonValid = false;
$json = null;
}
// As a backup use the heroku json site
if( !$isJsonValid ){
try{
$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){
$json = array();
foreach( $jsont as $station){
$json[] = array(
"id"=> $station["id"],
"name" => $station["name"],
"bikes" => $station["nb_bikes"],
"docks" => $station["nb_empty_docks"],
"lat" => $station["lat"],
"lon" => $station["long"]
);
}
$isJsonValid = true;
$jsonSource = "borisapi.heroku.com";
}
}catch( Exception $e ){
$isJsonValid = false;
$json = null;
}}
It is ugly sure, but does a marvelous job getting me the information I need :)
Want your own map?
A little while ago I designed a simple customization page that anyone can use to create their own light-weight bike stats web page. These pages are best viewed on a mobile device (such as a phone).
Get your own right here :)
Happy cycling...
In an earlier entry I mentioned that I was building a little app to display a bunch of static information. Sometimes I find it immensely useful to embed all resources into a single data-file. Not only does it make handling the data simpler but also retrieving it over the internet.
Realised today that I needed a good way to create a change list document for a small app that I am writing. The app has a few hundred screens, each of which is supported by a single data document that contains the information that the screen is showing.