Update #2 to Google Maps Directions to GPX data

I've applied several major updates to the tool discussed in this article. Update #1, Update #2, Update #3, Update #4 and Original post.

The Google Maps Directions to GPX has received its second update.

Try it

Change log

• Support for short urls (e.g. "goo.gl/maps/..." links).
• Support for multi destination directions.
• Waypoints will now get a full address into their <desc> field if available.
• Option to exclude creation of waypoints (external to the route).
• Option to provide a custom name for the route.
• Improved error handling and validation of Google Maps links.
• Addressing an issue where if URL is missing sections the Google service will estimate the destination (this will most likely cause an incorrect route to be generated).

Changes to JSON output only 

These have not yet been integrated into the GPX format.

• Inclusion of Google StreetView imagery where available.
• Preliminary support for elevation profile data (only for walking and cycling directions).
• Turn information available in markup format ("step" field).
• List of all countries the route passes through.
• List of all destinations in route (in addition to start and finish).

Try it


Map Link with Multi Destination Directions

Map Link with Cycling Directions (will have elevation data)

Update #1 to Google Maps Directions to GPX data

I've applied several major updates to the tool discussed in this article. Update #1, Update #2, Update #3, Update #4 and Original post.

In a previous post I shared an experimental tool that I had been tinkering with that converts Google directions to GPX format.

After quite a few interesting suggestions and discussions with readers I want to present Update #1 to this tool.

Try it

Major changes

1. New and more user friendly UI with error checking.
2. More control over output format and output file types (the tool now defaults to generating a file that has the broadest compatibility with the GPX standard).
3. Auto-generation of unique route point names (format "RPxxx").
4. Option to include the full direction text with each route point in the <desc> field.
5. Option to include a brief next turn information with each route point in the <cmt> field.
6. Optional JSON response for future web-development

The new UI with all options

Try it

Updating Tipsy (jQuery library) tooltip text on click to show different text

This blog has moved to https://blog.sverrirs.com/2015/08/updating-tipsy-jquery-library-tooltip.html

You will be automatically redirected.

New forecast data upgrade for the Météo website (#5)

This blog has moved to https://blog.sverrirs.com/2015/07/new-forecast-data-upgrade-for-meteo.html

You will be automatically redirected.

Updating the ESA Small & Medium Sized Enterprises map (#3)

Since I created the ESA contractor map in November I've been surprised to see that I seem to get regular traffic to it.

I must admit that when I built this first I was not concerned with tracking how often the underlying SME database changes on the ESA side and the project doesn't really have any built in stats tracking that (yet).

Neither have I found time to fully automate the extraction and parsing of the data from the SME database. This code to produce and prepare the background data all lives in a C# application that currently I need to run manually to get the updated lists of companies and fields.

Out of curiosity I decided to run an update on the underlying data today and was surprised at the number of new and updated entries that were located.

View Latest

The Changes

All in all there where 39 major changes to the registry including new, deleted and updated companies.

I've updated the map markers and fixed a few minor bugs relating to mobile clients.

I also introduced a change list section. This section is shown in the lower right hand corner of the map and lists the changes that I have made to the map source data.

I should do this to all of my applications actually, it is really handy to track what has changed and when.


A version of this article is also available on LinkedIn
https://www.linkedin.com/pulse/esa-small-medium-sized-enterprises-interactive-map-sigmundarson

Météo France, converting between different wind speed units (#4)

The small Météo France site I created has been updated with a new parameter that can be used to customize the unit that windspeed is presented as.

The u= parameter

The new parameter that controls the windspeed is ?u= and accepts the following values:

Value	Comment	Examples
u=bf	Beaufort scale of windforce between 0 and 12. More information about conversions. Unit "F"	Try it
u=ms	Meters per second. Unit "m/s".	Try it
u=kn	Knots. Unit "kn".	Try it
u=kmh	Kilometers per hour. Unit "km/h". This is the current default.	Try it

More information on conversion and meaning of these different scales can be found here and on Wikipedia.

This parameter works both for the website as well as the weather API service.

Examples

Knots
Kn

Beaufort Force
F

Kilometers per hour
Km/h

Meters per second
m/s

Expanding the Météo weather forecast site (#3)

I've discussed the work I've done to build a simpler weather UI for the French météo service in earlier posts (first entry and second entry).

We've been having exceptionally hot weather and I decided to expand the current web app with more information that is provided by the French weather service.

Try it

Severe weather warnings

The Météo service publishes a separate list of areas that have been issued with severe weather warnings (vigilance météorologique).  I thought it might be helpful to have this information readily available to the user.

After fruitlessly attempting to scrape the HTML on this website and being unable to locate the same elements that the browser debuggers indicated were rendered. I discovered to my dismay that most of the content on these pages are being rendered by a client side JavaScript. Why, I will probably never understand as the code is quite the mess to look at.

But luckily I noticed that the whole rendering depended on a single XHR call to an external XML service that luckily didn't require any sort of CrossSite authentication. So I simply went straight to the source and saved countless hours of debugging and fiddling with bad HTML. (http://vigilance.meteofrance.com/data/NXFR33_LFPW_.xml)

Unfortunately the source doesn't really represent the data in any easily understandable way so considerable digging around in the JavaScript code was needed to fully understand all the numerical codes presented in the XML.

The raw XML data from the Météo service

The rest of the JavaScript code and HTML gave me the pieces to understand the colour coding and images to render for each risk type and severity level.

Severe gales / strong wind

While digging around and trying to figure the weather warnings out I realized that my earlier design did not account for displaying warning and windspeed numbers for any forcasted and expected wind gales. This information was already available from my earlier parsing project and needed only minimal tweaking to get correct.

This information is now presented either within the day or as a whole day event, depending on which is applicable.

The new information elements.
Click for a larger version

Try it

This article is also available on LinkedIn:
https://www.linkedin.com/pulse/expanding-météo-weather-forecast-site-sverrir-sigmundarson

Upgrading the UI for the Météo weather forecast site (#2)

After building the initial version of the Météo weather page and confirming that the data was all correctly handled I went about sprucing the UI up a bit.

Try it

This project has been updated, click here for part three (data enhancements) of this article.

Not only was the original design purely a functional one and not particularly attractive or user friendly but it was also missing a few key data points such as intra-day weather and likelihood of rain.

The Old UI

The tools

With the help of JQuery and the JQuery UI I managed to wrestle it into a usable and aesthetically pleasing look and feel.

I ended up using the JQuery Accordion without too much customization. This widget suits the website quite well as I wanted to be able to show an overview of the next 10 days at a glance but then allow the user to drill down into each day to see the intra-day forecast in more detail.

The final UI design

The accordion is closed upon first loading the website but then the user can tap each day to drill down and see more detail data.

The initial view of the website
Accordion is closed

User has expanded a day and intra-day forecast is visible
Accordion is open

For longer term forecasts additional information is available on how confident the Météo weather model is about the prediction as well as a rough prediction of the likelihood of precipitation.

Extra data points are available for days further into the future

Customizing the forecast area

A new addition was the setup screen. This screen can be used to change the forecast area to a different city or area. This page is available by either clicking the name of the current area at the top of the page or scrolling all the way to the bottom of the page and clicking the "Configure" link.

Try it

The Setup Screen
The user can type into one of the three textboxes

Suggestions are offered as the user types.
Both place names and zip code entries are supported

Feel free to try this forecasting site out and even bookmark it for future use

Try it

This article is also available on LinkedIn
https://www.linkedin.com/pulse/ui-upgrade-météo-mobile-weather-forecast-site-sverrir-sigmundarson

Getting hassle free and fast(er) weather forecasts in France (#1)

The Météo-France Android app has been annoying me for the past 6 months with its excessive battery usage and frustrating UI navigation and experience. Finally this week I had enough and decided to come up with something simple that still gives me the same information on my mobile as the rather excellent France Meteorological office prepares and publishes.

Try it

This project has been updated, click here for part two (that discusses the UI redesign) and part three (data enhancements) of this article.

Design

As the French Météo does not publish any of its data in an easily programmable way I decided to do simple screen scraping of their existing forecast website. This is straight-forward enough to do in PHP and actually made considerably easier using the SimpleHTMLDOM library. I highly recommend it. It is the closest I've come to having BeautifulSoup in PHP.

The service is split into two pages: 

API

Scapes and re-renders the Météo data into either a JSON or JSONP format.

Try it

Supports the following URL parameters:

Parameter	Description
&area=	Lowercase name of the region or area you're interested in. E.g. strasbourg, eckbolsheim  or saverne. Defaults to "strasbourg"
&zip=	The zip code for the area. This should correspond to a zip code available in the area used. Defaults to "67000"
&callback=	Optional, if used then the call becomes a JSONP response and this value will hold the name of the client side Javascript function that should be called when the call returns.

Relies on support data from the following resources:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Uvs.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Temps.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/legend.css

Website presentation

Simple HTML/Javascript front on top of the forementioned API functionality. Supports both AREA and ZIP parameters described above.

Try it

Examples

Default call in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php

Eckbolsheim weather info in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php?area=eckbolsheim&zip=67201

Saverne API response:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?area=saverne&zip=67700

Default API response with JSONP callback:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?callback=weatherdatafunction



This acticle is also available on LinkedIn:
https://www.linkedin.com/pulse/getting-hassle-free-faster-weather-forecasts-france-sigmundarson

Filtering the ESA Small & Medium Sized Enterprises on a map (#2)

The project discussed in this article has now been updated with newer data see post 3.

As a follow up on the work I did in an earlier post I decided to go ahead and expand my initial implementation quite a bit in an attempt to make it more useful for people looking for ESA vetoed contracting firms.

Try it

So I implemented two major enhancements:

1. Richer company info

Now when you click on a marker it will show you a much richer details about each company. Specifically its primary fields of expertise and contact information (in case you're interested in working there).

2. Filtering of companies

You can now hover over a little blue icon in the lower right-hand corner to open up a filter menu. By selecting individual filters in this menu you can limit the markers shown on the map to only companies that specialize in these fields.

Super useful. Wow
!

ESA Small & Medium Sized Enterprises on a map

This article has been expanded and updated in post 2 and post 3.

Recently I had some business on the European Space Agency (ESA) SME Database. This database lists small and medium sized companies that have been cleared or granted privileges to work as contractors for the ESA.

The primary public access to this database is a long tabular list of company names and addresses. This data can be better presented today by overlaying it on an interactive map instead of a list. Using such a format can also make it easier for people to browse and investigate opportunities in specific areas.

So, in the spirit of being the change you want to see, I set out to create an offline parser (.NET) to source the list of companies from the SME database and then a simple PHP website to render the results onto a nice modern map (in this case Google Maps).

The change

First version is done and you are welcome to use it:

Try it

This first version is very basic. Currently the following improvements are planned

1. More details about companies shown when the markers are clicked
        2015-01-11: Page now shows company fields of expertise as well as contact information 
2. Filter companies by their sector and project types
3. Select companies and print a list for future reference (in case you're job hunting)

My ongoing list

My totally unoriginal list of things I'm still learning:

1. It is easier to ask for forgiveness than permission. Just do it!
     
2. It's better to fix problems than trying to prevent errors. So try to fail as fast and as early as you possibly can and learn from it.
     
3. It's not your baby, so leave your ego at the door. Your work is not "yours" but your employer's. So as soon as it's been typed, it can and should be criticized and worked on by anyone.
     
4. Everyone does things slightly differently. If things are correct, avoid judging other peoples work purely on style or differences from how "you would do it".

Improving the TFL Open Data Feeds and APIs #2

So following up from my earlier post about the TFL Data APIs I don't think I was being fair towards the TFL dev team. The API that is currently provided, called TrackerNet, was made available in August 2011 and is now going on its 3rd year. The API is still quite horrendous by modern (and 2011) standards but granted is the first step publishing the vast network of disparate data-sources in use used within the TFL after its merger in 2000.

Little did I know that there is indeed an impressive brand new API (api.beta.tfl.gov.uk) powering the newly redesigned tfl.gov.uk website that was launched in late March 2014. This I discovered by accident when investigating the seeming information difference between station disruptions in the TrackerNet API and what was displayed on the TFL website.

Below are images showing the difference in information provided for the same tube station "Paddington" between the old TrackerNet API and what is shown on the TFL website itself.

TrackerNet results for Paddington station

TFL website information for Paddington station
(only the past paragraph is what is available in TrackerNet)

The new API

api.beta.tfl.gov.uk offers information about it seems every aspect of the TFL network and facilities. The developers seem to have taken the approach to developing the new TFL site as simply one presentation layer on top of this vast and rather well performing API.

The guidelines ask that developers that want to leverage the API sign up for a key and an app-id that should be submitted with every query. But all API calls seem to work just fine without supplying these values. 

One minor gripe that I have with the API is that it is perhaps overly verbose with average data result sizes close to 1MB in size (uncompressed) . Comparing the size of the boris bikes between the APIs:

Old API		New API
Compressed	Uncompressed	Compressed	Uncompressed
31.8KB	266KB	110KB	1.6MB

But this is a small price to pay today for a much richer and consolidated API experience.

Unfortunately the data published by the old API is not compatible with the new one and most notibly is the station and tube line names that are different. Luckily developers such as Matt Hinchliffe have already gathered some really useful support data to make the data and its use easier.

Now I guess I should update my bike and bus websites to leverage this new found API gold :)

Improving the TFL Open Data Feeds and APIs

This blog has moved to https://blog.sverrirs.com/2014/07/improving-tfl-open-data-feeds-and-apis.html

You will be automatically redirected.

Programming Garmin Nüvi 1690 SatNav using C#

Update Oct 2015
I've now written a web-tool that automagically converts
Google Maps directions to GPX format. Its pretty awesome!

Take me to the new article
 

Oh dear, this is going to be one of these things that make you go d'oh.

If you just want to know how the tech bits and skip the narrative you can jump to the Solution section below.

I have a now rather ancient Garmin SatNav, a Nüvi 1690. I bought it a few years back when I realized that renting a GPS device with my rental car would be more expensive after the second time than buying a new device.

This little device has paid for itself many times over by now. However the process of getting data points to the device prior to a trip has always been a very complicated and round-about way one which I've vowed to make smoother every time I go through it.

My Process

I like to plan my medium to long trips using either Google Maps or Google Earth. These are excellent tools to find hotels, parking, places of interest and planning which roads to take. However neither of these two tools can handle the necessary GPS data formats (GPX being the most common) that the Garmin tools do. The Google tools (understandably if you remember their original acquisition source) only deal with KML and KMZ data formats.

This has required me to take the following steps:

1. Plan points and routes in Google Earth/Maps
2. Save waypoint data from Google Earth to KML file
3. Convert KML file to GPX file (excellent tool for this is GPS Visualizer)
4. Handle any data format errors
5. Use any of the many Garmin tools to import the data to the device
1. Sometimes I've used Google Map's "Send to GPS" feature for single points. This feature seems to have been removed from the new Google Maps.
2. BaseCamp has very good support for importing waypoints and routes. It regretfully lacks support to delete said points from the Nüvi device though.
3. myGarmin web page used to have an import feature with their GarminConnector plugin (but this is gone now)
4. MapSource had some support for this (but this tool is discontinued now and cannot be downloaded)
5. Garmin Express is simply useless when it comes to uploading waypoints (it is atrociously simple)

This time the only tool available to me was the rather nice BaseCamp tool. However I ran into some problems as I was unable to delete some previously loaded waypoints off my device. BaseCamp could not delete anything and even after deleting everything I could from the device menus there were still some points that lingered in there.

Deleting Everything

When using Windows 7 and newer the Garmin device shows up as an external USB storage drive. So after spending a very unsuccessful hour trying to clean my previous data from the device I finally decided to open up Windows Explorer and go hunting through the mounted drive.

I finally found that the GPX folder actually held an archive of my waypoints and trips that were somehow read by the device. Deleting every gpx file in this folder removed all custom points from the device (I also removed the Archive folder for good measure).

The Garmin USB Protocol

My perception of the device communication standard has always been the thing that has scared me away from actually diving in and creating a helper application. I decided to embark on trying to understand and leverage this protocol to automate my process as much as I could.

Cutting a very long story short, after numerous attempts using their Garmin SDK library and reverse engineering their .NET libraries I had made no real progress. The SDK samples I had, strangely just indicated that there seemed to be no compatible GPS device connected to my machine. This was weird as I could easily verify that the device was there by using their BaseCamp tool and the web-based GarminConnector (showed it connected and accessible).

Solution: USB Mass Storage Mode

Although embarrassing I, in desperation, went back to Windows Explorer and scoured through the device that was mounted as my G:\ drive trying to find any hints to how to access it.

I found a rather interesting file under the Garmin\ folder which was named GarminDevice.xml. Looking through this file it wasn't long until the shear stupidity of what I had been trying to do hit me.

I was already accessing the device! I had previously removed waypoints by deleting the files from the GPX directory, I could just as well have added a file in there with my new data!

And sure enough, this is indeed the case for a large range of Garmin devices. Below is an explanation of the relevant GarminDevice.xml section:

The GpsData DataType element also contains a File element with TransferDirection=InputToUnit which specifies a file type, extension and file path. The application places a GPS Exchange (.gpx) file containing routes, tracks and waypoints in the directory path specified. This file will be processed by the GPS device upon exit from mass storage mode.

This has reduced the problem to a simple FileIO and XML parsing exercise which is trivial to implement using any modern programming language (e.g. C# or Python).

Stay tuned for my new app :)

Failure resiliency and retry delay planning

When dealing with networked or external data sources I've learned the hard way that all code should be designed as to expect failures. It is significantly cheaper and easier to bake a graceful handling of errors this from the start rather than attempt to do it later on. A common first step in combating failures and providing resiliency is to have your logic retry an operation in case of a failure.

But how should you retry?

Simple retries

The most simplistic retry mode is to simply surround all your code with a while loop that executes your block a predefined number of times, ala:

int retry = 0;

do
{
   // Operation
   if( true == MyFlakyOperation() )

       break;
}

while ( ++retry < 6 )

The problem with this approach is that it completely ignores the most likely underlying reason for the failure. Congestion or resource load on the remote end could be causing your calls (and many others) to intermittently fail as the server cannot handle the incoming requests. In this case your naive implementation might actually be contributing to making the situation even worse.

So how do we solve this?

Spacing out retries

One common approach to spacing out retries is called Exponential backoff. This algorithm uses a predefined feedback (e.g. retry count) to systematically increase wait times between repeated executions of the same code to avoid congestion.

Example of exponential spacing based on 4sec base wait time.
The vertical bars indicate retry points.

The idea is that with every additional retry that is required it is more likely that the system we're communicating with is heavily congested and needs more "breathing" space to resolve its current backlog of requests.

Example of backoff retries

Below is an example of a very simple C++ algorithm snippet that performs this kind of exponential backoff based on 4sec intervals:

int success = OK;
int retry = 0;
do
{
   // Operation
   success = MyFlakyOperation();

   // Sleep if operation was not success
   if (success != OK)
   {
       int sec = static_cast(std::pow(4, retry));
       std::this_thread::sleep_for(std::chrono::seconds(sec));
   } 
}

while ( ++retry < 6 && success != OK)

In this example my algorithm has a maximum running time with full retry count of a whooping 22 min and 44 seconds! (4+16+64+256+1024 = 1364sec).

How much does the waiting time increase?

Care must be taken when choosing the interval to increment by when using a naive approach as my example above. Below is a table listing the waiting times in seconds for each retry for 2-7 second intervals.

Remember that your maximum running time is the cumulative waiting numbers for all  intervals!

Retry#	2sec	3sec	4sec	5sec	6sec	7sec
1	2	3	4	5	6	7
2	4	9	16	25	36	49
3	8	27	64	125	216	343
4	16	81	256	625	1,296	2,401
5	32	243	1,024	3,125	7,776	16,807
6	64	729	4,096	15,625	46,656	117,649
7	128	2,187	16,384	78,125	279,936	823,543
8	256	6,561	65,536	390,625	1,679,616	5,764,801
9	512	19,683	262,144	1,953,125	10,077,696	40,353,607
10	1,024	59,049	1,048,576	9,765,625	60,466,176	282,475,249

* so using 7 sec as a base and allowing up to 10 retries, the total maximum waiting time will be just shy of 10,5 years!

Simple k-means algorithm in C#

This blog has moved to https://blog.sverrirs.com/2013/08/simple-k-means-algorithm-in-c.html

You will be automatically redirected.

Base64 encoding image files using Python

In an earlier entry I mentioned that I was building a little app to display a bunch of static information. Sometimes I find it immensely useful to embed all resources into a single data-file. Not only does it make handling the data simpler but also retrieving it over the internet.

My data has a bunch of text and a few images as well to go with it. I wrote the code below (in python) to automate embedding of image data directly in with the other text that I will be serving.

@staticmethod
def encode_image_as_base64( image_path, base_path ):
    """ 
        Loads an image from the supplied path (using the base path as a 

        reference) and returns it as a base64 encoded string
    """
    
    # The image path can be a URI with a query string. 

    # Remove any query string elements, basically everything following 

    # a question (?) mark
    qs_split = image_path.split("?")
    image_path = qs_split[0]
    
    file_name = os.path.join( base_path, image_path)
    file_name = urllib.unquote(file_name)
    
    print "Encoding image: "+file_name
    
    encoded_string = ""
    with open(file_name, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())

    return encoded_string

It may not compress all that well but it makes data handling clean and simple.

Calculating File Change List using SHA256

Realised today that I needed a good way to create a change list document for a small app that I am writing. The app has a few hundred screens, each of which is supported by a single data document that contains the information that the screen is showing.

This data is downloaded in individual JSON files from the web the first time the user launches the application. Each of these JSON files can be amended after the app is released. I wanted to be able to provide a quick and easy way for the app to download changelist information to detect if anything needed updating.

I wanted the data to be separated in such a way that if only one file is changed the user only needs to download that one piece of data but not the rest. This is crucial as the entire dataset is just shy of 100MB whereas each individual file averages around 400K.

Python to the rescue

Wrote a little script that creates a single changelist.json file for each of my data files using the built in SHA256 key generation library that ships with python. So simple and quick:

out_file_path = r"C:\outfiles"
jdata = {}
for path, subdirs, files in os.walk(r"C:\myfiles"):
    for name in files:
        if( not name.endswith(".json") or path == out_file_path ):
            continue
            
        json_file_path = os.path.join(path, name)
        
        if( json_file_path == out_file_path ):
            continue
        
        jdata[name.replace(".json", "")] = hashlib.sha256(open(json_file_path, 'rb').read()).hexdigest()
        
out_file_name = os.path.join(out_file_path, "changelist.json")
print "Writing "+str(len(jdata))+" change list items to output file '"+str(out_file_name)+"'"

# Now write the file to the out location and quit
with open(out_file_name, 'w') as outfile:
    json.dump(jdata, outfile, indent=4, separators=(',', ': '))

Python, gotta love it :)

Failover resiliency is indeed important

When building software that is somehow dependant on data provided by someone or something that is out of your direct control I find myself wondering how reliable that service actually is and what the impact is when it eventually goes offline (because they all will at some point...)

I experienced this today with my super simple cycle hire page I built a few weeks for my mobile. The page quickly gives me the status of cycle stations close to my home and work which helps me decide where to go to get home/to work. An example of the page is below:

My simple page relied on the rather excellent free service at http://api.bike-stats.co.uk/ offered by Sachin Handiekar to provide it with the latest docking station information.

But today his service was down and my handy little website showed me nothing
:(

Adding failover providers

After gambling on today's docking stations numbers I decided to hunt for alternative data providers that I could use in case bike-stats were unavailable again in the future. One such alternative service is another excellent one provided at http://borisapi.heroku.com/ by Adrian Short.

A simple way to provide this failover would be to simply call my preferred one first and if that failed to give me valid data I would use the alternative as a backup. Easy enough to hack in php as both sites offered a JSON feed:

function URLopen($url) 
{ 
	// Fake the browser type
	ini_set('user_agent','MSIE 4\.0b2;');
	$dh = fopen("$url",'rb'); 
	$result = stream_get_contents($dh);
	fclose($dh);
	return $result; 
} 

global $json, $isJsonValid, $jsonSource;

// Indicates if we successfully parsed the json data
$isJsonValid = false;

// First try the bike-stats stuff (this might be slow or offline)
$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$json = json_decode($rawjson, true);
if( $json != null )
{		
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}

// As a backup use the heroku json site	
if( !$isJsonValid )
{
	$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
	$json = json_decode($rawjson, true);
		
	if( $json != null )
	{			
		$isJsonValid = true;
		$jsonSource = "borisapi.heroku.com";
	}
}

Now with my super basic failover my problem became that these two providers, although scraping the same data, present the JSON data in slightly different formats.

Going back to my original page I decided to normalise the json data that was used and simply have my failover code perform a quick transformation of the data to my new normalised format. So my final improved data sourcing with failover resiliency and normalisation (wow quite big words there...) ended up looking like so (sorry for the indentation):

try {

$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){		
	$json = array();
	foreach( $jsont["dockStation"] as $station){
		$json[] = array(
		"id"=> $station["@ID"],
		"name" => $station["name"], 
		"bikes" => $station["bikesAvailable"], 
		"docks" => $station["emptySlots"],
		"lat" =>  $station["latitude"],
		"lon" =>  $station["longitude"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}
}catch (Exception $e){
	$isJsonValid = false;
	$json = null;
}

// As a backup use the heroku json site	
if( !$isJsonValid ){
try{
$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){
	$json = array();
	foreach( $jsont as $station){
		$json[] = array(
		"id"=> $station["id"],
		"name" => $station["name"], 
		"bikes" => $station["nb_bikes"], 
		"docks" => $station["nb_empty_docks"],
		"lat" =>  $station["lat"],
		"lon" =>  $station["long"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "borisapi.heroku.com";
}
}catch( Exception $e ){
	$isJsonValid = false;
	$json = null;
}}

It is ugly sure, but does a marvelous job getting me the information I need :)

Want your own map?

A little while ago I designed a simple customization page that anyone can use to create their own light-weight bike stats web page. These pages are best viewed on a mobile device (such as a phone).

Get your own right here :)

Happy cycling...