Update #2 to Google Maps Directions to GPX data

I've applied several major updates to the tool discussed in this article. Update #1, Update #2, Update #3, Update #4 and Original post.

The Google Maps Directions to GPX has received its second update.

Try it

Change log

• Support for short urls (e.g. "goo.gl/maps/..." links).
• Support for multi destination directions.
• Waypoints will now get a full address into their <desc> field if available.
• Option to exclude creation of waypoints (external to the route).
• Option to provide a custom name for the route.
• Improved error handling and validation of Google Maps links.
• Addressing an issue where if URL is missing sections the Google service will estimate the destination (this will most likely cause an incorrect route to be generated).

Changes to JSON output only 

These have not yet been integrated into the GPX format.

• Inclusion of Google StreetView imagery where available.
• Preliminary support for elevation profile data (only for walking and cycling directions).
• Turn information available in markup format ("step" field).
• List of all countries the route passes through.
• List of all destinations in route (in addition to start and finish).

Try it


Map Link with Multi Destination Directions

Map Link with Cycling Directions (will have elevation data)

Update #1 to Google Maps Directions to GPX data

I've applied several major updates to the tool discussed in this article. Update #1, Update #2, Update #3, Update #4 and Original post.

In a previous post I shared an experimental tool that I had been tinkering with that converts Google directions to GPX format.

After quite a few interesting suggestions and discussions with readers I want to present Update #1 to this tool.

Try it

Major changes

1. New and more user friendly UI with error checking.
2. More control over output format and output file types (the tool now defaults to generating a file that has the broadest compatibility with the GPX standard).
3. Auto-generation of unique route point names (format "RPxxx").
4. Option to include the full direction text with each route point in the <desc> field.
5. Option to include a brief next turn information with each route point in the <cmt> field.
6. Optional JSON response for future web-development

The new UI with all options

Try it

Öryggisafritun og umbreyting á DVD efni til einkanota

This blog has moved to https://blog.sverrirs.com/2015/08/oryggisafritun-og-umbreyting-dvd-efni.html

You will be automatically redirected.

Converting Google Maps Directions to GPX data

This blog has moved to https://blog.sverrirs.com/2015/08/converting-google-maps-directions-to.html

You will be automatically redirected.

New forecast data upgrade for the Météo website (#5)

This blog has moved to https://blog.sverrirs.com/2015/07/new-forecast-data-upgrade-for-meteo.html

You will be automatically redirected.

Updating the ESA Small & Medium Sized Enterprises map (#3)

Since I created the ESA contractor map in November I've been surprised to see that I seem to get regular traffic to it.

I must admit that when I built this first I was not concerned with tracking how often the underlying SME database changes on the ESA side and the project doesn't really have any built in stats tracking that (yet).

Neither have I found time to fully automate the extraction and parsing of the data from the SME database. This code to produce and prepare the background data all lives in a C# application that currently I need to run manually to get the updated lists of companies and fields.

Out of curiosity I decided to run an update on the underlying data today and was surprised at the number of new and updated entries that were located.

View Latest

The Changes

All in all there where 39 major changes to the registry including new, deleted and updated companies.

I've updated the map markers and fixed a few minor bugs relating to mobile clients.

I also introduced a change list section. This section is shown in the lower right hand corner of the map and lists the changes that I have made to the map source data.

I should do this to all of my applications actually, it is really handy to track what has changed and when.


A version of this article is also available on LinkedIn
https://www.linkedin.com/pulse/esa-small-medium-sized-enterprises-interactive-map-sigmundarson

Météo France, converting between different wind speed units (#4)

The small Météo France site I created has been updated with a new parameter that can be used to customize the unit that windspeed is presented as.

The u= parameter

The new parameter that controls the windspeed is ?u= and accepts the following values:

Value	Comment	Examples
u=bf	Beaufort scale of windforce between 0 and 12. More information about conversions. Unit "F"	Try it
u=ms	Meters per second. Unit "m/s".	Try it
u=kn	Knots. Unit "kn".	Try it
u=kmh	Kilometers per hour. Unit "km/h". This is the current default.	Try it

More information on conversion and meaning of these different scales can be found here and on Wikipedia.

This parameter works both for the website as well as the weather API service.

Examples

Knots
Kn

Beaufort Force
F

Kilometers per hour
Km/h

Meters per second
m/s

Expanding the Météo weather forecast site (#3)

I've discussed the work I've done to build a simpler weather UI for the French météo service in earlier posts (first entry and second entry).

We've been having exceptionally hot weather and I decided to expand the current web app with more information that is provided by the French weather service.

Try it

Severe weather warnings

The Météo service publishes a separate list of areas that have been issued with severe weather warnings (vigilance météorologique).  I thought it might be helpful to have this information readily available to the user.

After fruitlessly attempting to scrape the HTML on this website and being unable to locate the same elements that the browser debuggers indicated were rendered. I discovered to my dismay that most of the content on these pages are being rendered by a client side JavaScript. Why, I will probably never understand as the code is quite the mess to look at.

But luckily I noticed that the whole rendering depended on a single XHR call to an external XML service that luckily didn't require any sort of CrossSite authentication. So I simply went straight to the source and saved countless hours of debugging and fiddling with bad HTML. (http://vigilance.meteofrance.com/data/NXFR33_LFPW_.xml)

Unfortunately the source doesn't really represent the data in any easily understandable way so considerable digging around in the JavaScript code was needed to fully understand all the numerical codes presented in the XML.

The raw XML data from the Météo service

The rest of the JavaScript code and HTML gave me the pieces to understand the colour coding and images to render for each risk type and severity level.

Severe gales / strong wind

While digging around and trying to figure the weather warnings out I realized that my earlier design did not account for displaying warning and windspeed numbers for any forcasted and expected wind gales. This information was already available from my earlier parsing project and needed only minimal tweaking to get correct.

This information is now presented either within the day or as a whole day event, depending on which is applicable.

The new information elements.
Click for a larger version

Try it

This article is also available on LinkedIn:
https://www.linkedin.com/pulse/expanding-météo-weather-forecast-site-sverrir-sigmundarson

Upgrading the UI for the Météo weather forecast site (#2)

After building the initial version of the Météo weather page and confirming that the data was all correctly handled I went about sprucing the UI up a bit.

Try it

This project has been updated, click here for part three (data enhancements) of this article.

Not only was the original design purely a functional one and not particularly attractive or user friendly but it was also missing a few key data points such as intra-day weather and likelihood of rain.

The Old UI

The tools

With the help of JQuery and the JQuery UI I managed to wrestle it into a usable and aesthetically pleasing look and feel.

I ended up using the JQuery Accordion without too much customization. This widget suits the website quite well as I wanted to be able to show an overview of the next 10 days at a glance but then allow the user to drill down into each day to see the intra-day forecast in more detail.

The final UI design

The accordion is closed upon first loading the website but then the user can tap each day to drill down and see more detail data.

The initial view of the website
Accordion is closed

User has expanded a day and intra-day forecast is visible
Accordion is open

For longer term forecasts additional information is available on how confident the Météo weather model is about the prediction as well as a rough prediction of the likelihood of precipitation.

Extra data points are available for days further into the future

Customizing the forecast area

A new addition was the setup screen. This screen can be used to change the forecast area to a different city or area. This page is available by either clicking the name of the current area at the top of the page or scrolling all the way to the bottom of the page and clicking the "Configure" link.

Try it

The Setup Screen
The user can type into one of the three textboxes

Suggestions are offered as the user types.
Both place names and zip code entries are supported

Feel free to try this forecasting site out and even bookmark it for future use

Try it

This article is also available on LinkedIn
https://www.linkedin.com/pulse/ui-upgrade-météo-mobile-weather-forecast-site-sverrir-sigmundarson

Getting hassle free and fast(er) weather forecasts in France (#1)

The Météo-France Android app has been annoying me for the past 6 months with its excessive battery usage and frustrating UI navigation and experience. Finally this week I had enough and decided to come up with something simple that still gives me the same information on my mobile as the rather excellent France Meteorological office prepares and publishes.

Try it

This project has been updated, click here for part two (that discusses the UI redesign) and part three (data enhancements) of this article.

Design

As the French Météo does not publish any of its data in an easily programmable way I decided to do simple screen scraping of their existing forecast website. This is straight-forward enough to do in PHP and actually made considerably easier using the SimpleHTMLDOM library. I highly recommend it. It is the closest I've come to having BeautifulSoup in PHP.

The service is split into two pages: 

API

Scapes and re-renders the Météo data into either a JSON or JSONP format.

Try it

Supports the following URL parameters:

Parameter	Description
&area=	Lowercase name of the region or area you're interested in. E.g. strasbourg, eckbolsheim  or saverne. Defaults to "strasbourg"
&zip=	The zip code for the area. This should correspond to a zip code available in the area used. Defaults to "67000"
&callback=	Optional, if used then the call becomes a JSONP response and this value will hold the name of the client side Javascript function that should be called when the call returns.

Relies on support data from the following resources:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Uvs.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Temps.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/legend.css

Website presentation

Simple HTML/Javascript front on top of the forementioned API functionality. Supports both AREA and ZIP parameters described above.

Try it

Examples

Default call in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php

Eckbolsheim weather info in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php?area=eckbolsheim&zip=67201

Saverne API response:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?area=saverne&zip=67700

Default API response with JSONP callback:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?callback=weatherdatafunction



This acticle is also available on LinkedIn:
https://www.linkedin.com/pulse/getting-hassle-free-faster-weather-forecasts-france-sigmundarson

Improving the TFL Open Data Feeds and APIs #2

So following up from my earlier post about the TFL Data APIs I don't think I was being fair towards the TFL dev team. The API that is currently provided, called TrackerNet, was made available in August 2011 and is now going on its 3rd year. The API is still quite horrendous by modern (and 2011) standards but granted is the first step publishing the vast network of disparate data-sources in use used within the TFL after its merger in 2000.

Little did I know that there is indeed an impressive brand new API (api.beta.tfl.gov.uk) powering the newly redesigned tfl.gov.uk website that was launched in late March 2014. This I discovered by accident when investigating the seeming information difference between station disruptions in the TrackerNet API and what was displayed on the TFL website.

Below are images showing the difference in information provided for the same tube station "Paddington" between the old TrackerNet API and what is shown on the TFL website itself.

TrackerNet results for Paddington station

TFL website information for Paddington station
(only the past paragraph is what is available in TrackerNet)

The new API

api.beta.tfl.gov.uk offers information about it seems every aspect of the TFL network and facilities. The developers seem to have taken the approach to developing the new TFL site as simply one presentation layer on top of this vast and rather well performing API.

The guidelines ask that developers that want to leverage the API sign up for a key and an app-id that should be submitted with every query. But all API calls seem to work just fine without supplying these values. 

One minor gripe that I have with the API is that it is perhaps overly verbose with average data result sizes close to 1MB in size (uncompressed) . Comparing the size of the boris bikes between the APIs:

Old API		New API
Compressed	Uncompressed	Compressed	Uncompressed
31.8KB	266KB	110KB	1.6MB

But this is a small price to pay today for a much richer and consolidated API experience.

Unfortunately the data published by the old API is not compatible with the new one and most notibly is the station and tube line names that are different. Luckily developers such as Matt Hinchliffe have already gathered some really useful support data to make the data and its use easier.

Now I guess I should update my bike and bus websites to leverage this new found API gold :)

The Extra Effort

“Perfection is achieved, not when there is nothing more to add, 

but when there is nothing left to take away.” 

– Antoine de Saint-Exupery

There is an interesting research published recently in Social Psychological and Personality Science, "Worth Keeping but Not Exceeding: Asymmetric Consequences of Breaking Versus Exceeding Promises".

It describes research into how promises and contracts are viewed by people, especially how their completion is perceived and how people are likely to react to under- and over-delivering on promises as compared to delivering on exactly what was agreed upon.

"Businesses may work hard to exceed their promises to customers or employees, but our research suggests that this hard work may not produce the desired consequences beyond those obtained by simply keeping promises. [...] The results of our experiments suggest that it is wise to invest effort in keeping a promise because breaking it can be costly, but it may be unwise to invest additional effort to exceed one’s promises. When companies, friends, or coworkers put forth the effort to keep a promise, their effort is likely to be rewarded. But when they expend extra effort in order to exceed those promises, their effort appears likely to be overlooked."

This somewhat confirms what I have experienced during my career when it comes to delivering software. It is the single most important thing to deliver what was originally agreed upon in a working state. A solution with the minimal level of features that each does only exactly what was agreed upon is in most cases much better received than a solution that has a bunch of bells and whistles even though both solutions deliver equally reliable software on the exact same promise.

Pro-Tip

Deliver only what was agreed on and at the right time. Nothing more, nothing less. Never mention any bonus features but consider saving any that may have been implemented as a free surprise a few weeks after a successful delivery on the initial contract.

Publication information:

Worth Keeping but Not Exceeding: Asymmetric Consequences of Breaking Versus Exceeding Promises Ayelet Gneezy and Nicholas Epley
Social Psychological and Personality Science published online 8 May 2014
DOI: 10.1177/1948550614533134

Abstract
Promises are social contracts that can be broken, kept, or exceeded. Breaking one’s promise is evaluated more negatively than keeping one’s promise. Does expending more effort to exceed a promise lead to equivalently more positive evaluations? Although linear in their outcomes, we expected an asymmetry in evaluations of broken, kept, and exceeded promises. Whereas breaking one’s promise is obviously negative compared to keeping a promise, we predicted that exceeding one’s promise would not be evaluated more positively than merely keeping a promise. Three sets of experiments involving hypothetical, recalled, and actual promises support these predictions. A final experiment suggests this asymmetry comes from overvaluing kept promises rather than undervaluing exceeded promises. We suggest this pattern may reflect a general tendency in social systems to discourage selfishness and reward cooperation. Breaking one’s promise is costly, but exceeding it does not appear worth the effort.

Base64 encoding image files using Python

In an earlier entry I mentioned that I was building a little app to display a bunch of static information. Sometimes I find it immensely useful to embed all resources into a single data-file. Not only does it make handling the data simpler but also retrieving it over the internet.

My data has a bunch of text and a few images as well to go with it. I wrote the code below (in python) to automate embedding of image data directly in with the other text that I will be serving.

@staticmethod
def encode_image_as_base64( image_path, base_path ):
    """ 
        Loads an image from the supplied path (using the base path as a 

        reference) and returns it as a base64 encoded string
    """
    
    # The image path can be a URI with a query string. 

    # Remove any query string elements, basically everything following 

    # a question (?) mark
    qs_split = image_path.split("?")
    image_path = qs_split[0]
    
    file_name = os.path.join( base_path, image_path)
    file_name = urllib.unquote(file_name)
    
    print "Encoding image: "+file_name
    
    encoded_string = ""
    with open(file_name, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())

    return encoded_string

It may not compress all that well but it makes data handling clean and simple.

Calculating File Change List using SHA256

Realised today that I needed a good way to create a change list document for a small app that I am writing. The app has a few hundred screens, each of which is supported by a single data document that contains the information that the screen is showing.

This data is downloaded in individual JSON files from the web the first time the user launches the application. Each of these JSON files can be amended after the app is released. I wanted to be able to provide a quick and easy way for the app to download changelist information to detect if anything needed updating.

I wanted the data to be separated in such a way that if only one file is changed the user only needs to download that one piece of data but not the rest. This is crucial as the entire dataset is just shy of 100MB whereas each individual file averages around 400K.

Python to the rescue

Wrote a little script that creates a single changelist.json file for each of my data files using the built in SHA256 key generation library that ships with python. So simple and quick:

out_file_path = r"C:\outfiles"
jdata = {}
for path, subdirs, files in os.walk(r"C:\myfiles"):
    for name in files:
        if( not name.endswith(".json") or path == out_file_path ):
            continue
            
        json_file_path = os.path.join(path, name)
        
        if( json_file_path == out_file_path ):
            continue
        
        jdata[name.replace(".json", "")] = hashlib.sha256(open(json_file_path, 'rb').read()).hexdigest()
        
out_file_name = os.path.join(out_file_path, "changelist.json")
print "Writing "+str(len(jdata))+" change list items to output file '"+str(out_file_name)+"'"

# Now write the file to the out location and quit
with open(out_file_name, 'w') as outfile:
    json.dump(jdata, outfile, indent=4, separators=(',', ': '))

Python, gotta love it :)

Failover resiliency is indeed important

When building software that is somehow dependant on data provided by someone or something that is out of your direct control I find myself wondering how reliable that service actually is and what the impact is when it eventually goes offline (because they all will at some point...)

I experienced this today with my super simple cycle hire page I built a few weeks for my mobile. The page quickly gives me the status of cycle stations close to my home and work which helps me decide where to go to get home/to work. An example of the page is below:

My simple page relied on the rather excellent free service at http://api.bike-stats.co.uk/ offered by Sachin Handiekar to provide it with the latest docking station information.

But today his service was down and my handy little website showed me nothing
:(

Adding failover providers

After gambling on today's docking stations numbers I decided to hunt for alternative data providers that I could use in case bike-stats were unavailable again in the future. One such alternative service is another excellent one provided at http://borisapi.heroku.com/ by Adrian Short.

A simple way to provide this failover would be to simply call my preferred one first and if that failed to give me valid data I would use the alternative as a backup. Easy enough to hack in php as both sites offered a JSON feed:

function URLopen($url) 
{ 
	// Fake the browser type
	ini_set('user_agent','MSIE 4\.0b2;');
	$dh = fopen("$url",'rb'); 
	$result = stream_get_contents($dh);
	fclose($dh);
	return $result; 
} 

global $json, $isJsonValid, $jsonSource;

// Indicates if we successfully parsed the json data
$isJsonValid = false;

// First try the bike-stats stuff (this might be slow or offline)
$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$json = json_decode($rawjson, true);
if( $json != null )
{		
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}

// As a backup use the heroku json site	
if( !$isJsonValid )
{
	$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
	$json = json_decode($rawjson, true);
		
	if( $json != null )
	{			
		$isJsonValid = true;
		$jsonSource = "borisapi.heroku.com";
	}
}

Now with my super basic failover my problem became that these two providers, although scraping the same data, present the JSON data in slightly different formats.

Going back to my original page I decided to normalise the json data that was used and simply have my failover code perform a quick transformation of the data to my new normalised format. So my final improved data sourcing with failover resiliency and normalisation (wow quite big words there...) ended up looking like so (sorry for the indentation):

try {

$rawjson = URLopen("http://api.bike-stats.co.uk/service/rest/bikestats?format=json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){		
	$json = array();
	foreach( $jsont["dockStation"] as $station){
		$json[] = array(
		"id"=> $station["@ID"],
		"name" => $station["name"], 
		"bikes" => $station["bikesAvailable"], 
		"docks" => $station["emptySlots"],
		"lat" =>  $station["latitude"],
		"lon" =>  $station["longitude"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "api.bike-stats.co.uk";
}
}catch (Exception $e){
	$isJsonValid = false;
	$json = null;
}

// As a backup use the heroku json site	
if( !$isJsonValid ){
try{
$rawjson = URLopen("http://borisapi.heroku.com/stations.json");
$jsont = json_decode($rawjson, true);
if( $jsont != null ){
	$json = array();
	foreach( $jsont as $station){
		$json[] = array(
		"id"=> $station["id"],
		"name" => $station["name"], 
		"bikes" => $station["nb_bikes"], 
		"docks" => $station["nb_empty_docks"],
		"lat" =>  $station["lat"],
		"lon" =>  $station["long"]
		);
	}
	$isJsonValid = true;
	$jsonSource = "borisapi.heroku.com";
}
}catch( Exception $e ){
	$isJsonValid = false;
	$json = null;
}}

It is ugly sure, but does a marvelous job getting me the information I need :)

Want your own map?

A little while ago I designed a simple customization page that anyone can use to create their own light-weight bike stats web page. These pages are best viewed on a mobile device (such as a phone).

Get your own right here :)

Happy cycling...

HTC Desire S USB Driver on Windows 7 (64bit)

This blog has moved to https://blog.sverrirs.com/2013/04/htc-desire-s-usb-driver-on-windows-7.html

You will be automatically redirected.

Degoogling

Like so many others I feel it is time to wean myself off my dependency on Google for search. It may come as a no surprise to anyone how relying on a single source for information opens you up to significantly biased view of the world and is inherently dangerous to do unconsciously. Although my field of work does not strictly require me to present an unbiased view of matters, recent policy and privacy changes for Google products have made me realise how, on an unconscious level, I immediately turn to Google search when I need to look something up and how that can impact my online experience in seemingly unrelated systems.


Cold Turkey
So I thought my best course of action was to completely remove Google search as my default go-to search provider. To begin with I chose to try out Bing (as it is the only search provider that offers a similar range of products I use from Google, i.e. search and maps). I am still disappointed that I couldn't use Office365 for free for personal use (I'd even settle for ads) but it seems that suite is solely meant for business users (big mistake I think).

So first I had to identify where my searches were originating from. I identified the following three areas in descending order of importance:

1. My keyboard launcher, Launchy
   This is by far the most common way that I perform search on my computers. Fast and efficient but has contributed to this mental association that search == google as I need to type in the first letter of the provider I want and then press tab to enter my search term.
   
    
2. Browser's address bar
   As I am a heavy user of Google's Chrome browser it has become almost natural to just start typing what I want into the address bar and then either press enter to search directly or pick from the drop down list. This is such a natural thing to do that I do it in any browser I use.
   
   
   
   
    
3. Browser's search provider box
   After I stopped using Firefox as much as I used to I've stopped using the search provider box as much. But I still occasionally press Ctrl+K and start typing when I find myself using Firefox now and again. Neither IE or Chrome have this box anymore.
   
    
4. Navigating to the search providers website to perform search
   This I almost never do on my main computers but occasionally do when using my phone to search for things. It is slow and annoying

How Easy Is It?
The single biggest difficulty I see is to change my own behaviour and the association I've made with Google and search, it will be hardest for item 1 as I need to stop typing "g <tab> <search term>" and rather type

"b <tab> <search term>"

Not so hard it seems. But I soon realised how instinctive that "g" had become. Most of the time I had already typed it and pressed <tab> before I realised. So that is going to be a challenge to change. Item 4 is so painful to do anyway that I don't see it as a problem at all (although it would be more of an issue if I used the search button on my Android phone more).

The rest is just a matter of configuring the technology to do what I want it to it seems.

Imagine my surprise when the simplest tool to configure to use Bing by far was Google Chrome and the biggest disappointment and betrayer was my old friend Mozilla Firefox.


Google Chrome 18
Simple. Clicking the spanner/wrench icon in the upper right hand side of the browser window, choosing Settings and simply choosing "Bing" from the dropdown list under Search on the Basics tab.

Voilà, now all my searches through Chrome are directed to the Bing website.

Total clicks: 3
Annoyances: None


Microsoft Internet Explorer 9
Strictly speaking then no configuration is needed out of the box for this browser as it comes with Bing already configured as default. However that is also the only search provider it comes with by default. To install any other is more painful that I would have wanted.

IE'sCog Icon

But to choose a different provider it is necessary to click the cog icon in the upper right hand corner of the browser window

Choose Internet Options, clicking the Settings button in the Search section on the first tab. That opens the Add-ons window which allows you to choose a different search provider. I use the term "allows" in a very generous way as it is near to impossible to see at a glance how to use anything else than Bing. But it is there near the bottom left corner disguised as one of those hard to see links.

It annoyed me that they made it this hard to choose as it is obvious that this was a deliberate UI design choice of theirs, one that probably went through a lot of disccusion, refinement etc before being approved and implemented.

Total clicks: 9
Annoyances: Had to close and re-open the Add-on window for the new search provider to appear in the list and manually close all those windows I had opened. Choosing the provider requires you to open an additional browser window to select (plus an extra window opens to confirm settings for the provider after selecting it).

Edit: 
After digging around a little more my curiosity got the better of me and I clicked the little magnifying glass  icon that is present on the far right in the IE address bar. While this gives you convenient access to your browser history and favourites, I discovered that it also lists your search providers and gives you a quicker way to install a new one (step that completely bypasses all the windows mentioned in the previous method). Much much better and actually more clear what provider you're using than it is in Chrome.

The IE magnifying glass (kudos to whoever added the downward
pointing triangle as I would otherwise never have clicked it)

Lists all the provider you have installed and allows easy admin

Total clicks: 6 (2 if you already have the provider installed)
Annoyances: None apart from the aforementioned annoyance having to open a separate website to choose the provider.


Mozilla Firefox 10

"Et tu, Brute?" - Julius Caesar, on seeing his friend among the assassins.

Many know by now that Google made a large contribution (pay off?) to be and stay the default search provider for the Mozilla Firefox browser. The figures from their recent December 2011 deal are estimated to be around $1 billion dollars just to stay the default search provider for the next three years. That sure is one valuable instance of a textbox control.

However it seems rather straight forward to change the default providers, Firefox even comes installed with a choice of a few search engines as alternatives to Google. Simply clicking on the icon in the search providers box gives easy access to choose a different one

That is until you attempt to search directly from the address box!

After choosing Bing search I could not understand why all my address box searches still sent my search query to Google. There was nothing in the configuration section of the browser that hinted that the two boxes did not use the same setting and why one was using Bing as I requested but the address box refused to do as I asked.

It wasn't until I dove under the covers and opened up about:config (which is an advanced configuration system for the browser and not recommended for the layman user). There I discovered quite the jungle of configuration settings related to search in the browser. The one that interested me the most I highlighted in the screenshot below

Sneaky bastards!

Firefox still retains Google as the default search engine even though I've clearly chosen to use Bing as my search provider. Luckily it is relatively simple to change by double clicking on the browser.search.defaultenginename row and type in "Bing". Sorted, now address bar searching will use Bing as default. Bada <pun>! Still I feel this kind of behaviour is deliberately misleading and under-handed.

Total clicks: 2
Annoyances: Does not change provider for address bar search, only for advanced users to discover and change.

Edit: Apparently Microsoft made a deal with Mozilla to produce a special Firefox with Bing browser.

Opera 11
I admit that I don't use the Opera browser on my computers all that much (almost never). However Opera is the only browser I use on my Android phone and it is simply brilliant there. But on desktop systems there are just to many alternatives that are better and sadly they have copied so many things from Firefox (or vice versa) that it just seems redundant to use when I already don't want to use Firefox.

Opera however suffers from a similar problem as Firefox, namely the disconnect between the search provider chosen in the provider box and the one that is used in its main address box. In addition Opera adds on yet another layer of confusion by adding the third search variant on their Speed Dial startup page.

These three boxes can all be set to different search providers. Confused?

Setting the default search engine to use is also not very straight forward as it should be. Same as with Firefox changing the search provider in the search provider box at the top right has no effect on either the address bar (still points to Google) or the Speed Dial box. You need to click the Opera button in the top left corner, choosing Settings and Preferences. Choose the third tab Search in that window and you are presented with a list of all configured search providers for the browser.

Editing the Bing Search Provider in Opera

To set the default search engine you need to double click the entry you want to set (Bing in this case) and click the "Details" button. Then tick the "Use as default search engine". Here you can also control the rather confusing Speed Dial search, reasons why you would want that different from your default one escape me.

Total clicks: 7
Annoyances: Obscurity and the disconnect between address bar and default search provider. Also the Speed Dial page is just plain confusing. But at least you don't have to dive under the hood to change these settings.


Conclusion
The technology can be changed easy enough (if you know where to look). But changing the learned behaviour is going to be harder, but I will give Bing an exclusive try for the next month.

This difficulty with Google in Firefox especially knowing how much Google is paying them for the install defaults feels like a sad regression back to the late 1990s when Microsoft was throwing its weight around in a similar manner bullying itself in a position to dominate the same market. I thought that showed clearly that all that got them was a lot of bad blood and eventually the involvement of US and European authorities when it became obvious to even the non-techies as they were clearly abusing their dominant market position to stifle competitors.

Perhaps moves such as these are the early indicators that the great cosine wave of technology firm's path to success has reached its maxima and has slowly but surely started its inevitable fall from grace. But I guess that might be better saved to a later post.

GPS points of all bus stops in London

Hope I am not the only one this happens to.

Dream of building the greatest app ever, come up with a decent enough idea, research the available technology to build it and the data needed. Even gather said data and work on converting it to a suitable format. Then realise that it is 5am and you need to get up for work in the morning. 

Anyways, in hopes that I help someone else out there kickstart their project. Here is a handy way to download the entire TFL london bus network, every route, GPS coordinates of every stop right to your doorstep.

1. Download cURL here.

2. Run the following commands 

curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={1,10,100,101,102,103,104,105,106,107,108,109,11,110,111,112,113,114,115,116,117,118,119}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={12,120,121,122,123,124,125,126,127,128,129,13,130,131,132,133,134,135,136,137,138,139,14}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={140,141,142,143,144,145,146,147,148,149,15,150,151,152,153,154,155,156,157,158,159,16,160}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={161,162,163,164,165,166,167,168,169,17,170,171,172,173,174,175,176,177,178,179,18,180,181}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={182,183,184,185,186,187,188,189,19,190,191,192,193,194,195,196,197,198,199,2,20,200,201,202}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={203,204,205,206,207,208,209,21,210,211,212,213,214,215,216,217,219,22,220,221,222,223,224,225}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={226,227,228,229,23,230,231,232,233,234,235,236,237,238,24,240,241,242,243,244,245,246,247,248}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={249,25,250,251,252,253,254,255,256,257,258,259,26,260,261,262,263,264,265,266,267,268,269,27}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={270,271,272,273,274,275,276,277,279,28,280,281,282,283,284,285,286,287,288,289,29,290,291,292}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={293,294,295,296,297,298,299,3,30,300,302,303,305,307,308,309,31,312,313,314,315,316,317,318,319}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={32,320,321,322,323,325,326,327,328,329,33,330,331,332,333,336,337,339,34,340,341,343,344,345,346}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={347,349,35,350,352,353,354,355,356,357,358,359,36,360,362,363,364,365,366,367,368,37,370,371,372}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={375,376,377,379,38,380,381,382,383,384,385,386,387,388,389,39,390,391,393,394,395,396,397,398,399}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={4,40,401,403,404,405,406,407,41,410,411,412,413,414,415,417,418,419,42,422,423,424,425,427,428,43}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={430,432,434,436,44,440,444,45,450,452,453,455,46,460,462,463,464,465,466,467,468,469,47,470,472}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={473,474,476,48,481,482,484,485,486,487,488,49,490,491,492,493,496,498,499,5,50,507,51,52,521,53}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={54,549,55,56,57,58,59,6,60,601,602,603,606,607,608,609,61,611,612,613,616,617,62,621,624,625,626}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={627,628,629,63,632,634,635,639,64,640,641,642,643,646,647,648,649,65,650,651,652,653,654,655,656}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={658,66,660,661,664,665,667,669,67,670,671,672,673,674,675,678,679,68,681,683,685,686,687,688}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={689,69,690,691,692,696,697,698,699,7,70,71,72,73,74,75,76,77,78,79,8,80,81,82,83,85,86,87,88}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={89,9,90,91,917,92,93,931,94,941,95,953,958,96,965,969,97,972,98,99,A10,B11,B12,B13,B14,B15,B16}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={C1,C10,C11,C2,C3,D3,D6,D7,D8,E1,E10,E11,E2,E3,E5,E6,E7,E8,E9,EL1,EL2,G1,H1,H10,H11,H12,H13,H14}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={H17,H18,H19,H2,H20,H22,H25,H26,H28,H3,H32,H37,H9,H91,H98,K1,K2,K3,K4,N1,N11,N13,N133,N136,N137}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={N15,N155,N159,N16,N171,N18,N19,N2,N20,N207,N21,N22,N253,N26,N279,N28,N29,N3,N31,N343,N35,N38,N381}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={N41,N44,N47,N5,N52,N55,N550,N551,N63,N68,N7,N73,N74,N76,N8,N86,N87,N89,N9,N91,N97,N98,P12,P13,P4}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={P5,PR2,R1,R10,R11,R2,R3,R4,R5,R6,R68,R7,R70,R8,R9,RV1,S1,S3,S4,T31,T32,T33,U1,U10,U2,U3,U4,U5,U7}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"
curl.exe "http://www.tfl.gov.uk/tfl/gettingaround/maps/buses/tfl-bus-map/dotnet/FullRoute.aspx?route={U9,UL1,UL2,W10,W11,W12,W13,W14,W15,W16,W19,W3,W4,W5,W6,W7,W8,W9,X26,X68}&run={1,2}" --create-dirs --output "busroutes\#1-#2.txt"