24 November 2014

ESA Small & Medium Sized Enterprises on a map

This article has been expanded and updated in post 2 and post 3.

Recently I had some business on the European Space Agency (ESA) SME Database. This database lists small and medium sized companies that have been cleared or granted privileges to work as contractors for the ESA.

The primary public access to this database is a long tabular list of company names and addresses. This data can be better presented today by overlaying it on an interactive map instead of a list. Using such a format can also make it easier for people to browse and investigate opportunities in specific areas.

So, in the spirit of being the change you want to see, I set out to create an offline parser (.NET) to source the list of companies from the SME database and then a simple PHP website to render the results onto a nice modern map (in this case Google Maps).

The change

First version is done and you are welcome to use it:

Try it


This first version is very basic. Currently the following improvements are planned

  1. More details about companies shown when the markers are clicked
         2015-01-11: Page now shows company fields of expertise as well as contact information 
  2. Filter companies by their sector and project types
  3. Select companies and print a list for future reference (in case you're job hunting)



19 August 2014

My ongoing list

My totally unoriginal list of things I'm still learning:

  1. It is easier to ask for forgiveness than permission. Just do it!
      
  2. It's better to fix problems than trying to prevent errors. So try to fail as fast and as early as you possibly can and learn from it.
      
  3. It's not your baby, so leave your ego at the door. Your work is not "yours" but your employer's. So as soon as it's been typed, it can and should be criticized and worked on by anyone.
      
  4. Everyone does things slightly differently. If things are correct, avoid judging other peoples work purely on style or differences from how "you would do it".

30 July 2014

Improving the TFL Open Data Feeds and APIs #2

So following up from my earlier post about the TFL Data APIs I don't think I was being fair towards the TFL dev team. The API that is currently provided, called TrackerNet, was made available in August 2011 and is now going on its 3rd year. The API is still quite horrendous by modern (and 2011) standards but granted is the first step publishing the vast network of disparate data-sources in use used within the TFL after its merger in 2000.

Little did I know that there is indeed an impressive brand new API (api.beta.tfl.gov.uk) powering the newly redesigned tfl.gov.uk website that was launched in late March 2014. This I discovered by accident when investigating the seeming information difference between station disruptions in the TrackerNet API and what was displayed on the TFL website.

Below are images showing the difference in information provided for the same tube station "Paddington" between the old TrackerNet API and what is shown on the TFL website itself.

TrackerNet results for Paddington station

TFL website information for Paddington station
(only the past paragraph is what is available in TrackerNet)

The new API

api.beta.tfl.gov.uk offers information about it seems every aspect of the TFL network and facilities. The developers seem to have taken the approach to developing the new TFL site as simply one presentation layer on top of this vast and rather well performing API.

The guidelines ask that developers that want to leverage the API sign up for a key and an app-id that should be submitted with every query. But all API calls seem to work just fine without supplying these values. 

One minor gripe that I have with the API is that it is perhaps overly verbose with average data result sizes close to 1MB in size (uncompressed) . Comparing the size of the boris bikes between the APIs:

Old APINew API
CompressedUncompressedCompressedUncompressed
31.8KB266KB110KB1.6MB

But this is a small price to pay today for a much richer and consolidated API experience.

Unfortunately the data published by the old API is not compatible with the new one and most notibly is the station and tube line names that are different. Luckily developers such as Matt Hinchliffe have already gathered some really useful support data to make the data and its use easier.

Now I guess I should update my bike and bus websites to leverage this new found API gold :)

3 June 2014

Programming Garmin Nüvi 1690 SatNav using C#

Update Oct 2015
I've now written a web-tool that automagically converts
Google Maps directions to GPX format. Its pretty awesome!

Take me to the new article
 

Oh dear, this is going to be one of these things that make you go d'oh.

If you just want to know how the tech bits and skip the narrative you can jump to the Solution section below.

I have a now rather ancient Garmin SatNav, a Nüvi 1690. I bought it a few years back when I realized that renting a GPS device with my rental car would be more expensive after the second time than buying a new device.

This little device has paid for itself many times over by now. However the process of getting data points to the device prior to a trip has always been a very complicated and round-about way one which I've vowed to make smoother every time I go through it.


My Process

I like to plan my medium to long trips using either Google Maps or Google Earth. These are excellent tools to find hotels, parking, places of interest and planning which roads to take. However neither of these two tools can handle the necessary GPS data formats (GPX being the most common) that the Garmin tools do. The Google tools (understandably if you remember their original acquisition source) only deal with KML and KMZ data formats.

This has required me to take the following steps:
  1. Plan points and routes in Google Earth/Maps
  2. Save waypoint data from Google Earth to KML file
  3. Convert KML file to GPX file (excellent tool for this is GPS Visualizer)
  4. Handle any data format errors
  5. Use any of the many Garmin tools to import the data to the device
    1. Sometimes I've used Google Map's "Send to GPS" feature for single points. This feature seems to have been removed from the new Google Maps.
    2. BaseCamp has very good support for importing waypoints and routes. It regretfully lacks support to delete said points from the Nüvi device though.
    3. myGarmin web page used to have an import feature with their GarminConnector plugin (but this is gone now)
    4. MapSource had some support for this (but this tool is discontinued now and cannot be downloaded)
    5. Garmin Express is simply useless when it comes to uploading waypoints (it is atrociously simple)
This time the only tool available to me was the rather nice BaseCamp tool. However I ran into some problems as I was unable to delete some previously loaded waypoints off my device. BaseCamp could not delete anything and even after deleting everything I could from the device menus there were still some points that lingered in there.

Deleting Everything

When using Windows 7 and newer the Garmin device shows up as an external USB storage drive. So after spending a very unsuccessful hour trying to clean my previous data from the device I finally decided to open up Windows Explorer and go hunting through the mounted drive.

I finally found that the GPX folder actually held an archive of my waypoints and trips that were somehow read by the device. Deleting every gpx file in this folder removed all custom points from the device (I also removed the Archive folder for good measure).

The Garmin USB Protocol

My perception of the device communication standard has always been the thing that has scared me away from actually diving in and creating a helper application. I decided to embark on trying to understand and leverage this protocol to automate my process as much as I could.

Cutting a very long story short, after numerous attempts using their Garmin SDK library and reverse engineering their .NET libraries I had made no real progress. The SDK samples I had, strangely just indicated that there seemed to be no compatible GPS device connected to my machine. This was weird as I could easily verify that the device was there by using their BaseCamp tool and the web-based GarminConnector (showed it connected and accessible).

Solution: USB Mass Storage Mode

Although embarrassing I, in desperation, went back to Windows Explorer and scoured through the device that was mounted as my G:\ drive trying to find any hints to how to access it.

I found a rather interesting file under the Garmin\ folder which was named GarminDevice.xml. Looking through this file it wasn't long until the shear stupidity of what I had been trying to do hit me.

I was already accessing the device! I had previously removed waypoints by deleting the files from the GPX directory, I could just as well have added a file in there with my new data!

And sure enough, this is indeed the case for a large range of Garmin devices. Below is an explanation of the relevant GarminDevice.xml section:
The GpsData DataType element also contains a File element with TransferDirection=InputToUnit which specifies a file type, extension and file path. The application places a GPS Exchange (.gpx) file containing routes, tracks and waypoints in the directory path specified. This file will be processed by the GPS device upon exit from mass storage mode.
This has reduced the problem to a simple FileIO and XML parsing exercise which is trivial to implement using any modern programming language (e.g. C# or Python).

Stay tuned for my new app :)

26 May 2014

The Extra Effort


“Perfection is achieved, not when there is nothing more to add, 
but when there is nothing left to take away.” 
– Antoine de Saint-Exupery

There is an interesting research published recently in Social Psychological and Personality Science, "Worth Keeping but Not Exceeding: Asymmetric Consequences of Breaking Versus Exceeding Promises".

It describes research into how promises and contracts are viewed by people, especially how their completion is perceived and how people are likely to react to under- and over-delivering on promises as compared to delivering on exactly what was agreed upon.
"Businesses may work hard to exceed their promises to customers or employees, but our research suggests that this hard work may not produce the desired consequences beyond those obtained by simply keeping promises. [...] The results of our experiments suggest that it is wise to invest effort in keeping a promise because breaking it can be costly, but it may be unwise to invest additional effort to exceed one’s promises. When companies, friends, or coworkers put forth the effort to keep a promise, their effort is likely to be rewarded. But when they expend extra effort in order to exceed those promises, their effort appears likely to be overlooked."

This somewhat confirms what I have experienced during my career when it comes to delivering software. It is the single most important thing to deliver what was originally agreed upon in a working state. A solution with the minimal level of features that each does only exactly what was agreed upon is in most cases much better received than a solution that has a bunch of bells and whistles even though both solutions deliver equally reliable software on the exact same promise.

Pro-Tip

Deliver only what was agreed on and at the right time. Nothing more, nothing less. Never mention any bonus features but consider saving any that may have been implemented as a free surprise a few weeks after a successful delivery on the initial contract.

Publication information:

Worth Keeping but Not Exceeding: Asymmetric Consequences of Breaking Versus Exceeding Promises Ayelet Gneezy and Nicholas Epley
Social Psychological and Personality Science published online 8 May 2014
DOI: 10.1177/1948550614533134

Abstract
Promises are social contracts that can be broken, kept, or exceeded. Breaking one’s promise is evaluated more negatively than keeping one’s promise. Does expending more effort to exceed a promise lead to equivalently more positive evaluations? Although linear in their outcomes, we expected an asymmetry in evaluations of broken, kept, and exceeded promises. Whereas breaking one’s promise is obviously negative compared to keeping a promise, we predicted that exceeding one’s promise would not be evaluated more positively than merely keeping a promise. Three sets of experiments involving hypothetical, recalled, and actual promises support these predictions. A final experiment suggests this asymmetry comes from overvaluing kept promises rather than undervaluing exceeded promises. We suggest this pattern may reflect a general tendency in social systems to discourage selfishness and reward cooperation. Breaking one’s promise is costly, but exceeding it does not appear worth the effort.

3 May 2014

Creating Cascading Menu Items in Windows Explorer 7, 8 and 8.1

After creating a rather useful command-line application the other day I wanted to expose it in a user friendly way through Windows Explorer. I remembered that this could be done quite easily through the Windows Registry but most of the documentation I found on the internet was quite confusing and most only gave me a piece of the puzzle. So after a lot of trial and error I finally managed to create what I wanted to see:


My criteria

  1. Cascading menu with sub items and keyboard shortcuts
  2. Consistent ordering of menu items
  3. Separator to divide the main action from the two sub actions
  4. Menu only visible for certain types of files (in this case *.ini files)
  5. Menu items execute my program via the shell command line

But first, the complete registry code

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini]

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell]

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel]
"MUIVerb"="My &Tool"
"subcommands"=""

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell]

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\a_myactionmain]
@="My Main &Action"
"CommandFlags"=dword:00000040

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\a_myactionmain\command]
@="C:\\tool\\mine.exe -i \"%1\""

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\b_myactionfirst]
@="My Action &One"

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\b_myactionfirst\command]
@="C:\\tool\\mine.exe -i \"%1\" -first"

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\c_myactionsecond]
@="My Action &Two"

[HKEY_CLASSES_ROOT\SystemFileAssociations\.ini\shell\myactiontoplevel\shell\c_myactionsecond\command]
@="C:\\tool\\mine.exe -i \"%1\" -second -debug"

1) The Cascading menu

The name of the key (myactiontoplevel) must be unique and ideally not conflict with other values.

The top level menu must have the string entry "subcommands" with no text in it. Otherwise none of the child menus will show in Explorer as nested.

The "MUIVerb" value controls the text displayed for the menu item (it is also possible to skip this and just use the "default" entry for the registry key. The ampersand (&) defines the shortcut key character (will be underlined when applicable).

All nested commands sit under the \shell\ sub-key defined for the top level menu entry.

2) Consistent ordering

Windows sorts the sub menu items according to their key name (e.g. myactionfirst, myactionsecond). Therefore to ensure the correct ordering of the sub items I prefix them with a sorting character to ensure consistent sorting. The image below shows an example of what happens to the menu items if the sorting prefix is skipped in the code above.

3) Separator

The CommandFlags <dword> entry can be added to instruct Windows to add a separator menu entry. The value is of type EXPCMDFLAGS. To control the placement of the separator use either 0x40 (separator below) or 0x20 (separator above).

Following values are supported, sum to combine features
ValueDescription
0x08Entry is a separator. Consecutive separators are collapsed into a single separator
0x10Shows a UAC shield next to the menu item.
0x20Show a separator above this item
0x40Show a separator below this item

4) Showing menu item only for *.ini files

To enable the menu item only for ini files.,the root \shell\ entry can be placed either under HKEY_CLASSES_ROOT\SystemFileAssociations\.ini (as shown above) or under HKEY_CLASSES_ROOT\inifile\. There is not a clear distinction between which of these keys is better placement for your custom entry. The \inifile\ entry should already have some shell keys defined by default and these commands seem to be placed before the \SystemFileAssociations\.ini entries in the menu. Otherwise I have not noticed any other difference than that.

Do not place the registry entries under HKEY_CLASSES_ROOT\.ini as this location is only used to map to HKEY_CLASSES_ROOT\inifile (as evident by the default value). If you place any \shell\ entries under HKEY_CLASSES_ROOT\.ini they will not show up.

5) Executing a custom application

This one is relatively easy to do as you only need to point the \command\ key to the location of your executable and then place any parameters you want in the command line. By default the %1 or %0 contain the selected file path (make sure you place all %1 type of variables in quotes).

Other % macros are:
MacroDescription
%0 or %1the first file parameter. For example “C:\docs\supersecret\world.txt”. Generally this should be in quotes and the applications command line parsing should accept quotes to disambiguate files with spaces in the name and different command line parameters. Just note that if the path is not a file but a directory path then the trailing backslash will not be included, e.g. C:\docs\ will be C:\docs
%<N>(where N is 2 - 9), replace with the nth parameter
%sshow command
%hhotkey value (number, 0 if not set)
%iIDList stored in a shared memory handle is passed here. Example ":1084009376:4272"
%llong file name form of the first parameter. Note win32 applications will be passed the long file name, win16 applications get the short file name. Specifying %L is preferred as it avoids the need to probe for the application type.
%d desktop absolute parsing name of the first parameter (for items that don’t have file system paths)
%vfor verbs that are none implies all, if there is no parameter passed this is the working directory or the file path.
%wthe working directory
%uunknown
%*not supported
%~not supported

Other resources

Explicitly about creating entries in right-click menus on the Desktop and the My Computer Icon.
http://www.askvg.com/add-cascading-menus-for-your-favorite-programs-in-windows-7-desktop-context-menu/

Links to various other types of menus that can be customized
http://www.askvg.com/ultimate-tutorial-to-customize-desktop-context-menu-in-windows-vista-server-2008-and-windows-7/