When Written: Feb 2008
There is an old saying ‘You get nothing for free’ and in the case of Google maps its API and Geo-coding of Postcodes, that is exactly what you get; ‘nothing’. Now before everyone emails me to say what a wonderful free service Google Maps is, (a sentiment that in the main I agree with), let me explain. I wanted to add a small fun enhancement to one of our websites the other day and I thought it would be fun to put a Goggle map with push pins showing the location of venues on it. This article is about my journey to achieve this; a journey that was at times very bumpy, with several dead ends, but which resulted in a couple of new contacts and a final working web page.
My first concern was how to locate these venues? Using post codes was the obvious choice; however mapping technology uses longitude and latitude to position a marker (also known as a ‘push pin’) on the map, so some way of converting a post code to longitude and latitude was required. A look through the documentation for the Google Maps API showed that it had a way of taking a post code and returning this information. The process is known as ‘Geo-Coding’. It tends to be fairly process heavy on the server and so if you want to do this for a lot of locations you need to approach the problem from a different angle which I will describe later.
However at this point in the development I was just concerned with a couple of locations. It was also at this point I discovered that the Geo-Coding capabilities of the Google API are so broken to render it totally useless! What is the point of putting location makers in the form of push pins on a map if these markers are off by a couple of miles! Yes that is what happened. I couldn’t believe it myself nor did my ‘Google fan’ colleagues. How could Google have a function that was so comprehensively broken to the point of being useless. I decided to do some testing. First I grabbed my Garmin GPS and got it to tell me the Longitude and Latitude of my test postcode which is the office here. This would give me the ‘real’ location. Next I checked the location returned by my Google Maps API, this is what I got:
Garmin GPS: 52.06609 : 0.71071
Google API: 52.04560 : 0.77727
Google maps does post code look up perfectly but…
Using the Google API the post code lookup is broken
I have also included a couple of screen shots to give an idea of what this level of inaccuracy means to an actual map. Now remember that here I am talking about the Google Maps API which is what you use if you want to use Google Maps with your own code and not the standard Google maps, where if you enter a post code you will get a pretty accurately placed push pin showing its location. So why the difference? I emailed Google’s PR department who forwarded my message to their technical department who replied that they were aware of this inaccuracy. Apparently this is caused by the API using a different way of looking up post codes than the main Google Maps. Strange I thought. So the next thing was to try Google Map’s main competitor in the market of free to use mapping software: Microsoft’s Virtual Earth. Now apparently this also had some issues with Geo-Coding from post codes but in the latest release ( ver 6 ) they claim to have improved this. So after an hour or so I had some code up and running that queried Virtual Earth which returned the Longitude and Latitude from my test post code. The results were promising:
Garmin GPS: 52.06609 : 0.71071
Microsoft Virtual Earth API: 52.06605: 0.711029
As you can see, this is a much more accurate result. So what was the reasoning behind this limitation on Google’s part? Well, in their email replies they were very cagy, and whilst they agree that it needs fixing, they can not give a time scale for this. The reasons they gave for the post code lookup being broken was due to ‘data usage restrictions’. Now this sounded like a licensing restriction to me, and when I asked Google’s Laura Scott if this was the case she replied that ‘we ( Google ) do not comment on the specifics of our relationships with data partners’. So that’s probably a yes then.
So why would a company like Google have such a problem? After some investigation it looked like it was down to costs. I started looking further into this and found an Open source file of basic UK postcode information which was free, however this only gives a rough location as it just works on the first group of three or four letters. There are some mathematical tricks that you can do with house numbers and ‘guestimating’ the offset, but this only works in towns or cities built in blocks. So why the problem with getting this data? Well strictly speaking there is no problem in getting it. You just need to pay several thousand pounds a year to the Royal Mail for the use of it… and that is just for single user use.
The Royal Mail maintains this data for its own use, obviously, but also sells it on to direct mail companies. But these companies are obliged to use accurate post coding so that they can qualify for the discounts on their mailings using the Post Office system called ‘MailSort’. Also when you start to look closely as to who pays to maintain this data you find that Local Councils have to provide the Royal Mail with information about all the new addresses created and the Ordinance Survey Office ( also a government body) then provides the location of the new address to the Post Office. The Post Office claims that they only make a profit of £1.6m from an income from PAF of £18.4m in 2005-06. A lot of this income is from selling the data back to the Local Councils and other government bodies to use. The ‘Free Our Data’ campaign suggests that as this data is so critical to the running of services in the country it should be made free online and copyright free, maintained by the government, because at the moment we are indirectly paying for this data via the input from the councils and the Ordinance survey and yet should you want to use the data in your web site you will have to pay again. And how!.
In many countries this information is open source and without the UK government doing something about this and making this data truly public, this country will be remain a backwater for mapping solutions at a time when the mapping capabilities of various technologies is growing at an escalating rate. Time and time again I sit through presentations on web based mapping technology, being excited and amazed at the functionality that is possible to achieve only to find out that it is prohibitively expensive to implement on a client’s web site. Wouldn’t it be cool for example to be able to show all the places within a radius of so many miles of a post code? Without the PAF data it is practically impossible in this country. In America, no problem. Wouldn’t it be great if we could jump on the rollercoaster ride along with our American cousins?
There are one or two sites out on the web which offer their own lists of Post Code and Longitude/ Latitude. How accurate these are I have no idea, but one that used to be available and only costs £90 was at Linuxbox (http://linuxbox.co.uk). However, on checking with the owner apparently the data is now quite inaccurate, possible due to all the new house building that has been going on, which highlights the main problem with such data and that is the maintenance of it. Aonther attempt a providing this data is http://www.freethepostcode.org which hopes to build a list up by users submitting known post codes with gps location data to build a totally open source datasource. It looks like they still have some way to go as they currently have 3547 post codes, but there are over 1.5 million in the UK! Also take a look at http://www.npemap.org.uk/which uses 1940 maps ( copyright free I guess) and you can find your house location and click on the map and submit the post code to their database. But none of these are ideal nor 100% accurate ( mind you the Post Office database is not 100% accurate but it’s the best we have), what we need is a way of being to update our web sites with the new data via an on-line resource. In the mean time there are plenty of companies who will licence you the data for various amounts depending on your use QAS ( www.qas.co.uk ) being possibly the best known. I have used their products in the past and have been impressed with their level of support, but for a lot of applications the price is prohibitive.
If you want to try it out for yourself then if you enter your post code followed by ‘,uk’ into the text box on Mike’s page at http://econym.googlepages.com/example_geo.htmyou can see for yourself the error caused by the incorrect geo-coding in the Google API. Next go to maps.google.com and enter the same post code and see the correct location. Fun isn’t it? Limitations like this just show how essential it is to undertake very thorough testing if you propose to use third party services and APIs for your line of business processes. Adding maps and pushpins is good fun on your Facebook page and who cares if it is not too accurate, but run a site that shows venues and you could have some irate customers on your hands. I always advise clients to think long and hard before they decide to use a third party free service on their web site. What would the implications to their business be if the service went down, or did not perform as expected?
Article by: Mark Newton
Published in: Mark Newton