Back on November 2nd, the City of Toronto launched their toronto.ca/open service – a project aimed to be the “official data set catalogue” of the city. Part of the OpenTO initiative to make various data that the city has collected available to developers in formats that make them easy to manipulate, toronto.ca/open is the first step to making the city and its services more “open and accessible”.

The hope is that if the city makes this information available in a readily-usable form, developers will take the time to create various services around them, helping citizens enjoy and take advantage of what Toronto has to offer. For example, data on garbage collection, public transit or upcoming city events could be used to create a service that would alert users through various means of communication. But at present, the amount and types of data available are fairly limited.

Announced back in April at the Mesh Conference in Toronto by Mayor David Miller, the openness initiative had a lot to live up to in the half-year following up to its release earlier this month at the Toronto Innovation Showcase.

Several cities have already implemented similar projects, leaving Toronto to play catch-up. Vancouver, for example, launched their own Open Data initiative earlier this year, and Toronto’s data catalogue appears to be modeled after it. Edmonton, a much smaller and less dense city than Toronto, has also recently begun an initiative to open their city’s data to the public in a similar manner, with promising results.

But the data is better late than never, so I applaud the City’s effort to democratize and help keep citizens better informed.

The data itself

Looking at what’s offered, you can see that most of the data sets or services have a geographical component to them. This only makes sense, as hyperlocal information is always of great interest to most people.

The data is presented in a variety of formats, but the most popular seems to be an ESRI Shapefile, a file format used by a popular suite of GIS software. With much of the data having a geospatial aspect to them, this makes sense as the data likely didn’t require much conversion before being released by the City. (I.E., they probably use ESRI’s application for city planning, etc.)

Thankfully, the format has been well-documented by its developer, and Wikipedia as well. All of these ESRI Shapefiles are intended to be downloaded for offline use, as they are typically made available inside of a zip file. Thus, the process of keeping them up to date may prove tedious, though I’m sure an enterprising developer could automate it nicely.

Other data has been made available in various XML formats. Some are intended to be accessed as online services, with a link to the XML document and the corresponding XSD file, while at least one is presumably meant to be used offline since it’s zipped up. Some of the XML “Feeds” are also in a format without a defined schema and are not RSS or Atom feeds. For example, the feed of City-sponsored events would be better served by an RSS or Atom feed, which would not only allow developers to integrate it into an application, but also regular users to sign up to the feed using any newsreader.

There are also some web services explicitly offered. The first is a geocoder service that can be used to validate addresses and other place names within the City. It is a WSDL service, and there is a full PDF file to document the input parameters and expected output.

The second is a series of services that provide access to “live geospatial data from the City of Toronto“. These are all Web Map Services that conform to a protocol specification developed by the Open Geospatial Consortium. It would have been nice if they had linked to the WMS Specification (PDF document) that explained how to query the services, instead of forcing you to find out the information yourself. (See page 14 and onwards of the mentioned document)

Lastly, some of the data is offered in plain old text files. In particular, the TTC Routes and Schedules is only available as a series of zipped text files that are periodically updated, presumably manually. This may be a bit backwards, but thankfully the formatting is well-defined.

Drawbacks

Though it may not be fair to criticize at this point, there are some clear limitations to the data. Firstly, much of the data is intended to be downloaded and then used in an “offline” capacity, that is, without further communication between the application and the City’s servers that host the data. Though this was probably done due to technical limitations (i.e. reducing bandwidth usage), it limits the ability of an application to stay up to date with the latest data. For things like the TTC Route and Schedules and the list of Apartment Bylaw Infractions, this severely reduced their effectiveness as the data will likely need to be periodically manually updated.

Furthermore, the current data set is somewhat limited and may impede the usefulness of any applications or services that can be developed. For example, making real-time TTC updates and other data available would be an immense benefit, yet the only TTC data currently available are in zipped text files that are only periodically updated. Thankfully, the City has setup a site to allow users to request more data, and there are already many requests for more TTC data.

It would also be nice if crime statistics for Toronto could be made available in an easy to use format. In particular, the underlying data used by the TPS to produce their crime maps and statistics could be beneficial in developing all sorts of interesting maps. In fact, this is what the Toronto Star has been doing for some time, but they had to get the data through an explicit request under access-to-information legislation. Since then, they’ve produced some very interesting maps and it would be beneficial if everyone had access to the same data so that the maps could be improved upon and kept up to date.

Producing such crime-based maps can provide people with one more factor to help them determine what part of the City they’d like to live in, or just allow them to have access to accurate statistics on crime. Not all of this is negative – as any study of crime in Toronto will show that it’s been following a downward trend for quite some time, so such information could help ward off any potential negative spin that media may put on crime. However, I do realize the privacy implications that come with releasing such data, so time should be taken to ensure that it is released in a manner where anonymity is preserved as best as possible.

Lastly, it’s important to note that the OpenTO initiative is still at a very early stage. Even with the way things move at City Hall, things will eventually improve. It’s also very important to give your feedback, either at DataTO or directly by email to opendata@toronto.ca. Additionally, if you’re a developer, the corresponding Google Group is a good place to get started. Even with the current limitations, I am sure we’ll see some great applications developed based on the data. One need only to look at MyTTC to see what can be done with some effort and ingenuity – and this was done well before the City officially released any data!

After much delay, possibly due to government meddling, Google Street View finally went live in select Canadian cities last week! Most of the GTA is covered, along with Kitchener/Waterloo, Ottawa, Montreal, Quebec City, Halifax, Calgary and Vancouver. This likely ensures coverage for a majority of Canada’s population, if only a very small minority of its geographical area.

This is a welcome move, since street view has been available for our neighbours to the south for the past two years.

According to some reports, Street View was supposed to launch in Canada earlier this year, and some of the imagery was collected as early as 2007. (Though the images around my area seem to be from early this year) Oh well, this is better late than never!

I’m speculating that the government’s concern with Google Street View may have slowed deployment, as in 2007 the privacy commissioner voiced strong concerns against the technology. I’ve never believed any of that FUD, but it seems that their concerns have been addressed with Google’s automatic detection and blurring of faces and license plates, along with new straightforwards processes to request that your image be removed from the service.

But it’s still been much ado about nothing, not really like the conflict between the government and Facebook earlier this year, which was also subsequently resolved. In the case with Facebook, privacy settings were often complicated and unclear; with Google Street view, they’re merely providing public views of public places, so I don’t see what the problem with that is.

Also available on the iPhone/iPod Touch

You can also use Street View on the built in Maps app for the iPhone OS, though it’s a bit convoluted since there’s no “Street View” button to toggle. Instead, you must first drop a pin (click on the “layer peel back” icon in the lower right corner) and then bring up the dialog box for that pin. If it’s in an area with Street View present, the little guy icon on the left will be lit up and you should be able to click it to bring up Street View.

Street View only works in landscape mode but it’s still pretty useful for getting to know an area before you visit it for the first time. Sometimes an actual image can convey sense better than just a map.

Competition?

It remains to be seen how the launch of Street View in Canada will fare for the similar service launched by Canpages in BC. That service only covers Vancouver, Whistler and Squamish but was launched back in March.

Nonetheless, it’s very neat technology, even if it was originally unveiled over two years ago. Witness this by the fact that reporters are still drooling over it by embedding the service in every article covering the launch.

Motivation and Usage

There are numerous reasons for using the user’s location to improve the quality of a service. For example, the restaurant guide and review site, Restaurantica, automatically makes a guess as to your location and populates the “Search” field so that you are one click away from finding information tailored to your geographic area.

This determination is based on the user’s IP address. Usually, the lookup is done based on a static set of mappings from IP addresses to geographical locations, like cities. In fact, this is exactly what the IP address geolocation database (that we’ll use) does. It has the advantage of being easy to understand but unfortunately the accuracy can degrade over time, as IP addresses are usually not hardwired to specific locations. For example, most DSL users are subject to DHCP, so the same IP address may be re-used across different cities. (The IP address database from IPInfoDB is updated periodically to help reduce this inaccuracy)

Thus, you should not rely on the information provided to be 100% accurate; it should only be used as a suggestion. This is why Restaurantica only populates the search field with your estimated location, rather than giving you the results for that location right away; if it was wrong, the results for a different location would certainly be confusing to a user. This gives the user a chance to correct any inaccurate lookups.

Using the IP Address geolocation database

The first step is to download the database and import it into your local database; for this example I’ll use MySQL. Note that the site also offers an IP location Lookup API, which has the advantage of not requiring a local database but may be subject to usage limits; if you’ll be getting a lot of traffic you’ll be better off using the local database.

Import the contents into MySQL

I downloaded the Complete (City) database as one table in SQL format. Extracting the file from the archive, you’ll see that it’s well over 300 MB in size. If you’re fond of using phpMyAdmin, you unfortunately won’t be able to import such a large file using that tool, so we’ll have to rely on the command line to import the contents. Go to your mysql/bin folder and enter the following command, assuming you extracted the SQL file to the same location:

mysql -u root -p [name of database] < ipinfodb_one_table_full.sql

Running this command will take some time, as it won’t be fast having to import such a huge table. Take a break and come back in a few minutes. (You may want to create a new database to hold just this table, do that before running this command) After that’s done, you should be able to see the contents in phpMyAdmin, like below:

Using the database

Querying the database is straightforward, and the IP Info DB website has plenty of examples at the bottom of the download page. Basically, once you have the user’s IP, you can use a query like this to find the most likely city it originated from:

SELECT *
FROM `ip_group_city`
WHERE `ip_start` <= INET_ATON([ip address as a string])
ORDER BY ip_start DESC
LIMIT 1;

The INET_ATON function converts a string that is the dotted-quad octet form of an IP address (the form you’re most used to seeing) into a 32-bit unsigned integer. This integer is then used to locate the record that most likely corresponds to the geographical location. The database is structured such that row with the highest ip_start value not greater than the given IP address is deemed to be the most likely location.

An example:

SELECT * FROM `ip_group_city` WHERE `ip_start` <= INET_ATON('69.156.150.155') ORDER BY ip_start DESC limit 1;

(This returns the city of Toronto, Ontario, Canada) If you want to get a list of likely locations, just increase the LIMIT argument to something like 10, and this will return 10 locations that are likely close to where the IP address is located. Note that there may be duplicates in this list.

A simple example

Here’s a simple PHP script that takes the user’s IP address and displays their most likely location on a map using the Google Static Maps API.

<?php
$ipGeolocationDatabaseName = 'ip_geolocation';

$link = mysql_connect('localhost', 'root', '');
$ipAddress = mysql_real_escape_string($_SERVER['REMOTE_ADDR']);

echo $ipAddress;

mysql_select_db($ipGeolocationDatabaseName, $link);

$query = "SELECT * FROM `ip_group_city` " .
         "WHERE `ip_start` <= INET_ATON( '$ipAddress' ) " .
         "ORDER BY ip_start DESC " .
         "LIMIT 1";
$resultLocation = mysql_query($query);

$numRows = mysql_num_rows($resultLocation);

if (0 == $numRows)
{
  echo "No likely locations found.";
}

while ($row = mysql_fetch_array($resultLocation))
{
  echo "<pre>";
  print_r($row);
  echo "</pre>";

  $lat = urlencode($row['latitude']);
  $lng = urlencode($row['longitude']);
  if (!empty($lat) && !empty($lng))
  {
    // NOTE: Replace the `key` parameter with your own, obtained from:
    // http://code.google.com/apis/maps/signup.html
    // (The one in use is only good for URLs on `localhost`)
    $queryString = "center=$lat,$lng";
    $queryString .= "&zoom=10";
    $queryString .= "&size=300x300";
    $queryString .= "&markers=$lat,$lng,blue";
    $queryString .= "&key=ABQIAAAAicCf6MqoHlR-MsfivtVWPRT2yXp_ZAY8_ufC3CFXhHIE1NvwkxTNmqRdvnM9orG9QdyALHoUZfmFuQ";
    $queryString .= "&sensor=false";

    echo '<img src="http://maps.google.com/staticmap?' . htmlentities($queryString) . '" alt="location" />';
  }
}
?>

Here’s what it display for my IP address:

As you can see, it’s not always accurate, but it’s a good start to enhancing your website to offer localized services and features.