This is by no means a complete list, and I’ll be adding more links over the next few days so check back often. And please leave me a note if you have any suggestions!

Update: thanks to Confreaks, the video is now up as well.

A few notes post-talk…

Thank you to those of you who attended my talk this morning! A few quick notes before getting into the links and downloads.

• One thing I should have made more clear is that your choice of coordinate system is not limited to just lat/long or Mercator. Those two will be useful for many cases, but there are a plethora of other coordinate systems, any one or more of which may be appropriate for your application.
• We covered PostGIS in this talk because it’s currently probably the best and most feature complete open source spatial database, and because it’s the best supported from our stack. But there are alternatives that may be more or less useful. MongoDB does have limited spatial capabilities, as does MySQL. Oracle and MS SQL Server are supposedly pretty full-featured in their spatial capability. There’s even a plugin for Sqlite that works quite well.
• I’m serious about Squeel. Use it. It’s awesome. Last year I was working on coming up with a solution for writing these complex spatial queries by extending Arel. But then I found Squeel, which does it a dozen times better than I could have come up with. Thank you Ernie Miller for this tool!

Now on to the links…

Talk-Related Material

• Talk slides on SpeakerDeck
• Talk slides in PDF format — Full size (about 20 megs) or compressed (about 3.6 megs)
• Video from Confreaks
• Talk notes — Posted by NewHaven.rb

Software — Databases

• PostGIS — spatial plugin for PostgreSQL
• Spatialite — spatial plugin for Sqlite3
• GeoCouch — spatial plugin for CouchDB and Couchbase
• Solr Spatial Search — available in recent versions of Solr

Software — Ruby

• rgeo — Spatial data types
• rgeo-shapefile — Shapefile reader
• rgeo-geojson — GeoJSON reader/writer
• ffi-geos — Low-level ruby bindings to GEOS library
• squeel — ActiveRecord enhancement for query writing
• activerecord-postgis-adapter — ActiveRecord adapter for PostGIS
• activerecord-spatialite-adapter — ActiveRecord adapter for Spatialite
• mongoid_geospatial — Mongoid extension with geospatial capabilities
• ruby geocoder — Integration with geocoding services
• GeoRuby and SpatialAdapter — Older libraries for spatial data types and ActiveRecord integration (not compatible with rgeo)

Software — Client Side

Software — Other Libraries

• libgeos — C library for geometric analysis
• libproj — C library for coordinate transforms
• libgdal — C library for rasters

Articles/Blogs/Resources Online

• Geo-Rails series — Nine (and counting) articles on geospatial development with Rails
• Cartographical Map Projections — A good introduction to projected coordinate systems
• SpatialReference.org — Source for coordinate system information.
• Shapefile spec from ESRI
• SimpleGeo migration guide — A very useful set of links put together by Urban Airship after they bought (and dismantled) SimpleGeo, including sources for data, services, and implementation guides.

Books and Other Reading

• GIS: A Computing Perspective by Michael Worboys and Matt Duckham — Very useful overview of the major computing challenges involving geometry, rasters, databases, and so forth, including discussions on modeling uncertainty and architectural considerations. Highly recommended.
• Geographic Information Analysis by David O’Sullivan and David J. Unwin — Good source for methods of statistical analysis and visualization of geo-data.
• GIS For Web Developers by Scott Davis (published by Pragmatic Programmers) — Somewhat dated, and based on Java. But still a lot of useful conceptual material. Paper book is out of print, but eBook is available.

Data Sources

• TZ Timezone Shapefiles — Polygon boundaries of world timezones
• TIGER/Line files from the US Census — lots of legal and statistical geographic data

Mailing Lists

• rgeo-users mailing list — Help and discussion group for RGeo users
• georails mailing list — Discussion for Rails and geospatial dev (not very active)
• georuby mailing list — Discussion for Ruby and geospatial dev (not very active)

Organizations and Conferences

• Open Geospatial Consortium — Lots of standards and info
• Open Source Geospatial Foundation — Umbrella organization for lots of geospatial software
• FOSS4G — International open source geospatial software conference

• rgeo 0.3.4 fixes a segfault under Ruby 1.8.7, and supports prepared geometries in the FFI-Geos factory.
• activerecord-postgis-adapter 0.4.1 fixes some Rails 3.2 compatibility issues.
• activerecord-mysql2spatial-adapter 0.4.2 also fixes some Rails 3.2 compatibility issues.

Just update your gems to get these fixes. Many thanks to those who reported issues and submitted patches on Github. They were very helpful.

I’m still investigating a couple of issues in the spatialite adapter. I hope to get those resolved in a few days.

One other note. I’m not using Rails 3.2 on my own projects yet. (In fact, for the most part I’m still on 3.0.x.) So there may still be regressions and compatibility issues on Rails 3.2 that I haven’t found yet. Please comment here or email me if you find anything.

In this article, we’ll take a brief look at the spatial_ref_sys table and how you can use it in your application. We’ll cover:

• What’s useful about the spatial_ref_sys table
• Where the spatial_ref_sys data comes from, and how you can populate your own custom data.
• Accessing spatial_ref_sys data from Ruby using RGeo’s SRSDatabase

This is part 9 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

So what’s in the spatial_ref_sys table anyway?

The spatial_ref_sys table is defined by an OGC specification entitled Simple Feature Access Part 2: SQL Option. This is a companion to the Simple Features specification we covered in earlier articles. It takes the standard data types and operations and specifies how such data should appear in an SQL-based relational database. Many of the “ST_*” functions that we’ve used when interacting with PostGIS are defined in this specification.

We recall from part 4 that when you’re working with spatial data, every coordinate references a coordinate system that defines what the coordinate means—whether it is latitude and longitude, or feet from your front door, or light-years from Alpha Centauri. Those coordinate systems are generally represented by a numeric ID reference known as the SRID. Spatial_ref_sys is merely a table of known coordinate systems keyed by their SRID. According to the spec:

Every Geometry Column is associated with a Spatial Reference System. The Spatial Reference System identifies the coordinate system for all geometric objects stored in the column, and gives meaning to the numeric coordinate values for any geometric object stored in the column. … The Spatial Reference System Identifier (SRID) constitutes a unique integer key for a Spatial Reference System within a database.

How is this useful? Generally, spatial_ref_sys will provide an actual definition of the coordinate system for each SRID. This theoretically provides enough information to correctly interpret every piece of coordinate data in your database, and even—where possible—to convert the data into whatever coordinate system you need.

According to the spec, all implementations of spatial_ref_sys include these columns:

• srid: The numeric SRID. This should be the table’s primary key.
• auth_name: An authority name as a string. This is set if this coordinate system is specified by an outside authority such as EPSG.
• auth_srid: The numeric ID of the coordinate system in the above authority’s catalog.
• srtext: The Well-Known-Text (WKT) representation of the coordinate system (as we described in part 4).

If you are using PostGIS, you’ll notice spatial_ref_sys has one more non-standard but very useful column:

• proj4text: The Proj4 representation of the coordinate system.

Where does the data come from and what does it do?

In many cases, a spatial database will prepopulate spatial_ref_sys for you with a standard set of EPSG data. If you are using PostGIS, this is handled by the spatial_ref_sys.sql script, which gets run when you create a spatial database. If you are using Rails and follow the steps in part 2, the activerecord-postgis-adapter will do this for you automatically when you create your database.

The EPSG spatial reference database is such a universal standard that in most cases it is probably best to use one of its coordinate systems and its corresponding SRID. This will maximize the chances that your SRIDs will match those used by any external data sources you will interact with—meaning your data will be easily portable. However, you may run into a case where you need to define your own coordinate system, perhaps because you’re using an unusual map projection. In such cases, you can add rows to the spatial_ref_sys table. You can also delete SRID rows that you don’t need. It is, after all, merely a table in your database.

The main caveat is that most spatial databases (including PostGIS) will establish a foreign key constraint between SRIDs and this table. This means that your data can’t just choose any SRID it wants. The SRID has to exist—and by “exist”, we simply mean it has to be present in the spatial_ref_sys table. So you just need to make sure that you don’t delete any SRIDs that you need, and you add any that aren’t provided by default.

Some databases will provide additional tools that leverage the spatial_ref_sys information. PostGIS, for example, provides the SQL function ST_Transform(), which lets you transform a geometry from one coordinate system to another. When you call it, you provide the SRID of the desired coordinate system. PostGIS then looks up both the original and the target SRIDs in spatial_ref_sys, and uses the coordinate system information there to figure out how to compute the transformation.

Accessing the spatial_ref_sys from Ruby

The RGeo library provides a convenient way to look up coordinate systems from the spatial_ref_sys table. If you’re already using PostGIS and activerecord-postgis-adapter, it’s quite easy:

srs_database = RGeo::CoordSys::SRSDatabase::ActiveRecordTable.new
entry = srs_database.get(4326)
entry.identifier  # => 4326
entry.name        # => "WGS 84"
entry.proj4.to_s  # => " +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs +towgs84=0,0,0"

You can now, for example, create a factory using the SRID (from entry.identifier) and the Proj4 object (from entry.proj4). As we saw in part 4, you generally need to provide Proj4 information if you are going to be converting data between coordinate systems.

As a convenient shorthand, some factories let you pass in an srs_database object when you construct a factory. The factory constructor will then go through the process of looking up the coordinate system definition and extracting the Proj4 specification. For example:

srs_database = RGeo::CoordSys::SRSDatabase::ActiveRecordTable.new
my_factory = RGeo::Geos.factory(:srs_database => srs_database, :srid => 3785)
my_factory.proj4.to_s  # => " +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +units=m +k=1.0 +nadgrids=@null +no_defs"

Other SRSDatabase sources

The spatial_ref_sys table is not the only source of coordinate system information. The Proj4 library itself installs a bunch of data files that contain most of the EPSG codes and other information. There are also online services that you can use to look up coordinate systems; one particularly important one is spatialreference.org.

The SRSDatabase mechanism in RGeo supports a common interface for coordinate system databases. In the section above, we looked up EPSG 4326 from the spatial_ref_sys table using the ActiveRecordTable class. Alternately, we can use the SrOrg class to look up information from spatialreference.org:

srs_database = RGeo::CoordSys::SRSDatabase::SrOrg.new('EPSG')
entry = srs_database.get(4326)
# ...

You can choose an appropriate source of coordinate system information based on the requirements of your application. In many cases, however, I recommend using the spatial_ref_sys table because it is convenient and readily available. See the RDocs for RGeo for more information on connecting to different SRSDatabase sources.

Where to go from here

In this article, we covered spatial_ref_sys, one of the standard tables that is included in most spatial databases. For more information on this table and how it is implemented in the PostGIS database, see the PostGIS documentation online. The official specification of the table is available in the OGS Simple Features for SQL spec.

RGeo provides basic convenience tools for accessing coordinate system information from the spatial_ref_sys table. We covered a few basic examples in this article. For detailed information, see the RGeo::CoordSys::SRSDatabase module in the RGeo documentation.

Observant readers may notice that PostGIS includes one more automatic table, called geometry_columns. I may cover this table in a later article that digs deeper into PostGIS, but if you’re interested now, it’s described in the PostGIS documentation.

This is part 9 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

My story

When I first started working with Rails back in 2006, I followed the common practice of abstracting all my data and business logic into ActiveRecord “models.” This worked out for a short time, but soon I realized that it didn’t scale all that well to complex applications. There were a number of reasons, some of which are now finally being discussed and addressed in the community.

In any case, what followed was the Big Refactor of my Rails development process and architecture. On the one hand, I rediscovered Ruby’s object-oriented roots, separating behavior from persistence, and building higher-level business objects focused on actually modeling my system. And on the other hand, I rediscovered the database, writing more and more custom SQL to leverage its capabilities.

I still use ActiveRecord where it suits the task at hand. In many cases, it makes sense to treat rows of a table as data objects principally identified by a numeric ID, and in those cases ActiveRecord still shines. However, in other cases I’m finding the ORM more of a hindrance than a help. Many of my model implementations have begun bypassing ActiveRecord-based persistence altogether, instead creating plain old tables and querying with custom SQL.

The good news is that ActiveRecord provides some amount of support for low-level queries. You can share ActiveRecord’s database connection, and you don’t need to know the API of the underlying database driver, making it convenient to add custom SQL incrementally to your Rails application. In this article, I’ll provide an overview of ActiveRecord’s connection API, a convenient low-level interface for writing raw SQL and interacting with your database on its own terms.

Obtaining a Low-Level Connection

Hidden underneath your ActiveRecord class is a useful low-level object called the ActiveRecord connection adapter. It wraps and abstracts away the underlying database-specific driver, and provides a common interface for database tasks such as creating and destroying databases, creating and modifying tables, inserting, updating, and deleting data, running queries, and managing transactions. Normally, the connection adapter is used internally by ActiveRecord, but you can access it yourself if you want to talk to the database directly without using ActiveRecord “models.”

To obtain a connection adapter object, simply call the connection method on your ActiveRecord class or any ActiveRecord object:

connection = User.connection
obj = User.find(1)
connection = obj.connection

Which ActiveRecord class should you use? In most Rails applications, you talk to just one database, as defined in your database.yml file. For such an application, every ActiveRecord class, including ActiveRecord::Base, will give you the same connection. So in those cases, it doesn’t matter which class you use. However, if your Rails application connects to a secondary database for some ActiveRecord classes (using the establish_connection method) then those classes will yield a connection object pointing at the secondary database. In those cases, you will need to pay attention to which database you need to talk to, and ask for a connection from the right class.

Managing connections and the connection pool

Each connection adapter object represents a single connection to a database. Rails generally opens several connections at once and manages them in a connection pool. When a task needs a database connection, it checks one out of the pool; when it finishes, it checks the connection back in so that the next task can use it. Connections can run only one SQL statement at a time, so generally one connection is opened per thread.

In most Rails applications, this takes place transparently. When you’re processing an HTTP request, ActiveRecord automatically checks out a connection and memoizes it for the duration of the request. Whether you use the standard ActiveRecord APIs, or obtain a raw connection by calling the connection method described above, you get the request’s memoized connection. At the end of the request, ActiveRecord’s Rack middleware automatically checks the connection back in.

However, if you access ActiveRecord outside the context of handling a request via your controller, then you need to do connection management yourself. ActiveRecord will still automatically check out and memoize a connection when you ask for one. However, if you are not using a normal Rails controller, or are otherwise not running inside ActiveRecord’s ConnectionManager middleware, you should check in the connection manually when you are done. You can do so using the clear_active_connections! method on any of your ActiveRecord classes:

User.clear_active_connections!

Now that we know how to get a connection, let’s see what we can do with one.

Running Low-Level Queries

Most calls in the standard ActiveRecord API return ActiveRecord “model” objects. However, there might be cases when you want to bypass the overhead of creating these full ActiveRecord objects, or maybe you want to query data that doesn’t have a corresponding ActiveRecord class. The connection adapter’s low-level query methods let you write your own SQL, and return “plain old data” as raw result tables.

In this first example, we get the “name” value from a single row in our “users” table. If we only need the name, and don’t need the full ActiveRecord object with the rest of its data and capabilities, we can grab the connection object and use the select_value method:

connection = User.connection
name = connection.select_value("SELECT name FROM users WHERE id=1")
# => "Dave"

Use select_value when you need a single value from a single row. The result will be either a string, or nil if no rows match. Note that the returned value is a string even if the column has a different type such as a number or timestamp. You will need to perform your own conversion:

time_str = connection.select_value(
  "SELECT created_at FROM users WHERE id=1")
# => "2011-12-24 11:35:02"
time = Time.parse("#{time_str} UTC")

(This often gets me when I ask for a numeric value from the database. The select_value method returns it as a string, so be sure to call to_i on it if you want to do any math.)

If you want a single column from multiple rows, use select_values, which returns an array of strings.

names = connection.select_values(
  "SELECT name FROM users WHERE created_at>'2012-01-01 00:00:00'")
# => ['Bob', 'Kathy']

To obtain multiple columns, you can use select_rows. This method returns an array of arrays, each representing a row in the order of the selected columns. For example:

rows = connection.select_rows(
  "SELECT id,name FROM users WHERE created_at>'2012-01-01 00:00:00'")
# => [['2', 'Bob'], ['4', 'Kathy']]

This returns an array of two-element arrays. Remember that all values are returned as strings. In the example above, the “id” column is numeric, so you may need to call to_i if you want to treat the IDs as integers.

There is no select_row method. To select a single row, just use select_rows and take the first element.

Alternately, you can return rows as hashes (of column name to value) using select_one (for a single row) or select_all (for multiple rows).

records = connection.select_all(
  "SELECT id,name FROM users WHERE created_at>'2012-01-01 00:00:00'")
# => [{'id'=>'2', 'name'=>'Bob'}, {'id'=>'4', 'name'=>'Kathy'}]

Low-Level Data Updates

Updating data is accomplished using similar calls. Insert rows into the database using the insert method:

connection.insert(
  "INSERT INTO users SET name='Sally', created_at='now'")

Update rows in the database using the update method. This method returns the number of rows affected by the update.

row_count = connection.update(
  "UPDATE users SET name='Robert' WHERE id=2")
# => 1

You can delete rows using delete, which again returns the number of rows deleted.

row_count = connection.delete(
  "DELETE FROM users WHERE created_at>'2012-01-01 00:00:00'")
# => 3

Normally, however, you won’t be hard-coding values when you insert and update. You’ll be injecting data obtained from the user or some other source. Therefore, to avoid SQL injection attacks and other issues, you need to quote these values when you construct your SQL statement. Use the quote method for this purpose. It takes a Ruby object and represents it, properly quoted, in the syntax expected by the database. Here are some examples from the PostgreSQL connection adapter. (Connection objects for other databases will yield slightly different results depending on the database.)

connection.quote("Hello") # => "'Hello'"
connection.quote("Joe's") # => "'Joe''s'"
connection.quote(2) # => "2"
connection.quote(true) # => "'t'"
connection.quote(Time.now) # => "'2012-01-23 07:46:41.540033'"

Use quote when constructing SQL statements that update the database:

new_name = "Robert"
row_id = 2
row_count = connection.update(
  "UPDATE users SET name=#{connection.quote(new_name)}"+
  " WHERE id=#{connection.quote(row_id)}")

Using Prepared Statements

Prepared statements are a common database optimization technique. The idea is that often many of the queries run by your application will have a common form. For example, you might find yourself running hundreds of queries of the form “SELECT name FROM users WHERE id=[something]“. Instead of reparsing the entire statement and rerunning the query planner on every query, you can often instruct the database to parse and plan only once, and re-use that information on subsequent queries. The way to do that is with a prepared statement.

As of Rails 3.1, ActiveRecord includes support for prepared statements. The standard ActiveRecord API will utilize prepared statements transparently. If you are using the low-level connection API, you can set up prepared statements manually.

For running queries, the only one of the methods we’ve covered that supports prepared statements is select_all. Instead of the full statement, you must send a statement template and a list of values to substitute in. The list of values should be an array of two-element arrays; the second element of each array is the value. The first element is generally set to the ActiveRecord column type governing the type of the value, but you can set it to nil if you’re confident of the type you’re sending in. The values must be sent in as the third argument. (The second is a “name” for the query, used for annotating the logs. You can set it to nil.) Here’s the earlier select_all example, rewritten using a prepared statement.

records = connection.select_all(
  "SELECT id,name FROM users WHERE created_at>$1",
  nil, [[nil, ::Time.utc(2012,1,1)]])
# => [{'id'=>'2', 'name'=>'Bob'}, {'id'=>'4', 'name'=>'Kathy'}]

Now, if you have additional requests of the same form (using the same template), they should run faster because the prepared statement (with its prepared SQL and predetermined query plan) is being reused.

records2 = connection.select_all(
  "SELECT id,name FROM users WHERE created_at>$1",
  nil, [[nil, ::Time.utc(2012,1,2)]])
records3 = connection.select_all(
  "SELECT id,name FROM users WHERE created_at>$1",
  nil, [[nil, ::Time.utc(2012,1,3)]])

The insert, update, and delete methods provide similar access to prepared statements. For the update and delete methods, use the same arguments as select_all: the template first, followed by a query name (which can be nil) and then an array of values to inject into the statement:

row_count = connection.update(
  "UPDATE users SET name=$1 WHERE id=$2",
  nil, [[nil, "Robert"], [nil, 2]])

The insert method is a little more complex because it takes a series of three more arguments before the values list. (Those extra arguments are for supporting databases that make you manage primary key sequences manually. In most cases, you can set those to nil.) So the values array should be passed as the sixth argument to insert. It’s a little messy, but this is the way the API is set up as of Rails 3.1.

connection.insert(
  "INSERT INTO users SET name=$1, created_at=$2",
  nil, nil, nil, nil, [[nil, "Sally"], [nil, "now"]])

Prepared statement support requires ActiveRecord 3.1 or later. On older versions, you will need to construct the entire SQL statement using quote to inject values.

Low-Level Migrations and Other Features

Migrations are a very common case for dropping to low-level SQL statements, since there may be many cases when you’ll want more control over your schema than ActiveRecord gives you.

When you write an ActiveRecord migration, you actually are already using the connection adapter API. The create_table and similar methods are connection adapter methods; the migration simply delegates them to the adapter. This means you can use all the methods we’ve been discussing, such as insert and update, directly in your migration if you wanted to inject data during that process. (With the caveat, of course, that many consider it bad practice to alter data during a migration.)

To perform schema changes using raw SQL, you should generally use the execute method. This method returns the underlying database driver’s result object; it doesn’t try to do any postprocessing on the result, and so it generally may be faster than the calls we’ve been looking at so far. It’s useful for cases when you want the very low-level driver-specific result, or (as in most migrations) for cases when you don’t care about the result. Here’s an example using PostgreSQL’s flavor of SQL:

class MyMigration < ActiveRecord::Migration
  def up
    execute <<-SQL
      CREATE TABLE users (
        id SERIAL PRIMARY KEY,
        name CHARACTER VARYING NOT NULL,
        created_at TIMESTAMP NOT NULL)
    SQL
  end
  def down
    execute 'DROP TABLE users'
  end
end

Nowadays when I write migrations, I use execute for almost everything because I generally want to fine-tune the database schema. It’s not strictly necessary in the above very simple case, but in more complex applications, you may want to set up constraints, triggers, and other data management features in your database, and you’ll need it in those cases.

Of course, if you use execute, you will have to write out both the forward and reverse migrations. You can’t use Rails 3.1′s “change” feature which attempts to auto-create the backward migration from the forward migration—Rails isn’t quite smart enough to figure out how to reverse a change made by arbitrary SQL. But you may find it an acceptable trade-off, as I have.

Another common use for the connection object is to delimit transactions. If you have a set of statements that should be wrapped in a transaction, use the transaction method:

connection.transaction do
  connection.update("UPDATE users SET name='Robert' WHERE id=2")
  connection.update("UPDATE users SET name='Catherine' WHERE id=4")
end

Because the connection adapter is shared with ActiveRecord, you can also include high-level ActiveRecord calls in your transaction block, interspersed with your low-level calls. For example, you could write the above code like this:

connection.transaction do
  connection.update("UPDATE users SET name='Robert' WHERE id=2")
  obj = User.find(4)
  obj.name = "Catherine"
  obj.save
end

I prefer the first version, however. For starters, it’s more performant—it doesn’t require an extra select to get the data, and it doesn’t require building the ActiveRecord object simply to update a single field. But in my opinion, it’s also cleaner and more succinct. I know some don’t like seeing SQL in your web app, but in many cases it’s a very expressive and readable language for tasks like this. I believe is simply a case of the right tool for the right job.

You can also obtain various information about the capabilities of the database and the database driver. Here are just a few examples:

connection.supports_savepoints?       # Database supports savepoints?
connection.supports_statement_cache?  # Supports prepared statements?
connection.table_name_length          # Maximum table name length
connection.columns_per_table          # Maximum columns per table

Finally, the connection adapter provides access to the underlying driver-specific connection for when you need to access really advanced or database-specific features. Here’s an example (assuming you’re using the postgresql database adapter).

raw_connection = connection.raw_connection  # Returns a PGconn object
pg_server_version = raw_connection.server_version.to_s

For More Information

This has been an overview of the tools that ActiveRecord provides for bypassing the ActiveRecord “models” and accessing the advanced features of your database directly. Most of us won’t write entire Rails applications using only this low-level API. However, it is a good tool to have in the toolbox. For those cases when the ActiveRecord ORM doesn’t quite do what you need, or does so clumsily or inefficiently, ActiveRecord lets you drop down to SQL quite easily.

The good news is that all these calls are well-documented on http://api.rubyonrails.org/:

• The ActiveRecord::ConnectionAdapters::DatabaseStatements module describes most of the low-level methods for querying and updating data that we covered in this article.
• The ActiveRecord::ConnectionAdapters::Quoting module describes various methods for quoting data for injection into SQL.
• The ActiveRecord::ConnectionAdapters::SchemaStatements module describes the schema-manipulation methods you are probably familiar with using for migrations.
• The ActiveRecord::ConnectionAdapters::TableDefinition class includes the methods you can call inside create_table.
• The ActiveRecord::ConnectionAdapters::DatabaseLimits module describes a miscellaneous set of informational methods you can call.

About the Author

Daniel Azuma is a Ruby developer specializing in geospatial technologies, computational geometry, graphics, and related fields. He is the author of rgeo and related gems for geospatial analysis in Ruby and Rails applications. He currently works as Chief Software Architect at Pirq.

Edits (Fri 27 Jan)— I made some corrections to the prepared statement examples so they actually work. Note to self: test examples before publishing.

In this article, we will cover:

• The goals for the service, and what is a ZCTA anyway
• Obtaining ZCTA data from the U.S. Census
• Developing our own ZCTA database
• Querying the database
• Improving performance using polygon segmentation

This is part 8 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

ZCT-what?

At Pirq, we do a lot of geospatial analysis. One task we have to perform fairly frequently is to group data into neighborhoods for demographic and spatial analysis.

Now, mapping coordinates to neighborhoods is a more challenging task than you might suspect. There is no “official” database, and boundaries on the ground are often imprecise and subject to rapidly evolving local information. One way that you could go about it is to define neighborhoods as zip codes or clusters of zip codes, but this carries its own challenges because (perhaps counter-intuitively) zip code “boundaries” are not well-defined either. Zip codes are defined by postal routes, and don’t necessarily correspond to or respect any other boundary information, including even city and state boundaries. (For further discussion, see the US Census position on zip codes at http://www.census.gov/geo/www/tiger/tigermap.html#ZIP.)

There are two different ways of tackling the problem of mapping neighborhoods or zip codes. One is to use a (paid) service to do the local heavy lifting—curating and cleaning the data, and tracking and applying local knowledge—for you. I generally recommend Maponics for this task, but there are a variety of services available. Alternatively, if you’re content with approximate data, and/or you have more direct access to some amount of local knowledge, you can build your own database using Zip Code Tabulation Areas (ZCTA).

ZCTA (pronounced “zik-tuh”) is a system created by the US Census to address the zip code problem. The idea is this. Sometimes you need to look up the actual zip code for a location for reasons related to postal delivery. In such cases you will need to work directly with the US Postal Service or a third-party curator like Maponics. But other times you don’t necessarily need the actual zip code, but you just want to use something like a zip code as a convenient unit of delineation for geo-analysis—for example, to approximate neighborhood boundaries. In this latter case, it doesn’t matter that the zip code itself isn’t always 100% accurate. Rather, what’s important is that the boundaries are stable and make geographic and demographic sense. ZCTA is designed for this latter case.

Each ZCTA is a collection of US Census blocks for which, at the time of the census, the addresses fell largely if not completely within a particular zip code. Because ZCTAs are made up of Census blocks, they are well-defined, stable, and statistically useful. And usually, they approximate the actual zip code boundaries fairly well.

For this article, we’ll build a simple ZCTA lookup tool: one that will let you query a location (latitude and longitude) for which ZCTA contains it.

Setting up our database to hold ZCTA data

I’ll assume we’ve set up a Rails project and a PostGIS database as covered in part 2. Let’s start by creating a model for the ZCTA data. For now we’ll just create a simple table capable of mapping geometry to ZCTA (zip code). You can of course modify this to add more fields.

% rails generate model Zcta zcta:integer region:polygon

This creates the following migration:

class CreateZctas < ActiveRecord::Migration
  def change
    create_table :zctas do |t|
      t.integer :zcta
      t.polygon :region
      t.timestamps
    end
  end
end

Before we migrate, we’ll do a few things here. First, we don’t need those timestamps so we’ll get rid of them. Second, we will want to do spatial queries against the :region column, so we’ll create a spatial index on that column, as we covered in part 6.

Third, we’ll choose a coordinate system for the :region column. For this service, I’ll use the EPSG 3785 projection. This coordinate system is often useful because of its affinity with mapping software, its local conformality, and its ability to keep political boundaries straight. It’s also good to choose a flat projection rather than a geographic coordinate system because we’re going to do some geometric manipulation. You can read more discussion on choosing a coordinate system for your database in part 7.

Our migration now looks like this:

class CreateZctas < ActiveRecord::Migration
  def change
    create_table :zctas do |t|
      t.integer :zcta
      t.polygon :region, :srid => 3785
    end
    change_table :zctas do |t|
      t.index :region, :spatial => true
    end
  end
end

Now we can run the migration:

% rake db:migrate

As we discussed in part 7, we also set up the ActiveRecord class to use the simple_mercator_factory.

class Zcta < ActiveRecord::Base
 FACTORY = RGeo::Geographic.simple_mercator_factory
 set_rgeo_factory_for_column(:region, FACTORY.projection_factory)
end

Obtaining ZCTA data and populating our database

A nice feature of ZCTA is that it is public data freely downloadable from the US government. A good start point for exploring the current (2010) Census ZCTA data is http://www.census.gov/geo/ZCTA/zcta.html. If you want to go straight to the downloads, head over here and choose “Zip Code Tabulation Areas” from the menu. You can download shapefiles for individual states, or the entire database as one huge shapefile. (Warning: the combined shapefile download is half a gigabyte compressed.)

For this example, we’ll download just the state of Washington, but it should be trivial to modify the code to deal with the entire database. When you download the data for Washington, you’ll end up with a zip file “tl_2010_53_zcta510.zip”. Unzipping this file yields a set of five files:

• tl_2010_53_zcta510.dbf
• tl_2010_53_zcta510.prj
• tl_2010_53_zcta510.shp
• tl_2010_53_zcta510.shp.xml
• tl_2010_53_zcta510.shx

The .shp extension clues us in that this is a shapefile, one of the formats we covered in part 5. Shapefiles are a very common format for public data sets. They’re great for downloading data, but not for running spatial searches. So our next task is to import the shapefile into our database.

The shapefile specifies that its geometric information is in the “NAD83″ geographic coordinate system (EPSG 4269). This is a geographic (latitude-longitude) coordinate system optimized for the United States. It does have very slight differences from the WGS84-based coordinate system (EPSG 4326) that we usually use, but for our purposes, the differences are negligible, so we’ll treat these as standard WGS84 geographic coordinates.

Now, our database uses the EPSG 3785 projection, so we’ll need to convert the polygons into the projection. We covered how to use the simple_mercator_factory to perform these projections in part 7. As we saw in part 4, converting lines and polygons between coordinate systems can change their shape if individual sides are long enough. For our purpose, ZCTA areas are small enough that we’ll ignore the effect.

One more issue is that the geometries we’ll read from the shapefile are actually MultiPolygons rather than Polygons. This is because some ZCTAs (such as those that cover islands) may include multiple disjoint areas. Since we defined our database column to store polygons, we’ll have to break up each MultiPolygon into its constituent parts. This means we may have some ZCTA numbers that are represented by multiple records in the database.

For the ZCTA number, we’ll take a quick peek at the Census-provided documentation at http://www.census.gov/geo/www/tiger/tgrshp2010/documentation.html. Here it states that the ZCTA number can be found in the “ZCTA5CE10” property field of each shapefile record.

Now we have enough information to write a script to read the shapefile and populate our database!

require 'rgeo-shapefile'
RGeo::Shapefile::Reader.open('tl_2010_53_zcta510.shp',
    :factory => Location::FACTORY) do |file|
  file.each do |record|
    zcta = record['ZCTA5CE10'].to_i
    # The record geometry is a MultiPolygon. Iterate
    # over its parts.
    record.geometry.projection.each do |poly|
      Zcta.create(:zcta => zcta, :region => poly)
    end
  end
end

Let that run for a minute or two, and now we have a fully populated database of ZCTA data for the state of Washington.

Running ZCTA queries

Now let’s write a simple API for querying the ZCTA for a given location.

First we’ll write some scopes in the ActiveRecord class to construct the queries. To find the ZCTA that contains a particular point location (latitude and longitude), we can use the ST_Intersects function. We just need to make sure we convert the location to the projected coordinate system, as we covered at the end of part 7.

For best performance, we’ll write our queries to speak PostGIS’s native language, which is EWKB (see part 5).

class Zcta < ActiveRecord::Base

  # ...

  EWKB = RGeo::WKRep::WKBGenerator.new(:type_format => :ewkb,
    :emit_ewkb_srid => true, :hex_format => true)

  def self.containing_latlon(lat, lon)
    ewkb = EWKB.generate(FACTORY.point(lon, lat).projection)
    where("ST_Intersects(region, ST_GeomFromEWKB(E'\\\\x#{ewkb}'))")
  end

end

We could also extend this to support queries by any arbitrary geometry, letting us find the ZCTAs that cover a line or polygon:

class Zcta < ActiveRecord::Base

  # ...

  def self.containing_geom(geom)
    ewkb = EWKB.generate(FACTORY.project(geom))
    where("ST_Intersects(region, ST_GeomFromEWKB(E'\\\\x#{ewkb)}'))")
  end

end

Now it’s pretty straightforward to write a web service API wrapper for this function. Here’s one way it could be done:

class ZctaController

  def lookup
    lat = params[:lat].to_f
    lon = params[:lon].to_f
    zcta = Zcta.containing_latlon(lat, lon).first
    render(:json => {:lat => lat, :lon => lon,
      :zcta => zcta ? zcta.zcta : nil})
  end

end

Segmenting polygons for improved performance

Now that we have a basic implementation, let’s see if we can improve performance a bit. In part 6, we saw that large, complex geometries, such as polygons with many sides, can result in slow queries. The ZCTA data, it turns out, does have some fairly large polygons with side counts in the tens of thousands. Since all we’re interested in is looking up ZCTA by location, we may be able to improve performance using the segmentation technique.

Segmentation involves breaking up large polygons into smaller polygons with fewer sides. It trades off a smaller polygon size for a larger number of rows in the database. However, the spatial index can help mitigate queries against a large number of rows, so such a trade-off may be a performance win in some situations. (Of course, we should measure so we know for certain—we’ll do that below.)

We’ll segment using four-to-one subdivision as described in part 6. For each polygon, we’ll count its sides, and if the count is larger than some threshold, we’ll divide it in half horizontally and vertically. An easy way to accomplish this division is to take the polygon’s bounding box and divide it four ways into smaller rectangles. Then, take the intersections of the original polygon with those sub-rectangles. These functions are available in the Simple Features interfaces, and are implemented by RGeo, as we covered in part 3.

Note that it is possible for a subdivision to result in multiple disjoint polygons in each quadrant (that is, a MultiPolygon). So we have to handle that case in the code.

We’ll also perform one more optimization: if the polygon is long and thin, we’ll divide it in two rather than in four, in order to make the pieces closer to square.

Ready? Here’s our implementation:

MAX_SIZE = 500
MAX_DEPTH = 12

require 'rgeo-shapefile'

# Handle a geometry of any type
def handle_geometry(depth, geom, zcta)
  case geom
  when ::RGeo::Feature::Polygon
    handle_polygon(depth, geom, zcta)
  when ::RGeo::Feature::MultiPolygon
    geom.each{ |polygon| handle_polygon(depth, polygon, zcta) }
  end
end

# Handle a polygon
def handle_polygon(depth, polygon, zcta)
  # Check the number of sides. We'll combine the number of sides for
  # the "outer edge" and any "holes" that the polygon might have.
  # A polygon boundary consists of a LineString that is closed, so
  # the first and last points are the same. Therefore, to count the
  # sides, count the number of vertices and subtract 1.
  sides = polygon.exterior_ring.num_points - 1
  polygon.interior_rings.each{ |ring| sides += ring.num_points - 1 }
  if depth >= MAX_DEPTH || sides <= MAX_SIZE
    # The polygon is small enough, or we recursed as far as we're
    # willing. Just add the polygon.
    Zcta.create(:zcta => zcta, :region => polygon)
  else
    # Split the polygon 4-to-1 and recurse
    depth = depth + 1
    # Find the bounding box for the polygon
    envelope = polygon.envelope.exterior_ring
    p1 = envelope.point_n(0)
    p2 = envelope.point_n(2)
    min_x = p1.x
    max_x = p2.x
    min_x, max_x = max_x, min_x if min_x > max_x
    min_y = p1.y
    max_y = p2.y
    min_y, max_y = max_y, min_y if min_y > max_y
    # dx and dy are the size of the bounding box.
    # cx and cy are the center point.
    dx = max_x - min_x
    dy = max_y - min_y
    cx = (min_x + max_x) * 0.5
    cy = (min_y + max_y) * 0.5
    # Check the aspect ratio of the bounding box. If it's very wide
    # or very tall, then only split in half. Otherwise, split in 4.
    if dy > dx * 2
      # The bounding box is tall, so split in half
      handle_quadrant(depth, polygon, min_x, min_y, max_x, cy, zcta)
      handle_quadrant(depth, polygon, min_x, cy, max_x, max_y, zcta)
    elsif dx > dy * 2
      # The bounding box is wide, so split in half
      handle_quadrant(depth, polygon, min_x, min_y, cx, max_y, zcta)
      handle_quadrant(depth, polygon, cx, min_y, max_x, max_y, zcta)
    else
      # The bounding box is close to square so split in four
      handle_quadrant(depth, polygon, min_x, min_y, cx, cy, zcta)
      handle_quadrant(depth, polygon, cx, min_y, max_x, cy, zcta)
      handle_quadrant(depth, polygon, min_x, cy, cx, max_y, zcta)
      handle_quadrant(depth, polygon, cx, cy, max_x, max_y, zcta)
    end
  end
end

# Take a polygon and a box. Run the algorithm on the part of the
# polygon that falls within the box.
def handle_quadrant(depth, polygon, min_x, min_y, max_x, max_y, zcta)
  # We do this by creating a rectangle for the box, and computing
  # the intersection with the input polygon. The result could be a
  # polygon, a MultiPolygon, or an empty geometry.
  box = Zcta::FACTORY.polygon(Zcta::FACTORY.linear_ring([
    Zcta::FACTORY.point(min_x, min_y),
    Zcta::FACTORY.point(min_x, max_y),
    Zcta::FACTORY.point(max_x, max_y),
    Zcta::FACTORY.point(max_x, min_y)]))
  handle_geometry(depth, polygon.intersection(box), zcta)
end

# The main shapefile reader.
RGeo::Shapefile::Reader.open('tl_2010_53_zcta510.shp',
    :factory => Zcta::FACTORY) do |file|
  file.each do |record|
    # For each MultiPolygon, analyze it and add to the database
    handle_geometry(0, record.geometry.projection,
      record['ZCTA5CE10'].to_i)
  end
end

Now whenever we consider an optimization, we have to measure its effect. Does it actually work? And if it does, what value of MAX_SIZE should we use?

To find out, I ran the segmentation on the Washington state ZCTA data with different values of MAX_SIZE, and then ran a simple benchmark on each segmentation. The benchmark consisted of 50000 queries randomly distributed across the state. I timed the results on my laptop (an early 2011 Macbook Pro running OSX 10.6.8, Ruby 1.9.2, PostgreSQL 9.0.6, and PostGIS 1.5.3).

This first graph shows the total number of polygons (database rows) created by the segmentation process, plotted against the MAX_SIZE parameter. The original database had 622 polygons, with a maximum of 9893 sides. As our threshold on the number of sides approaches the low hundreds and smaller, the number of polygons (and hence the number of rows in the database) gets very large.

This second graph shows the total time taken by 50,000 queries against the segmented database, plotted against the MAX_SIZE parameter. The benchmark against the original database took 18.15 seconds. As we can see, decreasing the size of each polygon (by running more subdivisions) improves our query performance up to a point, where the larger number of rows becomes significant.

The sweet spot seems to be around the 300-500 side range. At that point, our optimization has cut average query times roughly in half. When I segmented the entire US database of ZCTAs for Pirq, we set MAX_SIZE to 500.

A more difficult question is, is the benefit we measured worth the extra complexity introduced by segmenting? That will depend. At Pirq, we sometimes run these queries in an inner loop, so every optimization matters. We also loaded a few much larger polygons, for which the segmentation procedure had a much more dramatic effect. So for our application, we determined that it was worth it. However, as with all benchmarking, it’s important to do your own measurements for your application.

Where to go from here

This article concludes my original outline for this series on geospatial development with Rails. But not to worry—it’s not the end. There’s still plenty of material to cover and plenty of discussion to be had. I just don’t have the next few parts already planned out yet. So this is your chance to direct where this series goes from here. If you have a topic you’d like covered, leave me a comment!

That said, there are a couple of major topics that I haven’t yet covered.

• I’ve covered vector data (i.e. points, lines, and polygons) but not raster data (i.e. image overlays). This is for several reasons. First, raster support in PostGIS is relatively new and not yet that mature. (In fact, unless you are using prereleases of PostGIS 2.0, you have to install another third-party library for raster support.) Second, the Ruby tools, notably RGeo, don’t yet support raster data either. And third, you may have noticed that a major theme of these first eight articles has been understanding coordinate systems and projections. This is critical background knowledge for handling raster data, so I though it was important to cover it first.
• I haven’t covered much on visualization tools. This is largely because my own work has been largely focused on the back-end, so I don’t yet have a lot to contribute on the view side.

I will write something on those topics in the future, but I’m not sure when I’ll get to the point where I have enough useful material. In the meantime, the floor is open for other topics!

For now I’ll conclude with links to resources on the tools that we’ve been working with during these articles.

PostGIS is the open source geospatial database of choice. It is an add-on library to the venerable PostgreSQL open source database. For more information on PostGIS, see the documentation online, and sign up for the postgis-users mailing list.

RGeo provides the core geospatial vector data types for Ruby. It is installed as the gem “rgeo“. For more information on RGeo, see the documentation online, report issues and contribute to the source at Github, and sign up for the rgeo-users mailing list.

We have also covered several other ruby gems that are used for more specialized tasks. These include:

• rgeo-shapefile (reading ESRI shapefiles). documentation / issues / source
• rgeo-geojson (reading and writing GeoJSON). documentation / issues / source
• activerecord-postgis-adapter (ActiveRecord adapter for PostGIS). documentation / issues / source

There are, of course, many other ruby libraries for other related tasks such as geocoding. Some of these will likely be the subject of future articles.

Finally, I started a mailing list for general geospatial rails discussion, the “georails” google group. Sign up if you’re interested in more community discussion.

This is part 8 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

In this article, we’ll:

• Review geographic and projected coordinate systems
• Discuss the pros and cons of using the PostGIS geographic type
• See why I typically store data in a projection
• Look at some specific projections I recommend using (or avoiding)
• Learn how to handle projected data in Rails

My original series plan for this week called for a worked example of a location-based web service, bringing together much of the material that we’ve covered so far. But as I was writing it, I realized there was one more topic we probably ought to cover first. So I’ll publish the example next week.

This is part 7 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

A tale of two coordinate systems

In part 4, we took a first look at coordinate systems. We saw that coordinate systems are different ways of assigning meaning to coordinate values. Or, put another way, any particular meaning (such as a location) can be described in multiple ways. Each of those ways would use a different set of values, according to a different coordinate system.

Locations on the earth’s surface are typically specified using one of two general types of coordinate systems: geographic coordinate systems and projected coordinate systems. Geographic coordinate systems usually use some notion of latitude and longitude, measuring angles along the surface of the earth. They are also embedded in a curved domain. What this means is, you can’t technically show latitude and longitude on a flat piece of paper or computer screen. Objects described in latitude and longitude are always curved like the surface of the earth; distances measured between latitudes and longitudes are always measured along a curved surface.

Projected coordinate systems are formed by “flattening” the earth’s surface into a flat domain. Coordinates in a projected system are not in latitude and longitude. They do not measure angles. Instead, they measure distance and position along that flattened surface. Because of this, the actual coordinate values in a projection may not be immediately recognizable. However, the benefit is that objects in a projected coordinate system are flat, so you can draw them on a flat piece of paper or computer screen, and you can perform analysis and calculations the way you are used to used to from your high school geometry class.

Here are two sets of coordinates for the Space Needle in Seattle. The first uses a geographic coordinate system, and the values are the familiar longitude and latitude. The second, called “NAD83 / Washington North”, is the state plane projected coordinate system for northern Washington state. The coordinates in this projection may not be immediately recognizable, but it points to the same location.

POINT(-122.34978 47.620578)  -- geographic
POINT(1266457.58 230052.50)  -- projected

In the beginning of part 4, we looked at some of the ramifications of using different coordinate systems. They can drastically change the way that objects are shaped or computations are done. Now we’ll look at some practical advice regarding choosing coordinate systems to use.

The PostGIS geographic type

The PostGIS database provides two different types of spatial columns: geometric and geographic. We saw in part 2 that we can specify which type to use in our Rails migrations, through the use of the :geographic modifier:

class CreateLocations < ActiveRecord::Migration
  def change
    create_table :locations do |t|
      t.string :name
      t.point :latlon, :geographic => true
      t.timestamps
    end
  end
end

Geographic columns use a geographic coordinate system (latitude and longitude on a curved domain). Geometric columns use a projected coordinate system (on a flat domain). But which should you use for your application? To answer this question, we need to unpack what the coordinate system differences mean in the context of PostGIS.

Let’s start with the obvious. Geographic types use units of latitude and longitude. Since these are familiar concepts, we can put them directly into the database and pull them out for display without having to perform any transformations on the values. This makes the geographic type very convenient for many simple applications.

Second, the shape of lines and polygons in geographic columns will follow the curvature of the earth. We saw a dramatic demonstration of this in the beginning of part 4: a “straight line” from San Francisco to Athens passes over Iceland in a geographic coordinate system, even though Iceland is far to the north of either endpoint.

Third, as a corollary to the previous point, geographic coordinates for the most part let you ignore seams and singularities. Take a short line segment from POINT(179 0) to POINT(-179 0). On the globe, in a geographic coordinate system, this is a short line that crosses the International Date Line. Projections, in contrast, have to flatten the earth, and in order to do so, they have to “cut” the globe someplace. This cut becomes the edge of the map. Many projections perform this cut along the Date Line. Hence, if we take our two points on either side of the Date Line, and draw a line segment between then in such a projection, that line would run the other way, crossing most of the world.

A line segment connecting two points on either side of the Date Line, in a geographic coordinate system.

A line segment connecting the same endpoints, in some projections, may cross the entire world.

Similarly, the north and south poles also cause problems for many projections. As a result, if you deal with objects that cross the Date Line or live near or especially surrounding the poles, you may have to deal with these (literal) edge cases specially. Generally, the geographic type lets you avoid having to think about these special cases because a globe has no edges.

Now the bad news. Computations across a curved surface are more complex than across a flat surface. Distance calculation, intersections, and so forth, will be slower on geographic types than on projections. In fact, some computations will not be available at all. In part 6, we considered an example “counties” table, in which we chose to use a projected coordinate system to store polygons. The reason I did that is that I wanted to cover ST_Relate, a function that PostGIS supports for geometric types but not geographic types.

Finally, geographic types are also subject to the model of the earth that you are using. The earth is actually not a perfect sphere, but is slightly flattened along its axis of rotation. In order to perform computations across a large area with a high degree of accuracy, you need to take that flattening into account. Unfortunately, the flattening makes the already complex computations maddeningly complex (and correspondingly slower). Because of this, PostGIS gives you the option of choosing whether to perform computations using the spherical or flattened shape, trading off speed for accuracy. Each function that supports geographic inputs performs the more accurate computations by default, but you can change it to use the faster spherical formulas by passing FALSE as an optional final parameter.

ST_Distance(pt1, pt2)         -- Uses more accurate computation
ST_Distance(pt1, pt2, FALSE)  -- Uses faster spherical computation

A case for projections

So which type should you use? There will be some cases when the decision is clear. If you need to perform computations across large sections of the globe, for example, you will usually want to use the geographic type. However, my experience has been that, for most use cases that you’re likely to encounter in a Rails application, you’ll get better results by choosing a reasonable projection.

Why do I say that?

Spatial data storage should match its usage. This is, I think, the most important but most overlooked consideration. Often, your application will lend itself to particular projection based on what it does with the data, and it is almost always beneficial to structure your data storage accordingly. I know as engineers we often want to abstract our data representation from our application functionality. But you don’t always have that luxury with big data—whether you like it or not, you have to accommodate the resource and performance needs of your database. This goes double with geospatial data, because the queries and analysis can get quite expensive.

One very common application is simply the display of your database objects on a Google Map or similar visualization tool. In such an application, most of your queries might be of the form: Give me all the objects that appear within this rectangle on a Google Map. If your data is stored and queried in the same coordinate system as that used by Google Maps, then those rectangular map areas will translate directly into simple rectangular queries in your database. If, however, your database uses a geographic coordinate system or a different projection, your query may map into a distorted or non-rectangular area in your database’s coordinate system, resulting in more complex code and/or decreased performance.

Many shapes are best represented in a (particular) projection. Let’s take a look at a shape that should be familiar to most readers, the outline of the United States:

The United States, in a Lambert Conformal Conic projection. (credit: http://csanet.org/newsletter/winter03/nlw0303.html)

Now, much of the northern border with Canada follows a line of latitude, the “49th parallel”. A straight line. Except, in the above image, it’s not straight; it’s curved slightly. This map is in a Lambert Conformal Conic projection, very commonly used for US national and state maps. To represent the northern border of the country in this projection, you would need a curved line (or, in practice, a bunch of short straight lines that together approximate a curved line.) But in some other projections—for example a Mercator projection—lines of latitude are straight, making the shape much easier and more efficient to represent.

The United States, in a Mercator projection. (credit: http://csanet.org/newsletter/winter03/nlw0303.html)

East-west and north-south lines in most political boundaries tend to follow lines of latitude and longitude, respectively, and so are best represented in a projection (such as Mercator) that preserves those lines as straight. Remember, most lines of latitude are not straight in a geographic coordinate system, so a geographic latitude-longitude coordinate system is not particularly well-suited for large political boundaries such as states and countries.

Most data is hyperlocal. The geographic type’s advantages come to the foreground when you’re dealing with data spread over the entire globe, or when you need to deal with objects covering large areas or distances covering significant portions of the globe. However, in practice I’ve found there are very few applications like that. In most cases, you’ll be dealing with primarily point data, or if you do have line or polygonal data, the individual objects are small: streets, parcel boundaries, municipal and statistical boundaries, and so forth. Furthermore, in most cases, your data will be limited to a particular part of the world, or at least you’ll seldom need to handle data that crosses seams such as poles or the Date Line. So in practice, you seldom actually run into the problems that would be solved by using the geographic type.

Performance does matter. Many operations gain a substantial performance improvement from using the PostGIS geometry type rather than the geography type. Furthermore, using geometry saves you from having to think about which functions are available and which are not.

A projection to avoid and a projection to consider

You might be tempted to store latitude and longitude in a geometry type column. That is, to set up your PostGIS column with a geometry type, but use SRID=4326 (which is the EPSG number for WGS 84 latitude and longitude).

Don’t do this.

I did this a few times in my naive youth, and it came back to bite me. What you’re really doing here is employing a particular projection called Plate Carree, which simply maps latitude and longitude directly to x and y on the plane. Remember, any time you use geometry rather than geography, you are working with a flat coordinate system, and thus a projection. You might think you’re working with latitude and longitude, but you’re actually not.

The Plate-Carree projection. (Credit: http://kartoweb.itc.nl/geometrics/Map%20projections/body.htm)

Plate Carree is not a particularly useful projection (except that it is trivial to compute). It doesn’t preserve distances, angles, directions, areas, or any other cartographically useful properties, and its distortion in polar regions is severe. In almost all cases, you can do much better with a different projection.

The projection I tend to recommend for many applications is Mercator. In particular, a minor variation on Mercator that is used by Google and Bing Maps:

The Google world map, a slight variation on a Mercator projection

This coordinate system has EPSG number 3785, and has a number of helpful properties.

• It’s used by Google maps and Bing maps (and possibly other mapping systems as well), so if you use those systems for visualization, you have a good match between your data storage and application.
• It preserves angles and shapes locally. (In cartographic terms, it is conformal.) This means if you zoom into any part of the map, the shapes and aspect ratio will closely match the real shapes on the globe. This is, I think, the primary reason it is popular with mapping visualizations.
• Lines of latitude and longitude are straight, so political boundaries tend to work well.
• It’s relatively simple to compute.

As with any projection, there will be times when this one is not appropriate. By now, you should have enough understanding to identify many of these cases. However, a few of the common objections you might encounter, are not as important as they sound, and I think I should say a few words about them.

You might hear people object to using EPSG 3785 on the grounds that it contains a simplification that introduces cartographic inaccuracies. (Specifically, it treats its underlying geography as a sphere rather than a flattened ellipsoid.) In most cases, this argument makes too much of too little. All projections rely on simplifications that introduce inaccuracies in one form or another. If your application is to bounce a laser across a continent, then by all means dig deep into the corrective factors. But for most web applications, 3785 should be more than sufficient. Indeed, the inaccuracies in most of the data you will gather, including GPS and geocoded data, will far outweigh most of what can be introduced by the projection.

You also might hear people object to using the Mercator projection at all, on the grounds that it gives a distorted picture of the nature of the world. Because the projection magnifies areas further from the Equator, it generates map images that appear to privilege richer countries in higher latitudes while downgrading the importance of poorer countries closer to the Equator. In 1989, a well-publicized resolution, signed by a number of prominent geographers, was published in American Cartographer, decrying the use of Mercator and similar rectangular projections for these and other reasons. This point is well-taken, and if you are displaying a full world map, I generally do not recommend Mercator if you can help it. However, here we are talking specifically about database structure, not visualization, so for our purposes I think the point is moot.

Working with projected data in Rails

So let’s see some code! I’ll demonstrate how to set up your PostGIS database to store data using EPSG 3785, and how to read and write data using ActiveRecord.

We’ll use our code from part 2 as a starting point. But now, in our migration, we no longer set :geographic, but instead use a geometric (flat) coordinate system with SRID = 3785, as follows. (We’ll also set up a spatial index, as we saw in part 6.)

class CreateLocations < ActiveRecord::Migration
  def change
    create_table :locations do |t|
      t.string :name
      t.point :loc, :srid => 3785
      t.timestamps
    end
    change_table :locations do |t|
      t.index :loc, :spatial => true
    end
  end
end

We also need to specify a corresponding factory in our ActiveRecord class. Here I’m going to introduce a rather dirty little feature of RGeo: “projected geographic” factories. Now, if you cringed a little at that description, then you’re getting the hang of coordinate systems. Geographic coordinate systems are by definition not projected! However, sometimes when you’re working with a projection, you’ll want a quick way to interact with the data in latitude and longitude—a quick way to transform individual points to geographic coordinates and back again. This is where RGeo’s projected geographic factories come in handy.

These factories really use a projected coordinate system under the hood. In fact, they reference a full Cartesian factory internally, and you can gain access to that “real” projected factory by calling the projection_factory method. However, they provide you with a convenience interface that lets you look at the data as latitudes and longitudes, as if it were a geographic factory.

The “simple_mercator” factory is a useful example. Its “real” internal factory has SRID 3785, indicating the Google Maps style Mercator projection, but the wrapper factory reports latitudes and longitudes. In this way, it mirrors the Google Maps Javascript API. It talks latitudes and longitudes on the outside, but converts them internally to the projection for use with the map.

In our ActiveRecord class, we’ll set up the factory so it correctly interacts with the database in projected coordinates.

class Location < ActiveRecord::Base

  # Create a simple mercator factory. This factory itself is
  # geographic (latitude-longitude) but it also contains a
  # companion projection factory that uses EPSG 3785.
  FACTORY = RGeo::Geographic.simple_mercator_factory

  # We're storing data in the database in the projection.
  # So data gotten straight from the "loc" attribute will be in
  # projected coordinates.
  set_rgeo_factory_for_column(:loc, FACTORY.projection_factory)

  # To interact in projected coordinates, just use the "loc"
  # attribute directly.
  def loc_projected
    self.loc
  end
  def loc_projected=(value)
    self.loc = value
  end

  # To use geographic (lat/lon) coordinates, convert them using
  # the wrapper factory.
  def loc_geographic
    FACTORY.unproject(self.loc)
  end
  def loc_geographic=(value)
    self.loc = FACTORY.project(value)
  end

end

Now let’s do an example query. Suppose our basic query is a simple map search where we want to return all the locations in a given rectangle on our map visualization. Since our data is in the same projection as the original map, a rectangular query in the map translates into a rectangular query in our database. So we’ll take the latitudes and longitudes of the rectangle edges as parameters, and convert them to projected coordinates. Once there, we can use a simple PostGIS box intersection to run the query itself. It’s a simple query that can be accelerated using the spatial index.

We’ll add a scope to our class as follows:

class Location < ActiveRecord::Base

  # ...

  # w,s,e,n are in latitude-longitude
  def self.in_rect(w, s, e, n)
    # Create lat-lon points, and then get the projections.
    sw = FACTORY.point(w, s).projection
    ne = FACTORY.point(e, n).projection
    # Now we can create a scope for this query.
    where("loc && '#{sw.x},#{sw.y},#{ne.x},#{ne.y}'::box")
  end

end

Now rectangle searches are simple:

locations = Location.in_rect(-122, 47, -121, 48).all

Where to go from here

In this article, we saw some of the pros and cons of using different coordinate systems for your database. The right coordinate system will depend on your application, but I’ve found that for many applications, using a projection—often the specific projection EPSG 3785—produces good results.

It may be useful at this point to gain a general feel for the different types of projections, how they work, and what their pros and cons are. A very good online resource for this is provided here by the USGS.

The RGeo::Geographic.simple_mercator_factory is useful for storing data in EPSG 3785. However, if you want to use a different projection under the hood, you can use a more powerful method, RGeo::Geographic.projected_factory, which lets you specify arbitrary projections using Proj4. Read about it in the RGeo documentation.

Next time, I will get to the worked example I promised last week. Stay tuned, and let’s bring Rails down to earth!

This is part 7 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

In this week’s article, I’ll go over the basic issues every geospatial programmer should know about scaling, and provide tips for writing your geospatial Rails application so it doesn’t fall over when you go national. We will cover:

• The bottom line regarding scaling
• Building spatial indexes for your database
• Writing queries to take advantage of indexes
• Simplification and segmentation of large objects

This is part 6 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

Scaling is complex but not difficult

Scaling is a high-profile issue. We all notice when our blog gets slashdotted, and when Twitter or Amazon goes down it makes national news. Failures happen to everyone, and seem almost inevitable. Are they?

Now, I don’t want to downplay the complexity of the scaling task, but we do have to start with an important observation. Scaling, in its essence, is a solved problem. The techniques involved have been well-understood for decades. We all learned about logarithmic searching, branch and bound, and similar algorithms in our computer science classes. And as web developers, we should already know what these algorithms look like in practice: database indexes, sharding, caching, replication, load balancing, and so forth.

So why the hoopla?

Because scaling, though a solved problem, is not an automatically solved problem. It requires our attention. More to the point, it requires that we understand every aspect of the system we are building, how the various components work and how they interact. If we need our database to scale, at some level we need to understand how it works, how we’re using it, and thus what we need to do to make it scale.

This is probably the main reason why Rails has historically had a negative reputation about scaling. Rails purports to make web development simple. But that’s a bit misleading. If you think about it, a website is a complex system with a lot of moving parts: the network, the server, the MVC control flow, databases, caches, client-side code, control flow from one page to another, interactions with external systems, security layers…, and that doesn’t even include your application logic with all of its complexities. It’s hardly simple. Rails tries to deal with this complexity by hiding elements, at least partly. It hides the database behind ActiveRecord, and in doing so trains us not to think about the database. But that’s a deceit. We have to think about the database if we’re going to scale it.

And that is one of the key motivations behind this entire series. I’ve heard some comments that this material is difficult, that there’s no TL;DR. Yes, the material is difficult, and I’m not going to try to hide or gloss over that fact. I could try to make geospatial programming really easy, creating a one-size-fits-all tool or recipe for everyone to cargo-cult. But that is the wide road that leads to destruction. Eventually, you’ll need to figure out how to scale, and at that time, if you don’t have some understanding of what’s going on under the hood, you’ll get very stuck very quickly.

Now, that said, dealing with geospatial features does not fundamentally change the scaling task. Scaling is still a solved problem. As we prepare to scale our applications, there is a well-known, systematic process we all go through. We measure, find the bottlenecks, apply well-understood techniques to address those bottlenecks, and repeat. It can be a tedious process, and (believe me, I know) it is sometimes difficult to sell our business partners on the fact that we need to spend time on it before it’s too late. But it’s not like we don’t know what we’re doing. Scaling is complex, but not difficult. It simply requires that we have a general understanding of how spatial data works.

So, sermon over, let’s dive in.

About spatial database indexes

Spatial data is often big data, and as with any big data, our basic scaling task involves making it smaller.

In a database, this is generally accomplished by judicious use of indexes. An index provides a fast way to look up data by some criteria, without having to read and compare against every single row. For example, if your table has a million rows, each identified by a numeric ID, you can generally speed up ID lookups by creating an index on that column.

Similarly, spatial database indexes can accelerate queries that include spatial criteria. If your million-row table also contains latitude-longitude coordinate, and you want to find rows whose coordinate falls within a certain region, you should consider building a spatial index on the coordinate column. This allows your spatial search to avoid checking every row in the database, thus speeding up your queries.

The important thing to understand about spatial indexes is that although they are conceptually the same as “standard” database indexes, they are implemented differently under the hood. In most databases, a simple index on an ID column will use an algorithm known as a B-tree. Such an index relies on a global ordering of the data, and builds a balanced binary tree, which, as we remember from computer science, lets us do lookups in logarithmic time.

Spatial data, however, has some important differences from normal scalar data. A simple numeric ID is an infinitesimal point on a one-dimensional number line, whereas a polygon is a finite area on a two-dimensional surface. For data that covers finite areas or lives in more than one dimension, a B-tree does not work. We have to resort to a more complex algorithm, usually a variant on what is known as an R-tree.

I’ll save the gory details on R-trees for a later article on spatial index design, but there is one upshot you’ll need to understand: spatial indexes are heavier and more expensive than standard indexes. An R-tree takes up more space in memory and on disk than a similarly-sized B-tree. Queries against an R-tree can be a little slower than against a B-tree, and R-tree updates can be considerably slower. However, R-trees still provide logarithmic-time queries, and so will still give you speed-ups in many situations. So the usual database mantra still applies, and indeed goes double for spatial indexes: Index your common queries but don’t index everything. And of course, Measure, Measure, Measure.

Because R-tree updates can be slow, it is also usually a good idea to remove or disable a spatial index if you are going to be loading a lot of data, and then turn it back on once you are done. In this way, you pay the cost of building the index only once at the end, rather than having to incrementally update it on every insert.

Creating and using spatial indexes

Because a spatial index is constructed differently from most indexes, creating one usually requires a special syntax. For a Rails project, you can usually let RGeo’s ActiveRecord adapters handle this for you. Create a spatial index in a migration simply by providing the :spatial attribute. Following is a snippet from a migration that creates a “counties” table with polygons, along with a spatial index on the polygons. (Here we’ll use geometric column with the “NAD83 / Washington North” projection, which has SRID 2285—see part 4. If you’re using PostGIS, indexes do work for the geographic type, but have some limitations.)

create_table :counties do |t|
  t.string :name
  t.polygon :poly, :srid => 2285
end
change_table :counties do |t|
  t.index :poly, :spatial => true
end

If you are managing your schema manually, you’ll need to use the database’s particular syntax. In PostGIS, spatial indexes use the GIST framework, so you denote a spatial index with “USING gist”:

CREATE INDEX "counties_poly_idx" ON "counties" USING gist ("poly");

MySQL‘s spatial extension defines a separate index type:

CREATE SPATIAL INDEX `counties_poly_index` ON `counties` (`poly`);

In SpatiaLite, a spatial index is actually a separate table that you must join to. Creating a spatial index involves calling a special function provided by the SpatiaLite library:

SELECT CreateSpatialIndex('counties', 'poly');

Once you’ve created a spatial index, it is usually a good idea to verify that your queries will take advantage of it. You’ll want to make sure the database’s query planner, the component that analyzes a query and decides how to attack it, is producing an optimal plan. This is generally good practice for all your database design, but more so for spatial queries because they are less commonly used, and query planners do not always do as good a job with them as we would like.

Your best tool for interacting with the query planner, whether or not you’re using a spatial database, is EXPLAIN. This SQL command takes a query and returns the query planner’s plan of attack for that query, usually including which indexes it intends to use and its estimate of how expensive the query will be.

For most databases, you can invoke the EXPLAIN command simply by prefixing your query with “EXPLAIN“. For example, using PostGIS, let’s see what the query planner does with a query asking for the county containing the Seattle Space Needle:

EXPLAIN
  SELECT "name" FROM "counties" WHERE
    ST_Intersects("poly", ST_GeomFromEWKT('SRID=2285;POINT(1266457.58 230052.50)'));

Postgres will return a query plan that looks something like this:

QUERY PLAN
----------------------------------------------------------------------------------------------
Index Scan using counties_poly_idx on counties (cost=0.00..8.52 rows=1 width=68)
  Index Cond: (poly && '0101000020ED08000048E17A94195333410000000024150C41'::geometry)
  Filter: _st_intersects(poly, '0101000020ED08000048E17A94195333410000000024150C41'::geometry)

Note that it’s using the “counties_poly_idx” index that we created. PostGIS is currently quite good about knowing how to use a spatial index for most queries. With the EXPLAIN command, we can be sure this query will use our spatial index for maximum efficiency.

Optimizing difficult queries in PostGIS

Unfortunately, there are a few cases when the query planner won’t be able to figure out by itself that an index is useful. For example, suppose we want to perform a sanity check of our counties database, making sure we don’t have any overlapping polygons. More precisely, while we expect that county polygons will “touch”—that is, share boundaries—we don’t want counties to actually share interior points. That could mean a problem in our data.

Unfortunately, PostGIS doesn’t provide this kind of “interior intersection” function out of the box. The ST_Intersects function we used earlier will flag the “touch” case as well, and we don’t want that.

But we can build “interior intersection” using the function ST_Relate. This powerful function lets you test a wide variety of relationships using the Dimensionally Extended Nine-Intersection Model. I won’t cover this model in detail for now—you can read about it in the Simple Features Spec. For our purposes, what’s important is that, by giving it a particular specification string, “T********“, it can implement the relationship we want to test.

Unfortunately, because ST_Relate is such a powerful and general tool, the query planner can’t optimize it very well, and tends to fall back on the lowest common denominator, which is sequential scan.

EXPLAIN
  SELECT c1.name, c2.name FROM counties AS c1 INNER JOIN counties AS c2
    ON c1.id != c2.id AND ST_Relate(c1.poly, c2.poly, 'T********');

QUERY PLAN
----------------------------------------------------------------------
Nested Loop (cost=0.00..10372.17 rows=229633 width=64)
  Join Filter: st_relate(c1.poly, c2.poly, 'T********'::text)
  -> Seq Scan on counties c1 (cost=0.00..18.30 rows=830 width=68)
  -> Materialize (cost=0.00..22.45 rows=830 width=68)
       -> Seq Scan on counties c2 (cost=0.00..18.30 rows=830 width=64)

Ouch! That’s an unfortunate query plan. It does nested sequential scans, comparing every county polygon with every other county polygon, an n-squared operation. In a table with thousands of counties, this can be slow.

But it turns out we can do better. The “interior intersection” operation actually can be optimized using the index. The query planner doesn’t realize this, so we need to give it some help.

Here a bit of trivia about spatial indexes will help us. In general, the “native” operation for an R-tree index is bounding box intersection. It can take the bounding box of an input geometry, and determine which geometries in the table have bounding boxes that intersect the input. At the most basic level, the query planner for PostGIS works by looking for opportunities to apply this native operation. It asks, “How can I reduce the search space by applying a bounding box intersection?”

In our first example above, when we used ST_Intersects to find the polygon containing the Space Needle, the query planner reasoned thus: Every time two geometries intersect, their bounding boxes also intersect. So I can add a bounding box intersection and not change the result of the query. I like bounding box intersections because they let me use the index. So actually what happened behind the scenes, was that PostGIS rewrote our query from:

SELECT "name" FROM "counties" WHERE
  ST_Intersects("poly", ST_GeomFromEWKT('SRID=2285;POINT(1266457.58 230052.50)'));

to:

SELECT "name" FROM "counties" WHERE
  "poly" && ST_GeomFromEWKT('SRID=2285;POINT(1266457.58 230052.50)') AND
  ST_Intersects("poly", ST_GeomFromEWKT('SRID=2285;POINT(1266457.58 230052.50)'));

…using the PostgreSQL operator for bounding box intersection: “&&“. Now, when it creates the actual query plan, it uses the spatial index to optimize the bounding box intersection. Let’s take another look at that query plan:

QUERY PLAN
----------------------------------------------------------------------------------------------
Index Scan using counties_poly_index on counties (cost=0.00..8.52 rows=1 width=68)
  Index Cond: (poly && '0101000020ED08000048E17A94195333410000000024150C41'::geometry)
  Filter: _st_intersects(poly, '0101000020ED08000048E17A94195333410000000024150C41'::geometry)

See the bounding box intersection “&&” in the Index Condition? That wasn’t in our original query, but PostGIS rewrote our query and put it there so it could use the index. Pretty clever, PostGIS is.

Well, sometimes PostGIS isn’t quite clever enough, and we have to give it some help. In our “interior intersection” example, we can improve the query plan by going through this process manually. We reason thus: PostGIS doesn’t realize this, but every time two geometries have an “interior intersection” using ST_Relate, their bounding boxes also intersect. So I can add a bounding box intersection and not change the result of the query. Bounding box intersections are good because they let me use the index.

So let’s manually rewrite our query from:

SELECT c1.name, c2.name FROM counties AS c1 INNER JOIN counties AS c2
  ON c1.id != c2.id AND ST_Relate(c1.poly, c2.poly, 'T********');

to:

SELECT c1.name, c2.name FROM counties AS c1 INNER JOIN counties AS c2
  ON c1.poly && c2.poly AND
  c1.id != c2.id AND ST_Relate(c1.poly, c2.poly, 'T********');

Now we give this to PostGIS, and voila! The query planner now uses the index:

EXPLAIN
  SELECT c1.name, c2.name FROM counties AS c1 INNER JOIN counties AS c2
    ON c1.poly && c2.poly AND
    c1.id != c2.id AND ST_Relate(c1.poly, c2.poly, 'T********');

QUERY PLAN
----------------------------------------------------------------------------------------
Nested Loop (cost=0.00..296.81 rows=1 width=64)
  Join Filter: st_relate(c1.poly, c2.poly, 'T********'::text)
  -> Seq Scan on counties c1 (cost=0.00..18.30 rows=830 width=68)
  -> Index Scan using counties_poly_idx on counties c2 (cost=0.00..0.32 rows=1 width=68)
       Index Cond: (c1.poly && c2.poly)

This improved plan still does one full sequential scan, because it still has to check every county in the database. But the nested scan, which checks whether that county overlaps any other county, is now accelerated using the index. We’ve reduced the n-squared query to an n log n query. The computer scientist in us rejoices!

Going through this process does require some creativity, and it helps to have a bit of experience. The good news is that PostGIS is smart enough to handle most cases automatically. But you should still make liberal use of the EXPLAIN tool and look carefully at the query plan that is generated, to see if it’s doing as well as you think it ought. There may be opportunities to improve your query performance dramatically just by giving it a little bit of help.

Indexing and queries in MySQL and SpatiaLite

Generally, I recommend PostGIS as an open source spatial database. But there are others out there that you may need to use from time to time, and each one will have its quirks.

As we’ve seen, the spatial extensions to MySQL do support spatial indexes. However, there are some significant limitations in comparison with PostGIS.

First, spatial indexes currently work only on MyISAM tables. This means you can’t use spatial indexes and get the transaction safety benefits of InnoDB on the same table. Ugh.

Second, MySQL supports only a very limited set of spatial relationship functions. In particular, nearly all MySQL’s functions work on bounding boxes (which MySQL calls Minimum Bounding Rectangles, or MBR) rather than the geometry itself. So for example, MySQL’s Intersects function is actually only an alias to MBRIntersects, which tests the bounding boxes for intersection. If you want to test actual geometric intersection, you’ll have to do some post-filtering on the result set (which you can do using RGeo).

I generally don’t recommend using MySQL Spatial unless you’re already using MySQL. But then I typically don’t recommend using MySQL in general either…

SpatiaLite is a set of spatial extensions to the popular SQLite database. I haven’t used SpatiaLite much. It does seem to have a fairly complete feature set, at least in comparison with MySQL Spatial, though it doesn’t compare in maturity with PostGIS.

Spatial indexes in SpatiaLite are a bit of a pain, however. They are implemented as a separate set of managed join tables tied to your main table using triggers. All this is handled fairly transparently, except for queries. When you want to write a query that takes advantage of a spatial index in SpatiaLite, you must explicitly join to the index table.

For the sake of space, I won’t go into the details here. Instead, I highly recommend an excellent online publication by the author of SpatiaLite, the SpatiaLite Cookbook, which serves as the user’s manual for SpatiaLite, and provides a number of very helpful examples.

Simplifying and segmenting data

But wait—there’s more!

Remember that the basic scaling task is to make big data smaller. If we have big data, we try to do clever things, such as applying indexes, so that we don’t have to analyze all the data at once.

Now, there are two ways in which spatial data can be “big”. First, there may be a lot of objects, lots of rows. In this case, we can often speed up queries by adding a spatial index, as we have seen.

However, individual objects can also be “big”, particularly when you’re dealing with polygons. Take our table of county polygons. Some county boundaries are simple polygons with just a few sides, but many others have complex, crinkly boundaries that follow rivers, coastlines, mountain divides, or other natural features. The number of sides in such polygons can quickly rise into the thousands or more. When you want to compute, say, an intersection with such a polygon, it can be slow.

There are several different strategies you can use to address this problem. I’m just going to summarize a couple of the important ones here. But first I need to emphasize one thing. There is no one-size-fits-all solution. Each approach has its pros and cons, and your choice will depend on the requirements of your particular application.

To this end, measurement is absolutely critical. Before, during, and after applying any optimization technique, run a benchmark and make sure that (1) you’re addressing the right problem, and (2) the performance is going in the right direction. This is doubly important when dealing with spatial data, because the algorithms involved are somewhat more complex, and it may surprise you what’s fast and what’s slow.

There are two general techniques for dealing with large polygons: simplification and segmentation.

Simplification can be applied when you’re more concerned with speed than accuracy. For example, you might have a polygon with a thousand sides, but if you’re going to be displaying it in a relatively small area on a map, or you’re running some spatial queries where you don’t care too much if you’re a little off, then you can probably get away with an approximation of the polygon with fewer sides.

An example of polygon simplification for part of the coast of France. (Credit: http://vis4.net/blog/posts/rendering_country_maps/)

There are a number of polygon simplification techniques out there, useful for different circumstances. I don’t have space here for a full discussion, but I may write a specialized article on simplification at a later time, because it’s an interesting (and sometimes tricky) problem.

Segmentation is often useful for speeding up queries against big polygons, when you care not about the shape of the polygon itself but merely whether you’re intersecting it. Segmentation, for example, might be useful in our county boundary example.

The idea is to break up large polygons into smaller polygons that can be stored in separate rows in your database. That is, we trade “width” of the data (i.e. how big each object is, in terms of number of vertices) for “length” (i.e. how many objects there are). Since we have spatial indexes that mitigate big “length”, this trade-off can be a win for us.

There are many ways to break up a large polygon. The simplest approach, usually good enough in practice, is to perform recursive four-to-one subdivision. Don’t be scared off by the name; it’s actually quite straightforward. The idea is to take your large polygon with many sides, and split it down the middle horizontally and vertically. This will typically result in four polygons, each covering about a quarter of the area and containing about a quarter of the number of sides:

One four-to-one split of a province in France.

Now, if any of the four polygons still has too many sides, you can do the same thing again, recursively, and so forth until you reach a number of sides that you’re comfortable with. Once you’re done, the union of all the resulting polygons will still be your original polygon. So, in our counties example, a county now has_many polygons, and to find the county containing a particular point, do a spatial query for the polygon containing that point and map back to the county.

So how deeply should you segment a polygon? What’s the “sweet spot” in the number of sides? That’s where you have to test and measure, because it will depend on many factors. In one recent project in which I did some polygon segmentation, I measured the optimal number of sides between three and five hundred. But your mileage will vary.

Where to go from here

It is worth diving into the manual for your spatial database for tips on the effective use of spatial indexes. The PostGIS manual is online.

EXPLAIN is a very powerful tool for studying and optimizing your database performance in general, not only when you’re working with spatial data. I highly recommend getting familiar with using it in your database. For PostgreSQL, a good place to start is the Using Explain page in the manual. SQLite and MySQL also have sections in their manuals. Additionally, it looks like Rails 3.2 will include some useful EXPLAIN-based tools out of the box.

This week’s article didn’t include a lot of code. This was because I had a lot of material to get through, and I decided it was more important to cover the concepts at this stage, rather than encourage code cargo-culting. In next week’s article, however, I’ll go through a fully worked example, with code, that mirrors an actual task I recently had to do for my job at Pirq.

Until then, have fun and let’s bring Rails down to earth!

This is part 6 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

Changes include:

• The WKRep WKT parser recognizes MultiPoint WKTs in which individual points are not contained in parens. This syntax is technically incorrect, but we are now supporting it because there was some ambiguity due to an error in early versions of the spec, and apparently there are now examples in the wild. (Reported by J Smith.)
• The Geos CAPI implementation sometimes returned the wrong result from GeometryCollection#geometry_n. Fixed.
• Fixed a hang when validating certain projected linestrings. (Patch contributed by Toby Rahilly.)
• Several rdoc updates (including a contribution by Andy Allan).
• Separated declarations and code in the C extensions to avert warnings on some compilers.

RGeo is a spatial data library for Ruby, providing full implementations of the standard spatial data types. It is the basis for a suite of useful gems for writing geospatial applications in Ruby and Rails. For more information, see the documentation at http://virtuoso.rubyforge.org/rgeo/README_rdoc.html.

In this article, we will survey some of the important spatial data formats, including serialization, file formats, and api-oriented formats. Specifically, we will look at:

• Basic serialization using WKT and WKB
• Variants on WKT and WKB
• Reading public datasets from shapefiles
• Web service oriented formats such as GeoJSON
• XML-based formats commonly used in web services

We will also go over a few quick examples using Ruby and RGeo. This will be a fairly high-level overview and we won’t go into a lot of detail. We’ll take deeper looks at some of these formats in future articles.

This is part 5 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

The standard OGC serialization formats

If, after reading part 3, you looked through the Simple Feature interfaces (or the corresponding RGeo interfaces), you may have noticed two serialization methods provided for geometries: as_text and as_binary. These methods respectively output the “Well-Known Text” and “Well-Known Binary” representations of the geometry. These two standard serialization formats are defined by the OGC Simple Feature Access specification, and commonly supported by most GIS systems.

Well-Known Text (often abbreviated WKT) is a human-readable and parseable text-based format for all geometry objects. You can read the exact format specification in the Simple Features Spec, but a few examples are probably sufficient to get the general hang of it.

Point(-122.1 47.2)
LineString(2 4, 5 4, 5 8, 2 4)
Polygon((0 0, 5 0, 5 5, 0 5, 0 0), (2 2, 2 3, 3 3, 3 2, 2 2))
MultiPoint((-122.1 47.2), (-93.5 39.4))
GeometryCollection(Point(3 5), LineString(-2 0, -3 -4))
MultiLineString EMPTY

Don’t confuse the simple features WKT format with the coordinate system WKT format we covered in part 4. Unfortunately, both are commonly known as Well-Known Text (WKT), but they are distinct formats: one represents geometric objects whereas the other represents coordinate systems.

Well-Known Binary (often abbreviated WKB) is a binary format that uses numeric codes and IEEE floating-point representations. It is not human-readable but is much more compact than WKT.

Using RGeo, you can obtain the WKT and WKB representations of a geometric object by calling as_text and as_binary, respectively. Factory objects will provide methods to parse WKT and WKB format and recover the geometric object.

point = factory.point(1, 2)
wkt = point.as_text   # => "Point(1 2)"
point2 = factory.parse_wkt(wkt)
point == point2       # => true

Variants on WKT and WKB

As simple and well-supported as they are, WKT an WKB have several important weaknesses that have caused headaches for spatial databases and applications. In part 4, we saw that to properly interpret a geometric object, you need to know the coordinate system, which is usually specified by a spatial reference ID (SRID). Unfortunately, neither WKT nor WKB include a way to represent SRID. They expect SRID to be specified or implied elsewhere, which is sometimes but not always true.

Furthermore, some applications use additional coordinates in their geometric data. Applications that store altitude or other third-dimensional data may include a “Z” coordinate in their geometries. Other applications may include a measurement (such as temperature or population) stored in an “M” coordinate. Version 1.1 of the Simple Features Spec (and the corresponding WKT and WKB specifications) do not directly support these extra coordinates, (although version 1.2 does address this, as we will see.)

Finally, neither WKT nor WKB by themselves provide a way to associate metadata, such as object names or other properties, with geometric objects. This limits their usefulness as a complete format for data transmission.

Because of these limitations, several variants have appeared that you should be aware of. The PostGIS database supports an extension to WKT called “EWKT”, which supports SRID as well as “Z” and “M” coordinates. The SRID, if present, appears at the front of the EWKT string:

SRID=4326;Point(-122.34978 47.62058)

EWKT supporting “Z” and “M” coordinates to be appended to each pair of coordinates as third and fourth coordinate values. When both “Z” and “M” are present (i.e. four coordinate values per point), the third coordinate is used for “Z” while the fourth is used for “M”. If only one is used (i.e. three coordinate values per point), you must specify whether it is “Z” or “M”. Here are some examples:

Point(-122.34978 47.62057 20.0 -3)  # X,Y,Z,M in EWKT
PointM(-122.34978 47.62057 -3)      # X,Y,M
PointZ(-122.34978 47.62057 20.0)    # X,Y,Z

PostGIS also defines a corresponding “EWKB” format with appropriate extensions to the binary format to support SRID as well as Z and M. EWKB is (or at least appears to be) the native internal format used by PostGIS to represent geometric data.

More recent versions of the OGC Simple Features Spec (version 1.2 and later) also provide support for Z and M. However, beware that the OGC format is not the same as the PostGIS EWKT and EWKB. The WKT update expects a space between the geometry type and the Z/M specifier, and it also requires the modifier in the “four-dimensional” ZM case:

Point ZM(-122.34978 47.62057 20.0 -3)  # X,Y,Z,M in WKT 1.2
Point M(-122.34978 47.62057 -3)        # X,Y,M
Point Z(-122.34978 47.62057 20.0)      # X,Y,Z

Furthermore, the updated WKT format still does not support a SRID. The updated WKB similarly supports Z and M (but not SRID), but uses different binary codes than those used by EWKB. Hence, these two extensions are not fully compatible with each other.

Because of this fragmentation, neither of these extensions are, in practice, used frequently for long-term serialization. However, you will likely need to work with EWKT at some point if you use PostGIS, so it is important to be familiar with it.

RGeo provides support for parsing and generating both variants in the RGeo::WKRep module. See the rdocs for more details. Here’s a really quick code example as a starting point:

parser = RGeo::WKRep::WKTParser.new(nil, :support_ewkt => true)
point = parser.parse('SRID=4326;Point(-122.1 47.3)')
point.srid   # => 4326

Shapefiles and public datasets

Location is driven by data, and a lot of the data you will need to work with will likely come in the form of shapefiles. The shapefile is a flat file format for geospatial data originally developed by ESRI for storing sets of geographic features. It supports certain vector shapes– points, lines, and polygons– along with associated attributes. Although shapefile began as a proprietary format, the format specification is readily available, and it is now a de facto standard for large datasets, including those provided by government agencies such as the US Census Bureau.

A shapefile actually consists of three (and sometimes more) related files, each with the same base filename but different extensions. The main file has the extension “.shp” and contains the geometric data itself in a binary format. An auxiliary “.shx” file provides a simple flat index allowing random access into the shapefile. A second auxiliary “.dbf” file provides the attribute data in dBASE format. All shapefiles should have those three core files, although some shapefiles may include additional files containing coordinate system, spatial index, or other related information.

Most Rails applications will not read a shapefile directly, but will instead transfer the data to a spatial database such as PostGIS for rapid query and data retrieval. In Ruby, you can use the rgeo-shapefile gem to help with this task. This gem does the heavy lifting involved with parsing and analyzing a shapefile, and exposes the data to you as RGeo geometric objects. You should also install the dbf gem, which lets you read the dBASE attributes in the shapefile.

% gem install rgeo-shapefile
% gem install dbf

Once you have the gems installed, and you’ve downloaded and unpacked a shapefile, use the RGeo::Shapefile::Reader class to open and read the file. The following example reads objects sequentially:

factory = RGeo::Geographic.spherical_factory(:srid => 4326)
RGeo::Shapefile::Reader.open('myfile.shp', :factory => factory) do |file|
  file.each do |record|
    geom = record.geometry
    # geom is now an RGeo geometry object.
    name = record['Name']
    # You can read any other attribute similarly.
    # Now, you can do whatever you want with the data,
    # such as inserting rows into your database...
  end
end

Notice that we provide a factory for the objects being read. Shapefiles generally do not provide an SRID, so we must supply that. The above example assumes the shapefile contains latitude-longitude coordinates in WSG84.

The RGeo::Shapefile::Reader class also lets you do random access reads, and get other information about the shapefile’s contents. See the rdocs for more details. The gem does not currently support writing shapefiles, but that feature is on the roadmap.

For more information on the shapefile format itself, you can find the original ESRI specification at http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf. Another common (C-based) implementation of shapefile is Shapelib, which you can find at http://shapelib.maptools.org/.

Web services and GeoJSON

Another way to obtain location data is to call a web service such as Google Places, SimpleGeo, or Factual. These services do the heavy lifting of curating, deduping, and managing location data, and generally provide an http REST api letting you query for location information of interest.

There are a number of different types of web services, including geocoders, point of interest search, location properties, and others. I’ll write up a survey of useful location-oriented web services in a later article. For this current article, however, we are interested in data formats that would typically be returned from a point of interest search. When you make a query, what sort of data can you expect to get?

In many cases, the web service will define its own schema for the returned data. You must then parse the returned document yourself to extract the information you want. There are well-known gems available for this task, such as json for parsing JSON, and nokogiri for parsing XML. There are also, however, a few semi-standard schemas commonly used by a number of web services. Here we will take a quick tour of some of these formats and how you can go about using them.

GeoJSON is an important emerging standard commonly used by SimpleGeo and similar modern APIs. It provides a standard JSON representation for each geometric type, as well as support for bounding boxes, coordinate systems, and a set of properties. Following is an example of GeoJSON, lifted out of the specification:

{ "type": "FeatureCollection",
  "features": [
    { "type": "Feature",
      "geometry": {"type": "Point", "coordinates": [102.0, 0.5]},
      "properties": {"prop0": "value0"}
      },
    { "type": "Feature",
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0]
          ]
        },
      "properties": {
        "prop0": "value0",
        "prop1": 0.0
        }
      },
    { "type": "Feature",
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [ [100.0, 0.0], [101.0, 0.0], [101.0, 1.0],
            [100.0, 1.0], [100.0, 0.0] ]
          ]
      },
      "properties": {
        "prop0": "value0",
        "prop1": {"this": "that"}
        }
      }
    ]
  }

The core object type in GeoJSON is the Feature, which consists of a geometry and a set of properties. The geometry can be any of the OGC types, and its internal representation is closely modeled on WKT. Properties are simply named key-value pairs whose values can be any JSON object.

GeoJSON is simple and highly versatile, and is often an ideal format both for consuming and producing geospatial data. From Ruby, you can use the rgeo-geojson gem to read and write GeoJSON. Here are some quick examples to get you started:

require 'rgeo/geo_json'

str1 = '{"type":"Point","coordinates":[1,2]}'
geom = RGeo::GeoJSON.decode(str1, :json_parser => :json)
geom.as_text              # => "POINT(1.0 2.0)"

str2 = '{"type":"Feature","geometry":{"type":"Point","coordinates":' +
  '[2.5,4.0]},"properties":{"color":"red"}}'
feature = RGeo::GeoJSON.decode(str2, :json_parser => :json)
feature['color']          # => 'red'
feature.geometry.as_text  # => "POINT(2.5 4.0)"

hash = RGeo::GeoJSON.encode(feature)
hash.to_json == str2      # => true

For more information on GeoJSON, see http://geojson.org/. The actual spec hosted on the website is quite short and very readable. You can find more information on the rgeo-geojson gem from its rdocs.

XML-based formats

Although JSON is often a format of choice for many modern web services because of its simplicity and its close affinity with Javascript and similar high-level languages, XML is still the established standard in many fields and applications. GIS services, in particular, have a long tradition of XML-based representation, and there are a number of XML-based geospatial formats you may encounter when writing location-aware applications. Among them:

GeoRSS is a family of RSS extensions for embedding geospatial data into RSS or Atom feeds, often used to spatially tag feed entries. It comes in two flavors, Simple GeoRSS and GML GeoRSS. Simple GeoRSS is designed for simplicity, and supports a limited set of features. Notably, not all the OGC geometric types can be represented, and coordinate system is limited to WGS84 latitude/longitude. GML GeoRSS is a more full-featured but much more complex format, essentially a profile of GML, which we will cover below. Most actual implementations of GeoRSS are of the Simple flavor.

Below are a couple of examples of a basic GeoRSS element from an RSS feed, first in the Simple flavor and then in the GML flavor.

<georss:point>47.604828 -122.330779</georss:point>

<GeoRSS:where>
  <gml:Point>
    <gml:pos>47.604828 -122.330779</gml:pos>
  </gml:Point>
<GeoRSS:where>

As of this writing, the georss.org website appears to be unmaintained and possibly hacked. The best starting point I can recommend for GeoRSS is an OGC whitepaper at http://portal.opengeospatial.org/files/?artifact_id=15755.

I’m not currently aware of an RGeo-based Ruby implementation of GeoRSS. The older GeoRuby gem, however, does have basic support for GeoRSS.

Geography Markup Language (or GML) is an XML-based object model intended to describe geographic information. Its specification is maintained by the Open Geospatial Consortium. GML by itself is a highly general and flexible model that can represent not only geometric objects and coordinate systems such as we have looked at so far in this article series, but also observations, topological information, temporal information, and various other related entities.

You generally don’t work with GML directly, but instead use an application XML schema that references GML internally. Furthermore, most application schemas don’t utilize the entire GML specification, but a relevant subset, known as a GML profile. For example, GML GeoRSS is an application schema referencing a GML profile relevant to geotagging feed entries.

Another common GML-based XML schema is CityGML (http://www.citygml.org/), which is designed to model urban objects. CityGML is commonly used, for example, to model 3D visualizations of cities.

For more information on GML as a whole, you can review the OGC spec at http://www.opengeospatial.org/standards/gml.

I’m not currently aware of any specific Ruby support for GML or its various dialects.

KML (or Keyhole Markup Language) is an XML schema that originated at Google for describing features in Google Earth, but was later standardized by the OGC. Although it does have some overlap with GML, KML is often seen as complementary because of its particular emphasis on visualization. Its intended use is to describe how to display features within a Google Earth style application. You can, for example, open a KML file with Google Earth to display its contents.

For more information on KML, see the Google documentation at http://code.google.com/apis/kml/documentation/ or the OGC specification at http://www.opengeospatial.org/standards/kml.

I’m not currently aware of any specific Ruby support for KML.

Where to go from here

This article has covered just a few of the most common and/or promising major spatial data formats. There are a number of others currently in use, including many locale or application-specific forms. But as you can see, Ruby support for even the major formats is currently rather thin. We still have much work to do on our tools.

As the principal author of RGeo, I’m looking for help in this area. I released the rgeo-geojson and rgeo-shapefile gems based on work I’ve done to integrate my own applications with those formats. However, I haven’t yet had the need to actually use one of the XML formats, and as a result I haven’t written any tools to help with them. There is currently quite a bit of room to contribute to the community in this area.

Next week I’m going to take a break for the holidays, but I expect to release the next planned article on scaling spatial applications with the new year. Stay tuned, and let’s bring Rails down to earth!

In this article, we’ll take a first look at coordinate systems and geographic projections. We will:

• Examine the importance and effect of coordinate system differences
• Survey the various coordinate systems used for geospatial data
• Become familiar with coordinate system representations and SRIDs
• Specify coordinate systems in RGeo factories
• Use RGeo to convert data between coordinate systems
• Learn how to handle coordinate systems in Rails

This is part 4 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.

Why coordinate systems are important: a cautionary fable

Once upon a time, a plane took off from San Francisco, California on a routine flight to Athens, Greece. The captain, being both valiant and diligent with his passengers’ safety, sent a Twitter message to air traffic control. He wanted to ensure that his flight route would not be disrupted, for he had read on Hacker News that the diabolical EvilVolcano in Iceland was erupting, sending a deadly ash cloud high into the air lanes.

Meanwhile, an air traffic technician, having just finished installing a brand new Rails-based flight planning application, received the tweet. “@air_traffic_control pls chk flt path SFO-ATH. Far enuf fr #EvilVolcano?”

Excited to use his new tool, the technician got busy with his analysis. He looked up the latitude-longitude coordinates of San Francisco and Athens, and plotted a straight line between them, using Google Maps to verify that his path looked correct. Then he looked up the latitude-longitude coordinates of the diabolical EvilVolcano, and calculated the distance between it and the plane’s expected path.

“Lo!” he exclaimed. “Surely the flight path shall follow a straight line between two points, right along the 38-degree latitude line. But look– Iceland is nowhere near that path. The safety of our valiant pilot is assured!”

The flight path as seen by the air traffic technician

The technician proudly tweeted back his findings: “@valiant_pilot Flt path safe dist fr #EvilVolcano. Have pleasant journey.”

Having received this response on Twitter, the pilot proceeded to take off on his planned route. The last tweet received from the ill-fated flight, some seven hours later, read as follows: “Oceanic Air flt 815 encountering ash #SmokeMonster from #EvilVolcano. #Mayday #Mayday“. And the rest is history.

What went wrong?

For the most part, the air traffic technician did the right thing. He looked up the latitude-longitude coordinates of the departure and arrival cities, drew a straight line path between them, and then computed the distance bewteen that path and the EvilVolcano in Iceland. That is, he computed:

Distance(LineString(-122.4 37.8, 23.7 37.9), Point(-19.6 63.6))

However, he missed one thing. The shape of a “straight” line drawn between two points, and thus the distance calculated, may be vastly different depending on what coordinate system you are using, even though the latitude and longitude coordinates are the same!

Google Maps uses a Mercator Projection to display a world map. In this flat coordinate system, a straight line between San Francisco and Athens follows the 38-degree latitude line, passing through muggy Virginia and sunny southern Spain on its way to balmy (albeit bankrupt) Athens.

But the earth itself is not flat. Using a spherical coordinate system, the straight line shortest path between the two cities across the surface of the globe passes further north, directly over chilly and volcano-infested Iceland. This is the actual flight path of the ill-fated valiant pilot.

The actual shortest straight-line path from San Francisco to Athens

Simply by using the wrong coordinate system, the air traffic technician got a grossly innacurate answer, resulting in dozens of innocent passengers becoming instant prime-time celebrities.

Coordinate systems for geo data

At a basic level, a coordinate system can be thought of as providing “meaning” to a set of coordinates. When you see the location “Point(-19.6 63.6)“, how do you interpret those numbers? They could be the latitude and longitude of the Iceland volcano, but they could equally be measurements in feet from your front door, or light years from Alpha Centauri. The coordinate system is what differentiates these cases.

Location applications generally work with coordinate systems related to the earth’s surface, and these coordinate systems fall into three types.

Geocentric coordinate systems are three-dimensional coordinate systems with the origin located at the earth’s center. You won’t generally see much data in a geocentric coordinate system, but it is sometimes a convenient coordinate system to use for computational geometry and analysis algorithms.

Geocentric coordinates measure X, Y, and Z distances from the center of the earth. (Credit: http://kartoweb.itc.nl/geometrics/Coordinate%20systems/coordsys.html)

Geographic coordinate systems are the familiar latitude-longitude systems identifying points on the earth’s surface in terms of degrees. The most common geographic coordinate system, the one used by GPS and expected by most mapping applications, is known as “WGS 84“.

Geographic coordinates measure our familiar latitude and longitude. (Credit: http://kartoweb.itc.nl/geometrics/Coordinate%20systems/coordsys.html)

Projected coordinate systems involve taking a portion of the earth’s surface and “flattening” it. These coordinate systems are planar and generally Cartesian for easy display and computation; however, they introduce various kinds and amounts of distortion. Whenever you see a map displayed on a piece of paper, a computer screen, or other flat medium (basically anything other than a globe), you are looking at a projection.

There are hundreds of different projections in use, from common projections used for world maps, to special-purpose projections used for specific regions. When you view a Google Map, you are looking at a Mercator Projection. This is a projection designed to preserve shapes and straight-line directions, at the expense of distorting sizes and distances away from the Equator. A Google Map, for instance, implies that Greenland is larger than Africa, when in fact it is much, much smaller.

Google Maps uses a Mercator Projection

The United Nations logo includes a polar projection putting the north pole at the center. In this projection the pole is the least distorted region of the world, and everything else revolves around it, symbolizing the ideal of a world community privileging no particular country (except possibly the northern hemisphere, but we’ll ignore the politics for our purposes…)

The United Nations logo includes a north polar projection

Finally, whenever you look at a folding map– whether a street map for a city, a state map, or any other map that shows a limited area– you are usually looking at a local projection, one specifically tailored to that particular limited area. Such projections define a particular area in which they make sense. Objects inside that boundary can usually be displayed with minimal distortion, while objects outside that boundary often cannot be described at all.

Representing and specifying coordinate systems

Whenever you receive location data– whether from geocoding, user input, an external database, or any other source– the data should come with a coordinate system. Currently, there are two common ways a coordinate system can be defined:

Proj4 Syntax. One common way to specify a coordinate system is through a syntax defined by the Proj library. We won’t go into detail on the syntax here, but it is intended to describe how to convert data to and from that coordinate system. That is, if you receive data in a projected coordinate system, if you have the Proj4 syntax for that projection, the Proj library can convert the data back into latitude and longitude, and vice versa. Because of this useful property, Proj4 syntax is quite ubiquitous. Below is an example of the Proj4 syntax for “NAD83 / Washington North”, a local projection commonly used for topgraphic mapping in northern Washington state. For now, don’t worry if you don’t understand every field. This is just an example so that you can recognize Proj4 syntax when you see it.

+proj=lcc +lat_1=48.73333333333333 +lat_2=47.5 +lat_0=47
  +lon_0=-120.8333333333333 +x_0=500000.0001016001 +y_0=0
  +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 +no_defs

OGC well-known-text (or WKT). The Open Geospatial Consortium, the standards body that developed the Simple Feature Access specification described in part 3, also developed a syntax for representing coordinate systems and transformations. Below is the well-known-text for “NAD83 / Washington North”. Again, for now you don’t need to understand every field here, but you should be able to recognize WKT format when you see it.

PROJCS["NAD83 / Washington North (ftUS)",
  GEOGCS["NAD83",
    DATUM["North_American_Datum_1983",
      SPHEROID["GRS 1980",6378137,298.257222101,
        AUTHORITY["EPSG","7019"]],
      AUTHORITY["EPSG","6269"]],
    PRIMEM["Greenwich",0,
      AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.01745329251994328,
      AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4269"]],
  UNIT["US survey foot",0.3048006096012192,
    AUTHORITY["EPSG","9003"]],
  PROJECTION["Lambert_Conformal_Conic_2SP"],
  PARAMETER["standard_parallel_1",48.73333333333333],
  PARAMETER["standard_parallel_2",47.5],
  PARAMETER["latitude_of_origin",47],
  PARAMETER["central_meridian",-120.8333333333333],
  PARAMETER["false_easting",1640416.667],
  PARAMETER["false_northing",0],
  AUTHORITY["EPSG","2285"],
  AXIS["X",EAST],
  AXIS["Y",NORTH]]

As we saw above, each piece of geometry needs a corresponding coordinate system in order to specify the meaning of its coordinates and thus how to handle its data. Instead of attaching an entire Proj4 or WKT formatted string to every latitude-longitude point in a system, most geospatial systems provide a database of coordinate systems, each identified by an ID known as the Spatial Reference ID (or SRID). Each geometric data object then includes an SRID field referencing an entry in that database.

Technically, a geospatial system can provide its own spatial reference database and set its own SRIDs. However, in practice, many systems use a de facto standard dataset known as the EPSG dataset. This is a database of several thousand coordinate systems managed by the International Association of Oil & Gas Producers. The EPSG dataset is ubiquitous enough that most spatial database tools include a copy of it. A spatially-enabled PostGIS database, for example, automatically includes a table called spatial_ref_sys that is typically prepopulated with the EPSG dataset. You can look up SRIDs and get the coordinate system name, the WKT representation, and sometimes the Proj4 representation. The “NAD83 / Washington North” coordinate system example above has SRID 2285 in the EPSG database.

One important EPSG-specified SRID that you will encounter often is 4326. 4326 refers to the “WGS 84″ geographic (latitude-longitude) coordinate system we mentioned earlier– the coordinate system used by Global Positioning System (GPS). Typically, when you get a latitude-longitude coordinate from a GPS system, from a geocoder, from a Google map input, or most other common sources, it will implicitly be in the EPSG 4326 coordinate system. This is so universally true that PostGIS currently mandates that the SRID must be set to 4326 when you create a column of type “geography” (that is, a latitude-longitude column), as we saw in part 2.

You can browse the EPSG database at http://www.spatialreference.org/.

Coordinate systems for RGeo factories

In part 3, we discussed how RGeo manages coordinate systems through factories. Now that we understand coordinate systems in more detail, we can take a closer look at how to handle coordinate systems in Ruby.

RGeo supports two main types of factories: Cartesian factories and geographic factories. Cartesian factories are ideal for handling projected coordinate systems, in which the domain is flat. Geographic factories are useful for geographic coordinate systems representing latitude and longitude, in which the domain is curved like the surface of the earth.

When you create an RGeo factory, you may specify the exact coordinate system by passing arguments to the factory constructor. In most cases, you should provide an SRID using the :srid parameter. You may also provide Proj4 and/or WKT representations for the coordinate system using the :proj4 and :coord_sys parmeters, respectively. For example, here’s how you could create a factory designed to handle the “NAD83 / Washington North” projection we discussed earlier.

north_wa_proj4 = '+proj=lcc +lat_1=48.73333333333333 +lat_2=47.5 ' +
  '+lat_0=47 +lon_0=-120.8333333333333 +x_0=500000.0001016001 ' +
  '+y_0=0 +ellps=GRS80 +datum=NAD83 +to_meter=0.3048006096012192 ' +
  '+no_defs'
north_wa_wkt = <<WKT
  PROJCS["NAD83 / Washington North (ftUS)",
    GEOGCS["NAD83",
      DATUM["North_American_Datum_1983",
        SPHEROID["GRS 1980",6378137,298.257222101,
          AUTHORITY["EPSG","7019"]],
        AUTHORITY["EPSG","6269"]],
      PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
      UNIT["degree",0.01745329251994328,
        AUTHORITY["EPSG","9122"]],
      AUTHORITY["EPSG","4269"]],
    UNIT["US survey foot",0.3048006096012192,
      AUTHORITY["EPSG","9003"]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["standard_parallel_1",48.73333333333333],
    PARAMETER["standard_parallel_2",47.5],
    PARAMETER["latitude_of_origin",47],
    PARAMETER["central_meridian",-120.8333333333333],
    PARAMETER["false_easting",1640416.667],
    PARAMETER["false_northing",0],
    AUTHORITY["EPSG","2285"],
    AXIS["X",EAST],
    AXIS["Y",NORTH]]
WKT
north_wa_factory = RGeo::Cartesian.factory(:srid => 2285,
  :proj4 => north_wa_proj4, :coord_sys => north_wa_wkt)

Notice that, since the coordinate system is a projection, we’re using a Cartesian factory that will perform computations in a flat domain.

When you work with a projected coordinate system like this one, the coordinates themselves are expressed in the projection rather than in latitude and longitude. For example, the location of the Space Needle in Seattle, which is at latitude 47.620578, longitude -122.34978, is expressed as (1266457.58, 230052.50) in this coordinate system.

space_needle = north_wa_factory.point(1266457.58, 230052.50)

Let’s consider another example. If you want to work with latitudes and longitudes, say from a GPS system, then you should use the “WGS 84″ coordinate system, which has SRID 4326. Looking up this coordinate system in spatialreference.org, we can find its Proj4 and WKT forms:

wgs84_proj4 = '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs'
wgs84_wkt = <<WKT
  GEOGCS["WGS 84",
    DATUM["WGS_1984",
      SPHEROID["WGS 84",6378137,298.257223563,
        AUTHORITY["EPSG","7030"]],
      AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
      AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.01745329251994328,
      AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
WKT

To create a factory for this coordinate system, we should use one of the geographic factories provided by RGeo. These factories perform computations over a curved earth rather than a flat earth. For example, they will correctly draw the line between San Francisco and Athens so that it passes through Iceland. You can use RGeo::Geographic.spherical_factory to create a factory that performs computations on a spherical earth:

wgs84_factory = RGeo::Geographic.spherical_factory(:srid => 4326,
  :proj4 => wgs84_proj4, :coord_sys => wgs84_wkt)

We now have a factory that will correctly manage features using latitude/longitude.

space_needle = wgs84_factory.point(-122.34978, 47.620578)

Technically, the earth is not a perfect sphere, but is slightly flattened. The WGS84 coordinate system actually uses a flattened ellipsoid to more closely model this shape. However, RGeo does not yet support ellipsoidal computations, so we used a spherical factory instead as an approximation. In many cases, this will be good enough, but if you need accuracy down to the millimeter, this is something to be aware of.

Converting data between coordinate systems

The above examples may have sparked an important question. In the examples in parts 2 and 3, we created factories without the benefit of the proj4 and wkt strings, and in some cases we also omitted the SRID. Under what circumstances are proj4, wkt, and/or SRID needed when you create an RGeo factory, and under what circumstances can you leave them out?

SRID is generally needed when you want to store your data in a PostGIS database. This is because PostGIS generally puts an SRID constraint on spatial columns that you create, in order to make sure you don’t mismatch coordinate systems. So, if you are storing data in or pulling data from PostGIS, you should always specify the SRID in the factory.

The proj4 string has a different purpose: it is used by the Proj library to describe how to convert coordinates between coordinate systems. Here is how this works.

Suppose you are reading a set of points from a data source that uses the “NAD83 / Washington North” coordinate system, and you wanted to convert them to latitude and longitude. To perform this conversion, you need two factories, one in the source coordinate system and one in the destination coordinate system. Both factories need to have their :proj4 string set. This lets the Proj library understand both coordinate systems so it can figure out how to convert from one to the other.

In the above examples when we created our north_wa_factory and wgs84_factory, we provided the needed proj4 strings. So these factories are ready to be used.

Now, to actually convert the data, load it using the source factory, and then use RGeo’s “cast” mechanism to cast it to the other factory. Call RGeo::Feature.cast, pass it the original object and the destination factory, and set the :project argument to true to tell it to transform coordinates.

space_needle = north_wa_factory.point(1266457.58, 230052.50)
space_needle_latlon = RGeo::Feature.cast(space_needle,
  :factory => wgs84_factory, :project => true)

This code tells RGeo to take the space_needle object (which is in the projected coordinate system) and convert it to the WGS84 factory, while transforming (projecting) its coordinates so they are correct in the new factory’s coordinate system. As a result, space_needle_latlon is created, containing latitude and longitude coordinates, and with its factory set to wgs84_factory.

What about the WKT string? Currently, RGeo does not have any functional use for the WKT string; it is just an informational field. So you can omit it if you want. However, it turns out that in many cases you can theoretically use the WKT to transform coordinates in the same way as the Proj4 string. RGeo will likely provide this capability in the future– you will be able to pass WKT instead of Proj4 to allow factories to transform coordinates.

Using Coordinate Systems in Rails

In part 2, we looked at an example in which we created a PostGIS column that stored point data in geographic (latitude-longitude) coordinates. We did this by installing the activerecord-postgis-adapter, which extends ActiveRecord migrations to create spatial columns, and which exposes spatial attributes as RGeo data objects.

Now let’s consider another example. Suppose we obtained a set of polygons (say, zip code boundaries) from a data source that uses the “NAD83 / Washington North” projection. We could convert the polygons to latitude-longitude, but remember that doing so can actually change the shape of the polygon. So let’s suppose we decided this was unacceptable, and we need to store the polygons in the database in the projected coordinate system. Here’s the migration for this case:

class CreateZipCodes < ActiveRecord::Migration
  def change
    create_table :zip_codes do |t|
      t.integer :zip
      t.polygon :boundary, :srid => 2285
    end
  end
end

The name of our column is “boundary“, and we set its type to “polygon“. Now, in the example in part 2, we had set the :geographic property of the column, indicating that it was storing latitude and longitude and that it should use the PostGIS features designed for that case. In this new example, we are storing projected coordinates, so we do not set :geographic. Instead, we just set the SRID to match the coordinate system that we are using. Setting a SRID on the database column actually sets up a database constraint: it allows only geometries with a matching SRID to be stored in the column. This is PostGIS’s way of helping you avoid the mistake of our unfortunate air traffic control technician: that of mismatching our coordinate systems.

As we did in part 2, we also need to set the factory for this column so that Rails knows what factory to use when it reads geometries from the database.

class ZipCode < ActiveRecord::Base
  north_wa_factory = ... # use the factory we created earlier
  set_rgeo_factory_for_column(:boundary, north_wa_factory)
end

In general, it is important that the factory you set in your ActiveRecord class matches the constraints in the database column, so that both sides handle the data in the same way. In particular, the SRIDs should match, and either both should be Cartesian or both should be geographic.

Now we can read and write the polygon data. We just need to remember that the data is not in latitude-longitude, but in the projected coordinate system. So when we write polygons to the database and read from the database, we will receive projected coordinates.

This example, of course, brings up an important question. We need to decide up front whether the database should contain projected data or latitude-longitude data. How do we choose? This can be a somewhat complicated question, and I will dive more deeply into the pros and cons of different strategies in a later article. However, for now we know enough to understand a few of the issues.

In many simple cases– if you are working only with points, or with small features that do not cover a large part of the globe, or if extreme accuracy is not important– you may find it easiest to think in latitude and longitude. In those cases, you can just create a :geographic column in the database and convert everything to a geographic coordinate system. Just be aware that there are potential issues whenever you have to convert data from one coordinate system to another. As we saw in our story, lines and polygons that span a large area can change their shape dramatically when switching coordinate systems. So if accuracy is essential, it may be desirable for your database to use the same coordinate system as your data source. You also should consider what type of spatial queries you are likely to run against your data. Remember that coordinates in a query must match the SRID and coordinate system of the data in your database.

Where to go from here

Congratulations on making it through this article! Understanding coordinate systems can be tricky, but it is very necessary for doing nontrivial applications. In this discussion, I’ve deliberately left out a number of more advanced topics that I’ll probably cover in a later article. But you should now have enough information so you won’t get lost when people start talking about projections and SRIDs.

If you’d like to explore more about map projections and geospatial coordinate systems, the articles on Wikipedia are not too bad. I’m not aware of any good books on this material geared towards web developers, but if you know of any, please send me a line. For reference, you’ll probably find yourself going to spatialreference.org frequently when you need information on the coordinate system referenced by a particular SRID.

For the next article, I’m currently planning on covering common file and data interchange formats used for geospatial data. Stay tuned, and let’s bring Rails down to earth!

This is part 4 of my continuing series of articles on geospatial programming in Ruby and Rails. For a list of the other installments, please visit http://www.daniel-azuma.com/blog/archives/category/tech/georails.