Export SimpleGeo Layers with Python

Urban Airship has decided to shut down the SimpleGeo services that it acquired by March 31st, 2012. I personally see this as step-backwards for the services that they offer and to the SimpleGeo customer’s who rely on the data.

* stiQRd (I am lead developer) uses it for all of our location information.

What was most shocking to me after their announcement wasn’t the cut-off date or the fact that Urban Airship had decided to sunset the service but the fact that they “punted” the solutions for helping existing customer’s migrate their data.  Granted they did offer semi-replacements for Storage, but it basically amounts to you rolling your own solution.  That is why we were using SimpleGeo in the first place...because we didn’t want to roll our own because of limited time and resources.  

SimpleGeo was so easy to use and the company had great support. It was a no brainer.

In order to prepare and update the stiQRd app before the sunset date I need to get an export of my data.  Unfortunately, there isn’t an easy way to export layer data programmaticly or through the console.  I posted a question to the Google Group and got some half-ass response that didn’t help whatsoever. Not to be derailed I was hacking together a few solutions, none of which I really liked.  I then came across this gem (I know...it is bad) of a Ruby script that will iterate through all of your layers and export that data to csv. Score!

The tips to retrieving all your data from a particular layer are two fold. SimpleGeo’s API doesn’t have a “get_all_records(layer_name)” type of method, but they do have “get_nearyby”, which at face value doesn’t look like it will give you the necessary results, but it does when you use (0,0) as the lat/lon and add in the “bbox” (boundary box) parameter values of ‘-90,-180,90,180’.

Unfortunately, I don’t know Ruby very well and I am spending a lot of my free time sharpening my Python skills. Since there wasn’t an implementation for export in Python I decided to port the Ruby script.  With the help of Bob “I am a PEP freak” Waycott, we have a pretty elegant solution.

I am also happy to repost that Parse has taken it a few steps further and offered up the simplest/elegant solution for migrating this data over to their backend service.  If you are looking for a turnkey solution then head over their for their migration tool.

If ANYONE has suggestions or updates please let me know.

Python Migration Script: (written by Cory D. Wiles and Bob Waycott)

Fast String Concatenation

Besides collection/list iterations, string concatenation is probably the most common developer task. Doesn’t matter the language.  Unfortunately, this is done incorrectly, especially with large strings.  I myself am guilty of taking for granted what goes on under the hood when performing this operation. After reading this article I found myself wondering about the mutability of strings in Ruby. Strings are natively mutable so you don’t have as much performance hit when doing basic concatenation, unlike Python which treats strings as immutable, thus “+=” performs a copy-on-write. No efficient at all. For small string concats doing the standard: “+=” or “+” is still the shortest distance between two points, however manipulating a string of any significant size, what seems to be the most efficient across Python and Ruby is to add each separate string into an array and then join.

Ruby: ** Thanks to Travis Dunn for codev


Do to the fact that I am doing more and more large scale projects performance is more important than ever and paying close attention to these little details allow for performance boots very quickly.