Migrating Data to FatFractal with Node.js

One of the long term goals that I have for an app I’m working on is to move the backend over to FatFractal. This was not a consideration that I made lightly. Moving from one backend system, no matter if it is homegrown or using BaaS, is a daunting task. Schemas, permissions, number of records, order of operations, migration scripts, etc. Is the juice worth the squeeze? In my case it is. Even after careful evaluation of the pros and cons of staying on the current provider or moving to FatFractal I threw together a proof of concept with sample data. Because the data structure paradigms are so different between the two platforms I wanted to see what the migration effort would be. How would I accommodate for exceptions in the model? What I mean by this is that even if the PoC was successful I knew that we wouldn’t do the migration for several months and that the current provider’s schema would be changed and I didn’t want to have to go back and make large scale changes to my import scripts. We are talking about >50,000 users and ~300,000 object records including static files.

Most of the BaaS providers that I’ve used allow for dynamic schema creation, but generally in the context origination not in migration. Parse does allow for [file importation of data]((http://blog.parse.com/2012/05/22/import-your-csv-data-to-parse/) . Stackmob has pretty cool python script for importing data directly from Parse, but the limitations are important to take note, especially the level of subobjects and large datasets.

Though FatFractal doesn’t have an official migration tool to/from a specific provider, setting up a migration is extremely easy using node.js and their server side js lib. An added benefit was being able to test everything locally, including a production migration run before I ran it against cloud migration.

Mystique is the sample project which has two components. First is the actual FatFractal application and second is the importer. Inside of the Importer directory there are two json files which would represent your data. Obviously you have a lot more when doing a migration of any significant size, but in this sample project I wanted to highlight the functionality that makes migration with FatFractal so easy. The big “aha!” in making all this work is in FatFractal’s Definition Language (FFDL) which allows for objects, collections and properties to be automatically determined at runtime. I don’t have to explicitly define any objects, properties or collections in the FFDL.

Post-migration you can explicitly add object definitions in the FFDL for what has been auto-generated at runtime during the import process… but it’s not required. The app will work perfectly well without your intervention.

You can find the project on GitHub.