Test Driven Evaluation
To better explain what Test Drive Evaluation (TDE) is let’s first look at what the principle is based upon…Test Drive Development (TDD)
Source: Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or 'rediscovered' the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.[1] Test-driven development is related to the test-first programming concepts of extreme programming, begun in 1999,[2] but more recently has created more general interest in its own right.[3] Programmers also apply the concept to improving and debugging legacy code developed with older techniques.[4]
Emphasis mine
Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or ‘rediscovered’ the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.
With the growth, popularity and increased acquisitions of BaaS providers it becomes harder and harder for newcomers to properly evaluate which one will provide the best value added to their application. Please note that I didn’t specify mobile application. While I personally subscribe to the “mobile first” approach…native desktop applications and traditional web applications shouldn’t/can’t be seen as second class citizens, thus BaaS should provide support for those as well.
Myself and many others have written on the pro’s / con’s of the top BaaS players. Unforunately, there hasn’t been a set of standard tests that could be run and give a definitive “pass or fail” result for the features that the providers offered. The KitchenSync project aims to provide the tests for these features.
Everything and the KitchenSync
It doesn’t matter if you’re a developer, project manager or analyst there is never enough time to implement every feature, RTFMs or write POC’s (when evaluating different vendors, technology, etc) for an application. We tend to cut corners, take for face value what is on the brochure or go with a solution just because it was featured as the latest/greatest startup on Techcruch.
For example:
- What does it take implement object level security?
- Does the BaaS provide static file / webhosting?
- Which providers will allow full text searching?
- How many API calls to it take perform request abc
What is the Best Option?
Antonio Martinez does a great job of summarizing:
The most important factor here is to evaluate each service’s features and their usefulness to you — and your app! Preconfigured objects might seem attractive, unless your idea of what those objects should be differs from those of the service provider.
Services that offer just “users” along with general purpose objects might initially seem more difficult, until you see that substantial flexibility comes with the user-based approach.
Will you need a support agreement; and should you be paying for phone support? How accessible are the engineers if you need them, and what kind of user community support can you expect?
Take a close look at the documentation. This is one time you should read it thoroughly, even though the sample projects have given you enough to get started! :] The quality, coverage and currency of the documentation will give you a good idea of how committed each company is to your success.
Make yourself aware of the costs you will incur in using the service. This will require you to project your future usage for best case and worst case scenarios. Should you find yourself with 100 million users, will your app generate enough income to pay for the level of service you will require?
If your app is going to require a paid plan from the start but it flops, can you afford that? Some services require you to contact sales before you know what these costs will be; don’t skip this step!
Doing a search for the name of your service on discussion boards might expose interesting bits of information you might not have found otherwise. If you get no hits at all, how popular is the service — and what are the chances it will be around in six months? If all the hits say “it’s awesome” that might be a good indication – but pay attention to the negative comments as well!
You can arrive to most of the answers just by reading the documenation, but where the 1’s and 0’s meet the compiler is at implementation. Enter the KitchenSync project.
Even with my own evaluations and recommendations I wanted to give developers, analysts and just the curious, a set of common unit tests to run against each of the providers so that informed decisions could be made with the utmost transparency, while providiing a forum for feedback, additions and bug fixes.
All the tests are open for anyone to use for your own evaluations - just download and run them. In addition, as mentioned above, anyone is welcome to join the party - all enhancements, corrections and suggestions are welcome. There are a number of additional vendors as well - and I invite them, or any developers familiar with them to contribute implementations for them as well and I will be happy to add them to the report as time permits or I can get to them.
You Forgot About Feature ‘X’ and Provider ‘CDW’
No, I didn’t. There had to be a version 1.0 so I chose the providers that I’ve used (Parse, Stackmob, Kinvey and FatFractal) and a basic set of tests. I will be adding more tests just as quickly as I can write them, but I’m hoping for as many pull-requests as I can get. Specifically for new providers.
Disclaimer
I am not employed by any of the providers nor paid, bribed or coerced into writing this article or performing the tests nor is this an endorsement of my employer. I wrote it because…well…no one else had.