BaseSpace Native App Engine Overview

The BaseSpace Native App Engine offers the simplest way to get your computation-focused application into the hands of BaseSpace users. Built on top of our existing open RESTful API and best of breed open source technology, a Native App can be written in any language and simply needs to run within a Linux environment. Contrary to BaseSpace Web Apps, your Native App has no need to make API calls or to integrate with one of our BaseSpace SDKs as all these mechanics are handled for you. This simplified model allows you as an app developer to focus on the core strengths of your app without being bogged down with infrastructural complexities, security, data movement, and API integration.

How do Native Apps work?

The following diagram is an overview of the Native Apps workflow:

  1. Formbuilder: User configures the input form that the developer put together for the app in BaseSpace then clicks Continue
  2. BaseSpace starts a new Amazon EC2 Instance or multiple Instances and sends the data from the input form to the EC2 Instance(s)
  3. Docker Image: The EC2 Instance is loaded with the input data and the Docker image for the app, then the Docker image is run with the inputs mounted to the running Docker container
  4. Once the app is finished processing (gives exit code 0), SpaceDock will automatically upload files in the designated output folder back to BaseSpace
  5. Reportbuilder: Users can view reports created by the developer in BaseSpace, these reports have access to the underlying output data

Benefits for Users

  • Users start Native Apps the same way they're used to with Web Apps. A Native App may be kicked off from the App Store, or directly from the data they want to have analyzed.

  • Time to results may be shortened as it's now easy for app devs to split up the analysis into concurrently running nodes.

  • Running Native Apps may be monitored in one place. Real time performance metrics and visibility into progress may be made available to the user. Users may see high level summary of multi-node applications and zoom in to get details of a specific node.

Benefits for App Devs

  • No need to manage your own web server or hosting infrastructure. A Native App is kicked off and its runtime is entirely managed by BaseSpace from beginning to end.

  • Native Apps can be written in any language (or combinations thereof) as long as it can run on Linux. We'll simply kick it off with the command line you give us.

  • The need for interacting with the BaseSpace API is eliminated in most cases as authentication, file transfer, metadata availability are handled for you. Input data is automatically downloaded before your app starts and results are uploaded back in accordance to a few simple conventions. Your app may access the BaseSpace API should there still be a need.

  • Gathering user input is simple and powerful through the use of customizable input form which is maintained and tested independently of your core application. Scripting within the form allows custom validation and input parameter construction.

  • A templating engine is used to create rich scriptable reports for application output that also is maintained and tested independently of your core application. Improvements to the reports are applied to past and future runs of the application.

  • Getting started is easy with our downloadable VM allowing you to build and test your application locally using real data and input from BaseSpace.

  • All application dependencies and environment details are preserved when moving your application to BaseSpace through the use of a Docker Image.

Native vs Web Apps

The Native App engine does not replace our support for Web Apps (or desktop apps) as there are distinct benefits for both. Where Web Apps allow for a fully customizable and interactive user experience, Native Apps are designed for your non-interactive computation of BaseSpace data with the results being uploaded and presented back to BaseSpace users.

  • Web app may deliver rich UIs in which a user interacts with their data. Native Apps gather input data and settings before starting with users being able to monitor progress once an app is running. Once an app is complete a report is generated to best make sense of the results.

  • The scriptable input form and reporting support of Native Apps should be more than sufficient for most purposes. If it doesn't yet meet your needs then please let us know. A Web App that you build and manage yourself will always have more flexibility.

  • The Native App Engine is best suited for self-contained applications. If your app needs to use external APIs, data sources, etc then a Web Application that you host yourself may be more suitable. Native Apps are sandboxed and isolated ensuring predictable reproducible results with user data safeguarded.

Three Components of a Native Application

  1. Create the Input Form
  2. Create your Docker Image
  3. Create your Report using HTML5 and Liquid

Create the Input Form

Your app may start from a sample, a VCF file, the full output of another app, or whatever data exists in BaseSpace. It may also have a suite of options that need to be exposed as well. Our Formbuilder tool allows you to dynamically create a form that satisfies all of your app’s requirements. We have a full suite of configurable standard controls like textboxes, checkbox, dropdown, etc, and BaseSpace-specfic controls too. These can be put together in any way you want with a powerful rule engine to help validate all of your business rules. Below is a screenshot of this tool in action.

In the BaseSpace Developer Portal, once you have created a new application, you will see the Formbuilder tab available. You can use this to create an input form for your application.

In addition, the Formbuilder is also used for local testing and debugging of a Native Application using the Send to Local Agent button.

For more information please refer to the Formbuilder documentation.

Create your Docker Image

Docker is an open source Linux container engine that allows you to create portable application images. These images will run exactly the same on any hardware or virtualized hardware that Docker itself can run on. This means your app is completely portable to environments like Amazon EC2, Rackspace, OpenStack, or just a local Linux machine.

To learn more about Docker, visit the link above or try out the Interactive Tutorial!

In the Formbuilder tool you can specify your command line with options from the form that you built.

function launchSpec(dataProvider){
    var projectId = dataProvider.GetProperty("Input.project-id").Id;
    return [{
        commandLine: ["python", "/home/apps/fastqc/fastqc.py", projectId],
        containerImageId: "basespace/fastqc_v1"
    }];
 }

Docker images are similar to Virtual Machines but have a few key differences. Virtual Machines are run on the host machine using Hypervisor, each Virtual Machine has an operating system installed on it as well which results in the size of the Virtual Machine growing to more than 10 GB in some cases! Instead, Docker introduced a new engine called the Docker engine which is installed on the host operating system and manages all docker applications. A great comparison of Docker development to Virtual Machine development can be found here.

We have extensive documentation available to help you Set Up your Local Development Environment and to Manage Your Docker Image. In addition, we also have documentation to help you Test and Debug your Native App.

To understand how to build your docker image for BaseSpace, please refer to the Native App Conventions Documentation. This documentation provides more insight into the conventions that have to be followed within a Native app to interact with BaseSpace appropriately.

Create your Report using HTML5 and Liquid

Now that you’ve got your app running, you can create a nice set of reports for users so they don’t just see a list of files after running your app. We built a report builder tool that gives you the full power of HTML5 and ultimate flexibility to create a rich user experience. Accessing data from your analysis is done via the Liquid templating language (from Shopify) and a set of extensions that we’ve added to it.

For more information please refer to the Reportbuilder documentation.

Example Native Applications

For developers or users who would just like to understand more about Native Apps, we have put together a quick guide that will take you through many of the processes that a developer would have to do in order to create a basic Native Application.

The first tutorial is a beginner-friendly tutorial where the developer creates the FastQC application. The FastQC application is an open-source project that we have adapted as a Native App in BaseSpace. In the first guide, you will be able to send a job from BaseSpace to your machine, watch the FastQC App execute locally, and view the output data that was uploaded to BaseSpace.

The second tutorial is an advanced tutorial where the developer creates a new app from scratch using an example python script. This tutorial is very involved and requires coding. You will have created a new docker image using example code, created a new form, created a connection between BaseSpace and the app, and finally to create an output report for the results that are uploaded.

Here is a link to the Guides: Example Native Applications

Native App Conventions

The Native App Infrastructure in BaseSpace provides features for the developer that streamline processes such as gaining permission to the user's data, downloading input data to the machine where the app is running, reporting errors/logs to the user, and uploading results back to BaseSpace. In order to use these features, the developer must follow some basic conventions while creating their Native app. For more information about this information, please refer to the Native App Conventions Documentation.

A developer may choose to implement these features a little differently, in which case the Rest API for BaseSpace can be used from within a Native app to have access to this information as well. This would be for developers who are interested in more robust solutions for upload/download or need additional functionality that the native app engine does not provide. For more information about using the Rest API, please refer to the BaseSpace Rest API Reference.

Summary

This is really just scratching the surface of what is possible with the new Native App Engine. We look forward to your feedback and are really excited about what’s to come!

For extensive documentation, here are some links: