Example Native Apps

In this section, the developer will learn how to create example Native apps for BaseSpace. There are two walkthroughs in this section:

  • (Basic) The FastQC Native App Example - this example will take you through an overview of the different components of a Native app with a pre-existing docker image. By the end, you will have created the published FastQC application in BaseSpace in your account with the instructions that follow. We recommend this tutorial for anyone that is new to Native apps and Docker, or would just like to see what all the fuss is about!
  • (Advanced) The Sample Python Native App Example - this example will take you through an in-depth tutorial where you will create a Docker image from scratch. This tutorial is recommended for anyone that wants to learn in detail about Docker and how to develop their own Native apps within BaseSpace.

Basic - Create the FastQC Native App

The following guide will take the developer through a very simple example of creating and testing the FastQC Native App.

Set up your local Development Environment for FastQC

Please refer to the Set Up The Native Dev Environment to set up your local development environment to begin creating Native applications.

This guide assumes that the local development environment is already set up. In addition, the Native Apps Virtual Machine is running locally on your computer and you have already SSH'd into your Virtual Machine from a local terminal. The password is basespace to SSH into the Virtual Machine.

It is important to regularly update the SpaceDock package on your Virtual Machine. An outdated version of SpaceDock may cause issues while testing the app. For more information about updating SpaceDock, please refer to Native App Conventions Documentation.

Files Links for FastQC

The following files will be needed for the following walkthrough. You can either download these locally or simply copy and paste the contents of each file when it is needed.

FastQC App .txt files:

In addition, you will also have to import a dataset to your BaseSpace account, please follow this link for the Project Share in BaseSpace: https://basespace.illumina.com/s/LbRGgqcP0qTR

Login and Create a New App in the Developer Portal for FastQC

  1. In a browser window, go to BaseSpace: http://basespace.illumina.com

  2. If you already have a BaseSpace account, click Login, otherwise click Sign Up

  3. Once you are logged in, you will be asked to accept an Agreement, click Accept

  4. Now, you will be taken to the BaseSpace dashboard, find the Developers link in the footer and click it

  5. You are now on the developer portal, click on My Apps

  6. You may be asked to accept the Developer Agreement, click Accept

  7. Click on Create a new Application

  8. Enter App Information and ensure that Native is selected for App Type

  9. Click on Create Application

Import Example Data to Your BaseSpace Account

Navigate to https://basespace.illumina.com/s/LbRGgqcP0qTR, you should get a message that asks you to accept the share, click Accept.

You will now have the SmallSampleProject in your list of Projects in BaseSpace. This Project is small so that we can show a quick analysis of the FastQC app.

Create the Input Form for FastQC

What will be covered in the following section:

  • Create/Open a template form via the developer portal
  • Edit the form in the developer portal (the user-facing form that is shown before the appis executed)
  • Edit the callbacks.js script in Formbuilder (these are the arguments that will be sent to the command line of your Docker image)

To build the FastQC App's form, follow these simple steps:

  1. From the App Details page for your application, click on the Form Builder tab

  2. Click on Start building a custom form

  3. Select the Example Form on the Revisions tab

  4. Click Open

  5. Copy the contents of FastQC_Input_Form txt file

  6. In the left pane of the Formbuilder window, Select All of the code in that pane and delete it

  7. Paste the contents of the FastQC_Input_Form in the left pane of the Formbuilder window

  8. Click Save and Preview

Configure your Callbacks Script for FastQC

The callbacks.js file is important because it informs BaseSpace of the Docker container Id in the Docker registry, and it also includes command-line parameters that are passed to your application upon launch that are needed to run the application.

For the FastQC app, we will configure the proper callbacks.js script by doing the following:

  1. Click on the Current Template dropdown at the top and select callbacks.js

  2. Copy the contents of FastQC_Callbacks_js

  3. In the left pane of the Formbuilder window, Select All of the code in that pane and delete it

  4. Paste the contents of the FastQC_Callbacks_js in the left pane of the Formbuilder window

  5. Click Save and Preview

Run the FastQC App from BaseSpace and Send the Job to your Local Virtual Machine

This section assumes that you have:

  • Started the Native Apps Virtual Machine on your local computer
  • Used PuTTY or your Terminal to SSH into your local Virtual Machine (the password is basespace)
  • Have the FastQC app's form open in the Formbuilder window in your browser

What will be covered in the following section:

  • Select input data for the app in the Formbuilder window
  • Use the Send to Local Agent option in the Formbuilder window to send a FastQC analysis job to your local Native Apps Virtual Machine
  • Watch the job execute in your local terminal
  • Watch as the app uploads data back to BaseSpace and completes the AppSession with logging information
  • View the log and results of this analysis in BaseSpace

Now, lets use see this in action:

  1. In the Formbuilder window, under Developer Tools, find and copy the command after Sample command-line to start local agent:

  2. Keep this browser window open

  3. Paste the copied command from step 1 into your SSH terminal (e.g. sudo spacedock -a ... -m https://mission.basespace.illumina.com)

  4. Hit Enter, the terminal will show a series of commands executed. Once you see the message Successfully pinged docker service, you can move on to step 5. This just started the service on your Virtual Machine that will begin polling BaseSpace for work for your specific app.

  5. Now, switch over to your Formbuilder window where you have the FastQC app's form opened in the editor

  6. In the right pane of the Formbuilder window, click Select Sample

  7. Select SmallSampleProject and select truseql1_8899

  8. Click Ok, you have now select the input Sample for your application

  9. Click Select Project in the right pane of the Formbuilder window

  10. Click New... to create a new Project in your BaseSpace account for the output results (you can also select a pre-existing Project that you have write permission to)

  11. Enter Project information and click Create

  12. In the Developer Tools of the Formbuilder window, click Send to Local Agent to send this job from BaseSpace to your local Virtual Machine

  13. In your terminal or command prompt window on your local machine, you will see your Virtual Machine receive the job from BaseSpace and do the following:

    • Download the basespace/fastqc Docker container from the Docker registry (or update the one that already exists on the Virtual Machine)
    • Download the input data from BaseSpace
    • Run the command-line script from the form's callbacks.js to execute the basespace/fastqc
    • Periodically update the AppSession Status with the new Status and StatusSummary
    • Upload results back to BaseSpace
    • Mark the AppSession as Complete
  14. You can view the output log in the Formbuilder window, this will also be displayed for this Analysis in BaseSpace

Create and Activate the Output Report for FastQC

What will be covered in the following section:

  • Launch the Reportbuilder from the Developer Portal
  • Open the Example Report
  • Build a custom Report for your FastQC application
  • Save the Report
  • Activate the Report
  • View the Report in BaseSpace

Lets try it out:

  1. Navigate to the Developer Portal

  2. Click on My Apps

  3. Click on the FastQC application that you created in an earlier step

  4. Click on Reports Builder

  5. Click on Start building a custom report

  6. Click Ok on the popup message, this indicates that you will need to have a Completed AppSession with output results in order to proceed

  7. Select the available AppSession from the list and click Ok

  8. Select Example Report

  9. Click Open

  10. Select Options

  11. Uncheck the Summary checkbox

  12. Click Save

  13. Click on the dropdown menu labelled Current Template and select result

  14. Select all of the code in the left pane of the Reports Builder and delete it

  15. Copy the contents of FastQC_Output_Report and paste them into the left pane of the Reports Builder window

  16. Click Save and Preview

  17. Click on the Back Arrow Button

  18. Select your Example Report and click Activate

View the Report in BaseSpace for FastQC

Now, we will navigate to BaseSpace and view our output result as a general user would.

  1. Navigate to the BaseSpace Dashboard
  2. Click on Projects in the header
  3. Select the Project that you created before running the application
  4. Click on the Analysis that you just ran
  5. View the Reports that you just created! You will see two reports for this app.

Summary for Basic Tutorial

We have just gone through the following:

  • Created a Form for the FastQC app
  • Ran the fastqc.py script in our local Virtual Machine using the input form from BaseSpace
  • Created a Report for the FastQC app
  • Viewed the Report in BaseSpace

This guide is a very high-level guide to creating a Native application, for more detailed information please refer to the following Native Apps Documentation:

Or please feel free to post any questions to our google groups!


Advanced - Create an Example Native App using Python or any other language

This example will take a developer through the process of creating a new Native app in BaseSpace with an example python script. The script could also be written in another language, but for this example we are using python.

In this tutorial, the developer will learn how to create a new Docker image

Set up your local Development Environment for Example App

Please refer to the Set Up The Native Dev Environment to set up your local development environment to begin creating Native applications.

This guide assumes that the local development environment is already set up. In addition, the Native Apps Virtual Machine is running locally on your computer and you have already SSH'd into your Virtual Machine from a local terminal. The password is basespace to SSH into the Virtual Machine.

It is important to regularly update the SpaceDock package on your Virtual Machine. An outdated version of SpaceDock may cause issues while testing the app. For more information about updating SpaceDock, please refer to Native App Conventions Documentation.

We are using the Public Docker Registry for this example. For Private app development, please use the Private Docker Registry that is hosted by BaseSpace. Documentation can be found here.

Files Links for Example App

The following files will be needed for the following walkthrough. You can either download these locally, use the links and download the files, or simply copy and paste the contents of each file when it is needed.

Example App files:

In addition, you will also have to import a dataset to your BaseSpace account, you can use any public dataset (choose a small one for fastest results) or please follow this link for the Project Share in BaseSpace: https://basespace.illumina.com/s/LbRGgqcP0qTR

Login and Create a New App in the Developer Portal for Example App

  1. In a browser window, go to BaseSpace: http://basespace.illumina.com

  2. If you already have a BaseSpace account, click Login, otherwise click Sign Up

  3. Once you are logged in, you will be asked to accept an Agreement, click Accept

  4. Now, you will be taken to the BaseSpace dashboard, find the Developers link in the footer and click it

  5. You are now on the developer portal, click on My Apps

  6. You may be asked to accept the Developer Agreement, click Accept

  7. Click on Create a new Application

  8. Enter App Information and ensure that Native is selected for App Type

  9. Click on Create Application

Build a New Docker Image

Once you are in your development environment with Docker (See Set Up The Native Dev Environment):

Pull a base Docker Image

Type sudo docker pull ubuntu - this will pull the base ubuntu Docker image from the Public Docker Registry at index.docker.io. This is one of many docker images that can be used as starting points for development.

Run an Interactive Docker Container and install Python

  1. Type sudo docker run -i -t ubuntu /bin/bash into the development terminal - this command will run the ubuntu docker image in interactive mode which will log you into the container as the root user. This will allow you to make changes to the docker container

  2. Type sudo apt-get update - this command will update the latest pointers to the apt-get repository, this is the repository where we will be installing some basic packages from and is widely used

  3. sudo apt-get install python - this command will install the latest Python installation from the apt-get repository. Python will automatically be installed within the Docker image after running this command

  4. exit - exit the running docker container, we are done making changes for now

Save Changes to Docker Image Locally and Push to Public Registry

  1. sudo docker ps -a - this command will list all docker containers (a docker container is created each time you run a docker image), the docker container is where our latest changes are stored. This command will return a list of container_id's that will be used for the next step

  2. sudo docker commit [container_id] [docker_username]/[new_docker_image_name] - this command will save the changes made in the specified docker container to the docker image in the development environment. These changes are only saved locally after this request and the changes still need to be saved in the Docker registry. The docker_username field can be either an existing docker username or a new one that you would like to create, you will use this username in the future so please choose carefully

  3. sudo docker images - this command will list all docker images, you should see the one you just saved to listed at the top

  4. sudo docker push [docker_username]/[new_docker_image_name] - this command will save the changed docker image in the Docker registry. This command will fail if you have never logged in to Docker before, do not worry if this happens. All you will have to do is verify your email address that you specify in Docker before running the command again

  5. Login or create new Docker account. Ensure that the Username selected matches the docker_username used above, otherwise rebuild Docker image from step 2. Accept the verification email from Docker that is sent to the email you specify.

  6. sudo docker push [docker_username]/[new_docker_image_name] - your changes should now be successfully saved in the public Docker registry at http://index.docker.io

Adding a File to the Docker Image

This section will take you through adding a file to the docker image then saving those changes again.

Create folder on host machine

  1. On the host machine (where you are doing development), type cd /home - navigates to the home directory

  2. sudo mkdir script - making a folder called script that we will mount to a running Docker container in a later step

  3. cd /script - navigate to the folder that was just created

Download file and copy to Docker container

  1. sudo apt-get install wget - install wget from the apt-get repository to download a file in a later step

  2. sudo wget https://da1s119xsxmu0.cloudfront.net/sites/developer/native/hackathon/Sample_python_native_apps_script.py - use wget to download the python file within the /script folder

  3. sudo docker run -i -t -v /home/script:/script [docker_username]/[new_docker_image_name] /bin/bash - run an interactive docker container and mount the /home/script folder from the host machine as the /script folder on the Docker container

  4. cp /script/Sample_python_native_apps_script.py /home/Sample_python_native_apps_script.py - copy the Python script from the /script folder within the Docker container to the home folder location in the Docker container

  5. exit - exit the Docker container

Save the changes made to the Docker image

  1. sudo docker ps -a - this command will list all docker containers (a docker container is created each time you run a docker image), the docker container is where our latest changes are stored. This command will return a list of container_id's that will be used for the next step

  2. sudo docker commit [container_id] [docker_username]/[new_docker_image_name] - this command will save the changes made in the specified docker container to the docker image in the development environment. These changes are only saved locally after this request and the changes still need to be saved in the Docker registry. The docker_username field can be either an existing docker username or a new one that you would like to create, you will use this username in the future so please choose carefully

  3. sudo docker images - this command will list all docker images, you should see the one you just saved to listed at the top

  4. sudo docker push [docker_username]/[new_docker_image_name] - your changes should now be successfully saved in the public Docker registry at http://index.docker.io

Create the Input Form for Example App

What will be covered in the following section:

  • Create/Open a template form via the developer portal
  • Edit the form in the developer portal (the user-facing form that is shown before the appis executed)
  • Edit the callbacks.js script in Formbuilder (these are the arguments that will be sent to the command line of your Docker image)

To build the FastQC App's form, follow these simple steps:

  1. From the App Details page for your application, click on the Form Builder tab

  2. Click on Start building a custom form

  3. Select the Example Form on the Revisions tab

  4. Click Open

  5. Ensure that the Current Template is set to form.

  6. In the left pane of the Formbuilder window, select the Insert Fields dropdown menu and add a checkbox, radiobutton, textbox, and/or numeric

  7. In the left pane of the Formbuilder window, change the id fields for each of the newly added fields from step 5 to checkbox, radio, textbox, and numeric respectively

  8. Click Save and Preview

Configure your Callbacks Script for Example App

The callbacks.js file is important because it informs BaseSpace of the Docker container Id in the Docker registry, and it also includes command-line parameters that are passed to your application upon launch that are needed to run the application.

For the Sample Native app, we will configure the proper callbacks.js script by doing the following:

  1. Navigate to the Formbuilder window

  2. Open the Revision that you wish to change

  3. Ensure that the Current Template is set to callbacks.js

  4. Change the commandLine and containerImageId values to match your Docker image

    containerImageId ---> "[docker_username]/[new_docker_image_name]"
    
    
    commandLine ---> ["python","[location_of_python_script_within_containerImageId]"]  
    

    For Example:

    function launchSpec(dataProvider)
    {
        return 
            {
                commandLine: ["python","/home/Sample_python_native_apps_script.py"],
                containerImageId:"basespace/fastqc"
            };
    }
    
  5. Click Save and Preview

How Data is passed from the User to BaseSpace to the App

In this section, we will discuss how the app finds and uses the information used in the input form.

Saving Form Input Data

In this section, we will show how BaseSpace passes the information that the user selected on the input form to the app itself.

  1. Navigate to the Formbuilder tool and open a revision
  2. Ensure that the Current Template has form selected
  3. Fill out the right side of your form
  4. Click Simulate Launch in Developer Tools
  5. Scroll down to the section called AppSession
  6. Copy/paste the contents of the AppSession into a JSON editor in order to follow along
  7. Navigate to Properties --> Items in the AppSession's JSON

In the Properties Items list, you should see many Items listed. Each item corresponds to each input field element on the form. The value entered in the form for a field can be found by looking for the Value or Content of a Property Item with a Name of input.[field_id]. For example, for a checkbox where we renamed the Id in the form to checkbox, we would look for a Property Item with a Name of input.checkbox to find the value that was selected in the form.

An application would parse through this JSON to find all of the data the user selected on the input form.

This AppSession is saved to your docker image automatically when it is launched from BaseSpace. It is saved as a file called AppSession.json under /data/input/AppSession.json. The app would first parse through this file and assign all of the input values for later computation.

Reading Form Data within the app (Walkthrough Python Script)

In this section, we will walk through the Sample Python Native Apps Script from above to show the general workflow for an application.

Going through the script:

We begin with importing libraries that we will need for the app

import os
import fnmatch 
import sys
import json

Next, we are creating an array to store all of the parameters from the input form via the AppSession:

parameter_list = []
sampleID = []
sampleHref = []
sampleName = []
sampleDir = []

Load json file

jsonfile = open('/data/input/AppSession.json')
jsonObject = json.load(jsonfile)

Determine the number of properties

numberOfPropertyItems = len( jsonObject['Properties']['Items'])

Add parameters from the AppSession to parameters list

if jsonObject['Properties']['Items'][index]['Name'] == 'Input.textbox':
        parameter = jsonObject['Properties']['Items'][index]['Content']
        parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.radio':
            parameter = jsonObject['Properties']['Items'][index]['Content']
            parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.checkbox':
            parameter = jsonObject['Properties']['Items'][index]['Items'][0]
            parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.numeric':
            parameter = jsonObject['Properties']['Items'][index]['Content']
            parameter_list.append(parameter)

Notice here that the we are simply navigating through the AppSession's Property Items and pulling out the values for the input fields by looking for Properties named Input.textbox, Input.radio, Input.checkbox, and Input.numeric. Other field values will not be saved by this script. There are a few global Properties that are created by BaseSpace including Input.Samples, Input.AppResults, Input.Projects, and Input.Files.

This script is jsut to show an example of how to parse some of the field elements, but in the future you will need to ensure that you look for the appropriate Property Items by matching the field Ids to the Property Item Name.

Next, we set the Project id for the output Project

if jsonObject['Properties']['Items'][index]['Name'] == 'Input.Projects':
    projectID = jsonObject['Properties']['Items'][index]['Items'][0]['Id']

We will pull Sample information (from the Sample Chooser in the form)

if jsonObject['Properties']['Items'][index]['Name'] == 'Input.Samples':
for sample in range(len(jsonObject['Properties']['Items'][index]['Items'])):
    sampleID.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Id'])
    sampleHref.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Href'])
    sampleName.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Name'])
        sampleDir = '/data/input/samples/%s/Data/Intensities/BaseCalls' %(sampleID[sample])
        if not os.path.exists(sampleDir):
                sampleDir = '/data/input/samples/%s' %(sampleID[sample])
        for root, dirs, files in os.walk(sampleDir[sample]):
                R1files = fnmatch.filter(files,'*_R1_*')
                R2files = fnmatch.filter(files,'*_R2_*')
        if len(R1files) != len(R2files):
            print "number of R1 and R2 files do not match"
                sys.exit()

The above is simply taking all of the Samples selected on the input form and pulling out the Id, Name, Href, and sets the directories where the Samples can be found.

Now, we'll need to set up the output directory so that results are uploaded back to BaseSpace, extensive documentation about this is available in Native App Conventions

sampleOutDir = '/data/output/appresults/%s/%s' %(projectID,sampleName[sample])
os.system('mkdir -p "%s"' %(sampleOutDir))

For the next step, we are going to create an example output file. This file is simply written with the parameter list with the values from the input form. We will eventually create an example report to display these values as a table in BaseSpace. Normally, this is where your app or algorithm would be executed, so you should not be creating this file but rather running something more for analysis. The output should be the output from your analysis.

file = '%s/parameters.csv' %(sampleOutDir)
outFile = open(file ,'w')
count = 0   
    for parameter in parameter_list:
    count += 1
    outFile.write('%s,%s\n' %(count,parameter))  
        outFile.close()

Optional/Advanced - Create custom Properties on the output using the metadata.json file. This is described further in Native App Conventions.

metadataObject = metadatajson()
metaJsonObject = json.loads(metadataObject)
metaJsonObject['Name'] = jsonObject['Properties']['Items'][index]['Items'][sample]['Id'] 
metaJsonObject['Description'] = 'Sample Description' 
metaJsonObject['HrefAppSession'] = jsonObject['Href']
for href in sampleHref:
    metaJsonObject['Properties'][0]['Items'].append(href) 
metadataFile = '%s/_metadata.json' %(sampleOutDir)
outMetadataFile = open(metadataFile, 'w')
json.dump(metaJsonObject,outMetadataFile)

Run the Sample App

  1. Navigate to the Formbuilder window
  2. In the right pane of Formbuilder, start filling out the form (using small input data)
  3. In the Developer Tools section of this pane, copy the spacedock command-line that is listed there
  4. Paste the spacedock command-line to the development terminal
  5. In the Formbuilder window, click on Send to Local Agent

Create a Simple Example Report

This step requires an analysis with this app to be Complete with results uploaded to BaseSpace.

  1. Navigate to the BaseSpace Developer Portal
  2. Click on My Apps
  3. Select your application from the list
  4. Click on the Reports Builder tab
  5. Click on Start building a custom report
  6. Click OK
  7. You will be asked to select an analysis that has completed for this app, from the list select the Project where you saved the data and you should see the Analysis (AppSession) listed there
  8. Select the Analysis (AppSession)
  9. Click OK
  10. From the Revisions tab, select the Example Report or Duplicate the Example Report
  11. Click Open to open the report for editing
  12. Within the Reportbuilder window, select Options
  13. Uncheck the checkbox next to Summary
  14. Click Save
  15. Click on the Current Template dropdown menu and select result
  16. Replace the left side pane contents with the contents of the Sample App Example Report:

    <head>
    <h2>Output Report for Example Native App</h2>
        <link href="/Content/css/bssdk/basespace.charts.css" rel="stylesheet" type="text/css" />
        <link href="/public/cdn/basespace/css/basestrap.min.css" rel="stylesheet" type="text/css" />
        <link type="text/css" href="/asset.axd?id=ExtraCss" rel="stylesheet" />
    </head>
    <div class="row-fluid">
        <span class="col-span-6">
            <section class="bs-panel">
                <hgroup>
                    <h5>Parameters in Output File</h5>
                </hgroup>
                <table class="table bs-table bs-table-justify">
                    <tbody>
                    {% for key in result.files %}
                        {% for row in result.files[key].noheader.parse.rows %}
                        <tr>
                            <td>{{ row[0] }}</td>
                            <td>{{ row[1] }}</td>
                        </tr>
                        {% endfor %}
                    {% endfor %}
                    </tbody>
                </table>
            </section>
        </span>
    </div>
    

    This is a simple HTML report for the Reportbuilder tool. This code is simply creating an HTML Table (using Twitter Bootstrap) and then it is parsing the output files to pull out the values from the parameters.csv file that is created by the application. If the Ids of the fields from the input form do not match those in the script then this table will be blank, otherwise you should see the field key and value pairs listed in the app's output table as follows:

    Parameters in Output File

    1 50
    2 1
    3 1

View the Report in BaseSpace for Example App

Now, we will navigate to BaseSpace and view our output result as a general user would.

  1. Navigate to the BaseSpace Dashboard
  2. Click on Projects in the header
  3. Select the Project that you created before running the application
  4. Click on the Analysis that you just ran
  5. View the Reports that you just created! You will see two reports for this app.

Summary for Advanced Tutorial

We have just gone through the following:

  • Created a Form for the an Example Python app
  • Ran the Sample_python_native_apps_script.py script in our development environment using the input form from BaseSpace
  • Created a new Docker Image from scratch and saved it in the registry
  • Created output Report for the Example Python app
  • Viewed the Report in BaseSpace

This guide is a more in-depth guide to creating a Native application, for more detailed information please refer to the following Native Apps Documentation:

Or please feel free to post any questions to our google groups!