In this section, the developer will learn how to create example Native apps for BaseSpace. There are two walkthroughs in this section:
The following guide will take the developer through a very simple example of creating and testing the FastQC Native App.
Please refer to the Set Up The Native Dev Environment to set up your local development environment to begin creating Native applications.
This guide assumes that the local development environment is already set up. In addition, the Native Apps Virtual Machine is running locally on your computer and you have already SSH'd into your Virtual Machine from a local terminal. The password is basespace to SSH into the Virtual Machine.
It is important to regularly update the SpaceDock package on your Virtual Machine. An outdated version of SpaceDock may cause issues while testing the app. For more information about updating SpaceDock, please refer to Native App Conventions Documentation.
The following files will be needed for the following walkthrough. You can either download these locally or simply copy and paste the contents of each file when it is needed.
FastQC App .txt files:
In addition, you will also have to import a dataset to your BaseSpace account, please follow this link for the Project Share in BaseSpace: https://basespace.illumina.com/s/LbRGgqcP0qTR
In a browser window, go to BaseSpace: http://basespace.illumina.com
If you already have a BaseSpace account, click Login, otherwise click Sign Up
Once you are logged in, you will be asked to accept an Agreement, click Accept
Now, you will be taken to the BaseSpace dashboard, find the Developers link in the footer and click it
You are now on the developer portal, click on My Apps
You may be asked to accept the Developer Agreement, click Accept
Click on Create a new Application
Enter App Information and ensure that Native is selected for App Type
Click on Create Application
Navigate to https://basespace.illumina.com/s/LbRGgqcP0qTR, you should get a message that asks you to accept the share, click Accept.
You will now have the SmallSampleProject in your list of Projects in BaseSpace. This Project is small so that we can show a quick analysis of the FastQC app.
What will be covered in the following section:
To build the FastQC App's form, follow these simple steps:
From the App Details page for your application, click on the Form Builder tab
Click on Start building a custom form
Select the Example Form on the Revisions tab
Click Open
Copy the contents of FastQC_Input_Form txt file
In the left pane of the Formbuilder window, Select All of the code in that pane and delete it
Paste the contents of the FastQC_Input_Form in the left pane of the Formbuilder window
Click Save and Preview
The callbacks.js file is important because it informs BaseSpace of the Docker container Id in the Docker registry, and it also includes command-line parameters that are passed to your application upon launch that are needed to run the application.
For the FastQC app, we will configure the proper callbacks.js script by doing the following:
Click on the Current Template dropdown at the top and select callbacks.js
Copy the contents of FastQC_Callbacks_js
In the left pane of the Formbuilder window, Select All of the code in that pane and delete it
Paste the contents of the FastQC_Callbacks_js in the left pane of the Formbuilder window
Click Save and Preview
This section assumes that you have:
What will be covered in the following section:
Now, lets use see this in action:
In the Formbuilder window, under Developer Tools, find and copy the command after Sample command-line to start local agent:
Keep this browser window open
Paste the copied command from step 1 into your SSH terminal (e.g. sudo spacedock -a ... -m https://mission.basespace.illumina.com)
Hit Enter, the terminal will show a series of commands executed. Once you see the message Successfully pinged docker service, you can move on to step 5. This just started the service on your Virtual Machine that will begin polling BaseSpace for work for your specific app.
Now, switch over to your Formbuilder window where you have the FastQC app's form opened in the editor
In the right pane of the Formbuilder window, click Select Sample
Select SmallSampleProject and select truseql1_8899
Click Ok, you have now select the input Sample for your application
Click Select Project in the right pane of the Formbuilder window
Click New... to create a new Project in your BaseSpace account for the output results (you can also select a pre-existing Project that you have write permission to)
Enter Project information and click Create
In the Developer Tools of the Formbuilder window, click Send to Local Agent to send this job from BaseSpace to your local Virtual Machine
In your terminal or command prompt window on your local machine, you will see your Virtual Machine receive the job from BaseSpace and do the following:
basespace/fastqc Docker container from the Docker registry (or update the one that already exists on the Virtual Machine)basespace/fastqcCompleteYou can view the output log in the Formbuilder window, this will also be displayed for this Analysis in BaseSpace
What will be covered in the following section:
Lets try it out:
Navigate to the Developer Portal
Click on My Apps
Click on the FastQC application that you created in an earlier step
Click on Reports Builder
Click on Start building a custom report
Click Ok on the popup message, this indicates that you will need to have a Completed AppSession with output results in order to proceed
Select the available AppSession from the list and click Ok
Select Example Report
Click Open
Select Options
Uncheck the Summary checkbox
Click Save
Click on the dropdown menu labelled Current Template and select result
Select all of the code in the left pane of the Reports Builder and delete it
Copy the contents of FastQC_Output_Report and paste them into the left pane of the Reports Builder window
Click Save and Preview
Click on the Back Arrow Button
Select your Example Report and click Activate
Now, we will navigate to BaseSpace and view our output result as a general user would.
We have just gone through the following:
This guide is a very high-level guide to creating a Native application, for more detailed information please refer to the following Native Apps Documentation:
Native Apps Overview : For a general overview of Native Apps
Native App Conventions : to learn more about the Native apps infrastructure and to go in-depth on allof the features that the SpaceDock package supports
Setting up Local Native Dev Environment : To learn how to set up your local development enviroment for Native Apps
Manage Your Docker Image : To learn more about Docker and how to build your Docker container for your application
Testing and Debugging Your Native App : To learn more about using the tools offered by BaseSpace to test and debug your application locally
Publishing Your BaseSpace App: For more information about how to publish your BaseSpace App
Or please feel free to post any questions to our google groups!
This example will take a developer through the process of creating a new Native app in BaseSpace with an example python script. The script could also be written in another language, but for this example we are using python.
In this tutorial, the developer will learn how to create a new Docker image
Please refer to the Set Up The Native Dev Environment to set up your local development environment to begin creating Native applications.
This guide assumes that the local development environment is already set up. In addition, the Native Apps Virtual Machine is running locally on your computer and you have already SSH'd into your Virtual Machine from a local terminal. The password is basespace to SSH into the Virtual Machine.
It is important to regularly update the SpaceDock package on your Virtual Machine. An outdated version of SpaceDock may cause issues while testing the app. For more information about updating SpaceDock, please refer to Native App Conventions Documentation.
We are using the Public Docker Registry for this example. For Private app development, please use the Private Docker Registry that is hosted by BaseSpace. Documentation can be found here.
The following files will be needed for the following walkthrough. You can either download these locally, use the links and download the files, or simply copy and paste the contents of each file when it is needed.
Example App files:
In addition, you will also have to import a dataset to your BaseSpace account, you can use any public dataset (choose a small one for fastest results) or please follow this link for the Project Share in BaseSpace: https://basespace.illumina.com/s/LbRGgqcP0qTR
In a browser window, go to BaseSpace: http://basespace.illumina.com
If you already have a BaseSpace account, click Login, otherwise click Sign Up
Once you are logged in, you will be asked to accept an Agreement, click Accept
Now, you will be taken to the BaseSpace dashboard, find the Developers link in the footer and click it
You are now on the developer portal, click on My Apps
You may be asked to accept the Developer Agreement, click Accept
Click on Create a new Application
Enter App Information and ensure that Native is selected for App Type
Click on Create Application
Once you are in your development environment with Docker (See Set Up The Native Dev Environment):
Type sudo docker pull ubuntu - this will pull the base ubuntu Docker image from the Public Docker Registry at index.docker.io. This is one of many docker images that can be used as starting points for development.
Type sudo docker run -i -t ubuntu /bin/bash into the development terminal - this command will run the ubuntu docker image in interactive mode which will log you into the container as the root user. This will allow you to make changes to the docker container
Type sudo apt-get update - this command will update the latest pointers to the apt-get repository, this is the repository where we will be installing some basic packages from and is widely used
sudo apt-get install python - this command will install the latest Python installation from the apt-get repository. Python will automatically be installed within the Docker image after running this command
exit - exit the running docker container, we are done making changes for now
sudo docker ps -a - this command will list all docker containers (a docker container is created each time you run a docker image), the docker container is where our latest changes are stored. This command will return a list of container_id's that will be used for the next step
sudo docker commit [container_id] [docker_username]/[new_docker_image_name] - this command will save the changes made in the specified docker container to the docker image in the development environment. These changes are only saved locally after this request and the changes still need to be saved in the Docker registry. The docker_username field can be either an existing docker username or a new one that you would like to create, you will use this username in the future so please choose carefully
sudo docker images - this command will list all docker images, you should see the one you just saved to listed at the top
sudo docker push [docker_username]/[new_docker_image_name] - this command will save the changed docker image in the Docker registry. This command will fail if you have never logged in to Docker before, do not worry if this happens. All you will have to do is verify your email address that you specify in Docker before running the command again
Login or create new Docker account. Ensure that the Username selected matches the docker_username used above, otherwise rebuild Docker image from step 2. Accept the verification email from Docker that is sent to the email you specify.
sudo docker push [docker_username]/[new_docker_image_name] - your changes should now be successfully saved in the public Docker registry at http://index.docker.io
This section will take you through adding a file to the docker image then saving those changes again.
On the host machine (where you are doing development), type cd /home - navigates to the home directory
sudo mkdir script - making a folder called script that we will mount to a running Docker container in a later step
cd /script - navigate to the folder that was just created
sudo apt-get install wget - install wget from the apt-get repository to download a file in a later step
sudo wget https://da1s119xsxmu0.cloudfront.net/sites/developer/native/hackathon/Sample_python_native_apps_script.py - use wget to download the python file within the /script folder
sudo docker run -i -t -v /home/script:/script [docker_username]/[new_docker_image_name] /bin/bash - run an interactive docker container and mount the /home/script folder from the host machine as the /script folder on the Docker container
cp /script/Sample_python_native_apps_script.py /home/Sample_python_native_apps_script.py - copy the Python script from the /script folder within the Docker container to the home folder location in the Docker container
exit - exit the Docker container
sudo docker ps -a - this command will list all docker containers (a docker container is created each time you run a docker image), the docker container is where our latest changes are stored. This command will return a list of container_id's that will be used for the next step
sudo docker commit [container_id] [docker_username]/[new_docker_image_name] - this command will save the changes made in the specified docker container to the docker image in the development environment. These changes are only saved locally after this request and the changes still need to be saved in the Docker registry. The docker_username field can be either an existing docker username or a new one that you would like to create, you will use this username in the future so please choose carefully
sudo docker images - this command will list all docker images, you should see the one you just saved to listed at the top
sudo docker push [docker_username]/[new_docker_image_name] - your changes should now be successfully saved in the public Docker registry at http://index.docker.io
What will be covered in the following section:
To build the FastQC App's form, follow these simple steps:
From the App Details page for your application, click on the Form Builder tab
Click on Start building a custom form
Select the Example Form on the Revisions tab
Click Open
Ensure that the Current Template is set to form.
In the left pane of the Formbuilder window, select the Insert Fields dropdown menu and add a checkbox, radiobutton, textbox, and/or numeric
In the left pane of the Formbuilder window, change the id fields for each of the newly added fields from step 5 to checkbox, radio, textbox, and numeric respectively
Click Save and Preview
The callbacks.js file is important because it informs BaseSpace of the Docker container Id in the Docker registry, and it also includes command-line parameters that are passed to your application upon launch that are needed to run the application.
For the Sample Native app, we will configure the proper callbacks.js script by doing the following:
Navigate to the Formbuilder window
Open the Revision that you wish to change
Ensure that the Current Template is set to callbacks.js
Change the commandLine and containerImageId values to match your Docker image
containerImageId ---> "[docker_username]/[new_docker_image_name]"
commandLine ---> ["python","[location_of_python_script_within_containerImageId]"]
For Example:
function launchSpec(dataProvider)
{
return
{
commandLine: ["python","/home/Sample_python_native_apps_script.py"],
containerImageId:"basespace/fastqc"
};
}
Click Save and Preview
In this section, we will discuss how the app finds and uses the information used in the input form.
In this section, we will show how BaseSpace passes the information that the user selected on the input form to the app itself.
In the Properties Items list, you should see many Items listed. Each item corresponds to each input field element on the form. The value entered in the form for a field can be found by looking for the Value or Content of a Property Item with a Name of input.[field_id]. For example, for a checkbox where we renamed the Id in the form to checkbox, we would look for a Property Item with a Name of input.checkbox to find the value that was selected in the form.
An application would parse through this JSON to find all of the data the user selected on the input form.
This AppSession is saved to your docker image automatically when it is launched from BaseSpace. It is saved as a file called AppSession.json under /data/input/AppSession.json. The app would first parse through this file and assign all of the input values for later computation.
In this section, we will walk through the Sample Python Native Apps Script from above to show the general workflow for an application.
Going through the script:
We begin with importing libraries that we will need for the app
import os
import fnmatch
import sys
import json
Next, we are creating an array to store all of the parameters from the input form via the AppSession:
parameter_list = []
sampleID = []
sampleHref = []
sampleName = []
sampleDir = []
Load json file
jsonfile = open('/data/input/AppSession.json')
jsonObject = json.load(jsonfile)
Determine the number of properties
numberOfPropertyItems = len( jsonObject['Properties']['Items'])
Add parameters from the AppSession to parameters list
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.textbox':
parameter = jsonObject['Properties']['Items'][index]['Content']
parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.radio':
parameter = jsonObject['Properties']['Items'][index]['Content']
parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.checkbox':
parameter = jsonObject['Properties']['Items'][index]['Items'][0]
parameter_list.append(parameter)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.numeric':
parameter = jsonObject['Properties']['Items'][index]['Content']
parameter_list.append(parameter)
Notice here that the we are simply navigating through the AppSession's Property Items and pulling out the values for the input fields by looking for Properties named Input.textbox, Input.radio, Input.checkbox, and Input.numeric. Other field values will not be saved by this script. There are a few global Properties that are created by BaseSpace including Input.Samples, Input.AppResults, Input.Projects, and Input.Files.
This script is jsut to show an example of how to parse some of the field elements, but in the future you will need to ensure that you look for the appropriate Property Items by matching the field Ids to the Property Item Name.
Next, we set the Project id for the output Project
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.Projects':
projectID = jsonObject['Properties']['Items'][index]['Items'][0]['Id']
We will pull Sample information (from the Sample Chooser in the form)
if jsonObject['Properties']['Items'][index]['Name'] == 'Input.Samples':
for sample in range(len(jsonObject['Properties']['Items'][index]['Items'])):
sampleID.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Id'])
sampleHref.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Href'])
sampleName.append(jsonObject['Properties']['Items'][index]['Items'][sample]['Name'])
sampleDir = '/data/input/samples/%s/Data/Intensities/BaseCalls' %(sampleID[sample])
if not os.path.exists(sampleDir):
sampleDir = '/data/input/samples/%s' %(sampleID[sample])
for root, dirs, files in os.walk(sampleDir[sample]):
R1files = fnmatch.filter(files,'*_R1_*')
R2files = fnmatch.filter(files,'*_R2_*')
if len(R1files) != len(R2files):
print "number of R1 and R2 files do not match"
sys.exit()
The above is simply taking all of the Samples selected on the input form and pulling out the Id, Name, Href, and sets the directories where the Samples can be found.
Now, we'll need to set up the output directory so that results are uploaded back to BaseSpace, extensive documentation about this is available in Native App Conventions
sampleOutDir = '/data/output/appresults/%s/%s' %(projectID,sampleName[sample])
os.system('mkdir -p "%s"' %(sampleOutDir))
For the next step, we are going to create an example output file. This file is simply written with the parameter list with the values from the input form. We will eventually create an example report to display these values as a table in BaseSpace. Normally, this is where your app or algorithm would be executed, so you should not be creating this file but rather running something more for analysis. The output should be the output from your analysis.
file = '%s/parameters.csv' %(sampleOutDir)
outFile = open(file ,'w')
count = 0
for parameter in parameter_list:
count += 1
outFile.write('%s,%s\n' %(count,parameter))
outFile.close()
Optional/Advanced - Create custom Properties on the output using the metadata.json file. This is described further in Native App Conventions.
metadataObject = metadatajson()
metaJsonObject = json.loads(metadataObject)
metaJsonObject['Name'] = jsonObject['Properties']['Items'][index]['Items'][sample]['Id']
metaJsonObject['Description'] = 'Sample Description'
metaJsonObject['HrefAppSession'] = jsonObject['Href']
for href in sampleHref:
metaJsonObject['Properties'][0]['Items'].append(href)
metadataFile = '%s/_metadata.json' %(sampleOutDir)
outMetadataFile = open(metadataFile, 'w')
json.dump(metaJsonObject,outMetadataFile)
This step requires an analysis with this app to be Complete with results uploaded to BaseSpace.
Replace the left side pane contents with the contents of the Sample App Example Report:
<head>
<h2>Output Report for Example Native App</h2>
<link href="/Content/css/bssdk/basespace.charts.css" rel="stylesheet" type="text/css" />
<link href="/public/cdn/basespace/css/basestrap.min.css" rel="stylesheet" type="text/css" />
<link type="text/css" href="/asset.axd?id=ExtraCss" rel="stylesheet" />
</head>
<div class="row-fluid">
<span class="col-span-6">
<section class="bs-panel">
<hgroup>
<h5>Parameters in Output File</h5>
</hgroup>
<table class="table bs-table bs-table-justify">
<tbody>
{% for key in result.files %}
{% for row in result.files[key].noheader.parse.rows %}
<tr>
<td>{{ row[0] }}</td>
<td>{{ row[1] }}</td>
</tr>
{% endfor %}
{% endfor %}
</tbody>
</table>
</section>
</span>
</div>
This is a simple HTML report for the Reportbuilder tool. This code is simply creating an HTML Table (using Twitter Bootstrap) and then it is parsing the output files to pull out the values from the parameters.csv file that is created by the application. If the Ids of the fields from the input form do not match those in the script then this table will be blank, otherwise you should see the field key and value pairs listed in the app's output table as follows:
Parameters in Output File
1
50
2
1
3
1
Now, we will navigate to BaseSpace and view our output result as a general user would.
We have just gone through the following:
Sample_python_native_apps_script.py script in our development environment using the input form from BaseSpaceThis guide is a more in-depth guide to creating a Native application, for more detailed information please refer to the following Native Apps Documentation:
Native Apps Overview : For a general overview of Native Apps
Native App Conventions : to learn more about the Native apps infrastructure and to go in-depth on allof the features that the SpaceDock package supports
Setting up Local Native Dev Environment : To learn how to set up your local development enviroment for Native Apps
Manage Your Docker Image : To learn more about Docker and how to build your Docker container for your application
Testing and Debugging Your Native App : To learn more about using the tools offered by BaseSpace to test and debug your application locally
Publishing Your BaseSpace App: For more information about how to publish your BaseSpace App
Or please feel free to post any questions to our google groups!