Testing and Debugging your Native Application using the Formbuilder

When building your Native App, you will find the need to test your app locally using real data from your BaseSpace account.

You can accomplish this by using the Formbuilder tool in the developer portal. If you are not familiar with the Formbuilder, please refer to the Formbuilder Documentation.

Getting started with a new Native App and Form

Setting up your Local Development Environment

The following steps assume that you have the local development environment set up for Native Apps. This guide will not go through the steps for setting this up, but you can refer to this documentation to guide you through the process.

Once you have your Native App Virtual Machine running locally and have a terminal open that is logged into your virtual machine via SSH, you are ready to begin testing the application locally.

Point BaseSpace to your Local Virtual Machine

Visit the developer portal and click on My Apps
Click on your existing Native Application or create a new one (ensure that the App Type is set to Native)
Under your app's name, you will see a series of tabs starting with Details
Click on the Form Builder tab
Click on the Begin Creating and Editing Custom Forms For Your App button
Select the form you wish to open (or use the Example form for now) and click Open
Configure the form to have the proper inputs in order for your app to run using the Formbuilder Documentation
Click on the Current Template dropdown at the top and select callbacks.js
Modify the launchSpec function in the callbacks.js file to send the appropriate command line arguments to your Docker image to execute the app. Parameters may also be passed in that are needed for the app.

Using Developer Tools in the Formbuilder

You will now see the Developer Tools section in the right side pane of the Formbuilder window.

In this section, you will see the Sample command-line to start local agent:. Copy the contents after this command (e.g. sudo spacedock -a [agent_id] -m https://hoth-mission.basespace.illumina.com) into the terminal where you are logged into the virtual machine via SSH and hit enter, you should see a message that says something like Successfully pinged Docker service if this was completed successfully.

Now, you have pointed BaseSpace to your local Virtual Machine to run jobs that are submitted by the app in BaseSpace via the Send to Local Agent button in the formbuilder.

There are three buttons listed here:

Simulate Launch: Simulates launching the app by showing the developer a page that has a mock AppSession created with the Properties of the input form.
Create AppSession: Simulates launching the app by creating the AppSession with Properties based on the input form
Send to Local Agent: Creates a new job specified by the Input Form and sends that job from BaseSpace to your local Virtual Machine to be run locally, if you followed the previous step and ran the sudo spacedock ... command you will see a job being submitted to your local terminal as soon as this button is pressed

Simulate Launch

The Simulate Launch option takes the developer to another page that shows the information that is created upon submittal of the form and the launch of the app.

This information includes:

The AppSession id in BaseSpace for the AppSession that would normally be generated (this option does not actually create the AppSession, see message at the top of the screenshot above)
The access token that was generated for the app to have access to the appropriate data in a user's account
The scope of the permissions that are allowed by the access token (based on your configured form)
The launch specification which includes the Id of your Docker Image in the registry and the command line that will be fed to your app upon launch
An area that displays the full AppSession with properties included.

Create AppSession

The Create AppSession option takes the developer to another page that shows the information that is created upon submittal of the form and the launch of the app. This option is different from Simulate Launch because this option actually creates the new AppSession instead of just simulating the AppSession that would be created, so you can access that AppSession via the REST API for BaseSpace if needed.

Otherwise, the page that the developer is taken to after clicking on this button is the same as the one for Simulate Launch except that the message at the top is different and indicates that an AppSession was actually created.

Send to Local Agent

The Send to Local Agent option is slightly different from the other options in the Developer Tools. Clicking on Send to Local Agent will send the job from BaseSpace's job scheduler to the local Native Apps Virtual Machine that is running on your machine. BaseSpace will send this job to the Virtual Machine where you pasted the Sample command-line to start local agent information from the Developer Tools.

The job that BaseSpace sends to your local Virtual Machine will include all of the information that is generated via the Create AppSession and Simulate Launch options. The command-line argument in your callbacks.js function is fed to the Docker image once it is found or downloaded locally and then it is executed.

To use the Send to Local Agent option, you will need to follow these steps:

Run the Native Apps Virtual Machine locally
SSH into the Virtual Machine from your terminal and login (the username and password are both basespace)
Go to your form in the Formbuilder and copy the line after Sample command-line to start local agent and paste it into the terminal where you are logged into the virtual machine via SSH
```
sudo spacedock -a ... -m https://hoth-mission.basespace.illumina.com
```
Once successful, you should see a message in your terminal that says successfully pinged docker service, this means that your local Virtual Machine is now polling the BaseSpace Job Scheduler for work. Any time a new job for this application is submitted from BaseSpace, if the Send to Local Agent option is chosen by the developer, the job will execute in their local Virtual Machine and you can see this process in real-time!

The value after the -a is your specific Agent Id, this is how BaseSpace knows to send jobs from this application to this local Virtual Machine. The url is the URL to BaseSpace's Job Scheduler.
Ensure that your callbacks.js file is referencing the correct Docker image for your app and that the command line argument that is being sent is sufficient for your app to run

The callbacks.js function can be edited by opening a form, clicking on the dropdown next to Current Template, and selecting callbacks.js. The callbacks.js function is used specifically for Native apps and determines the information that is passed to your Docker image from BaseSpace.

Native apps can be run on one Node or the process can be optimized to be run on multiple Nodes. The documentation will go through both scenarios.

Single-Node Template:

function launchSpec(dataProvider)
{
    return 
        {
            commandLine: ["/helloWorld.sh"],
            containerImageId:"tliu1/helloworld"
        };
}

This is a Single-Node analysis template, which means that the app will run on one node and that node will have all of the input data.

In the above example, we see the following actions in the callbacks.js file:

return : This is the return statement that will be sent to the machine that is running the app.
commandLine: Designates that the following commands will be entered into the command-line of your Docker container to start your application
- ["/helloworld.sh"] : This will generate the following command-line argument python home/apps/fastqc/fastqc.py projectId. /helloworld.sh means to run the /helloworld.sh Shell script located in the root folder. If it is in another path, the path should be specified in this command.
containerImageId: The Docker container in the Docker registry for this application
- "tliu1/helloworld": This is the Docker container in the Docker registry. tliu1 is the username and helloworld is the name of the Docker image. This container is where the above command-line script will be entered.

In addition to the above, the Callbacks.js function can also be used to specify running an analysis across multiple nodes.

Single-Node Analysis Using Multi-Node Template:

function launchSpec(dataProvider)
{
    var retval = { nodes: [] };

    var samples = dataProvider.GetProperty("Input.sample-id");

    retval.nodes.push({
          commandLine: ["/helloworld.sh"],
          containerImageId:"tliu1/helloworld”,
                  properties: {
                                “Input.Samples”: samples[0]
                  }
    });

    return retval;
}

var samples = dataProvider.GetProperty("Input.sample-id"); : This command is setting the variable samples as the Id of the Property Input.samples-id that was set for a field in the form for this app. If multiple Samples are selected as input, this variable will be an array of Sample Ids.
return : This is the return statement that will be sent to the machine that is running the app.
commandLine: Designates that the following commands will be entered into the command-line of your Docker container to start your application
- ["/helloworld.sh"] : This will generate the following command-line argument /helloworld.sh which means to run the /helloworld.sh Shell script located in the root folder. If it is in another path, the path should be specified in this command.
containerImageId: The Docker container in the Docker registry for this application
- "tliu1/helloworld": This is the Docker container in the Docker registry. tliu1 is the username and helloworld is the name of the Docker image. This container is where the above command-line script will be entered.

Multi-Node Analysis Using Multi-Node Template:

function launchSpec(dataProvider)
{
    var retval = { nodes: [] };

    var samples = dataProvider.GetProperty("Input.sample-id");
    var project = dataProvider.GetProperty("Input.project-id");

    for(var i = 0; i < samples.length; i++)
    {
        retval.nodes.push({
              commandLine: ["/helloworld.sh"],
              containerImageId:"tliu1/helloworld",
          properties: {
            “Input.Samples”: [ samples[i] ],
            "Input.project-id": project
          }
        });
    }

    return retval;
}

In the above callbacks.js function, the samples variable is an array of multiple Samples. This array is iterated through in the following for loop and each Node gets a new Sample from the sample[i] value.

In the future, the callbacks.js function will also be used to specify compute requirements for your application.

Click Send to Local Agent

You should now be taken to the AppSession page for this job in BaseSpace in the Formbuilder window. You can see a running log of output from the app.

If you switch to the terminal where you are logged into the virtual machine via SSH, you will see the app execute by doing the following:
- Downloading the Docker image
- Downloading the input data from the form to the data/input folder
- Running the command-line argument from the callbacks.js
- Execute your script within your Docker container
- Uploading results back to BaseSpace from the data/output
- Mark the AppSession as Complete
The app will keep updating the AppSession with Status and Status Summary throughout the above process.

The Launch Button

The Launch option will launch the app in the BaseSpace Infrastructure instead of sending it to your local Virtual Machine. The job is sent to the BaseSpace Job Scheduler, which then finds or creates an Amazon Machine Image (AMI) to run the app. This AMI's specifications are similar to that of the Native Apps Virtual Machine that we provide. Once the AMI receives the job from the Job Scheduler, it downloads the Docker image to the AMI from the Docker registry and runs the command-line argument in the callbacks.js.

The Launch button is meant to test launch the app as a user would using BaseSpace infrastructure rather than your local machine with the Virtual Machine running.

Using the API to Test your Native App

The API can now be used to programmatically launch a native app. The details of this mechanism are described in the API Reference page under POST: applications/{id}/appsessions.

There are two ways to launch a Native app using this API, either in the BaseSpace infrastructure or on your local machine. In order to launch the app in the BaseSpace infrastructure, you will first need to submit the app for review and ask us for permission to launch. If you are using a local machine to launch the app, an AgentId can be provided with this request to send the job to your local machine. The AgentId is found in the Native Settings tab for your app in the developer portal.

You'll need an access-token and the Id of the application that you wish to launch. This access token can be found in the Credentials tab for any of your apps in the developer portal, with any of these access tokens you will be able to launch any apps that you have access to as a user or developer.

The API Reference will provide more detailed information about the process of lauching the app programmatically.