Setting up your Native Apps Dev Environment - BaseSpace Native Apps

The following section will take you through setting up your development environment to begin developing Native Apps for BaseSpace. If you are unfamiliar with Native Apps, please refer to the Native App Overview before proceeding.

There are two ways to develop Native applications:

  1. Using a Virtual Machine loaded onto your machine
    • Advantages: No cost to the developer, in control of host machine, jobs sent to local machine, preconfigured with all necessary packages
    • Disadvantages: High dependency on developer's network and latency between US East Virginia region where BaseSpace is hosted, generally host machine is not as powerful as machines available in Amazon, no access to the full genomes drive
  2. Using a machine configured with bsfs, spacedock, and docker packages.
    • Advantages: Can use any machine or setup, no cost to developer incurred through BaseSpace

All of these options share three packages, bsfs, spacedock, and docker. Both docker and spacedock are available via the apt-get repository and mono can also be found online.

Our reccomendation is to use an amazon instance configured with these packages for testing, as it is the closest enviromenment to what we use for BaseSpace.

Please follow the instructions in this page for each setup.


Using a Virtual Machine loaded onto your machine

  1. What you will need for the following installation instructions:

    • VirtualBox Installers for Mac and Windows (.exe for Windows, .dmg for Mac), get the latest version of VirtualBox here and download the appropriate one for your Operating System
    • SSH Client for Windows (if on Windows, otherwise this is built in to your Operating System)
    • BaseSpace Native App VM (.ova file), the latest Native Apps Virtual Machine can be downloaded directly from S3 by clicking here. This VM contains the latest installations of BSFS and SpaceDock, and the latest compatible version of Docker which is 1.6.2.
  2. Copy these files to your local computer

  3. Ensure that you are connected to the internet on your local computer

Instructions for Mac or Linux

Install Virtual Box and Import the Native Apps Virtual Machine on Mac or Linux

  1. Install VirtualBox

    a. Select the VirtualBox installer for Mac

    b. Click through the dialogs and install VirtualBox

    c. Restart your computer

    d. Find "Oracle VM VirtualBox" in your applications and open the program

  2. In the VirtualBox window, select File -> Import Appliance...

  3. Click Open Appliance...

  4. Select the BaseSpace Native App VM.ova

  5. Click Next

  6. Ensure that the Reinitialize the MAC address of all selected network cards option is selected

  7. Click Import, this will start importing the VM and the process may take several minutes

  8. Verify that the Virtual Machine was successfully imported by selecting the new VM from the list in Virtual Box and clicking Start. A new window will open and begin loading your VM, if you are prompted to enter a password you have completed setting up your local dev environment.

SSH into your local Virtual Machine

  1. On your local machine, open a terminal by launching the Terminal application

  2. In your terminal's command line, please enter the following: ssh basespace@localhost -p2222 and hit Enter

  3. You will be prompted for a password, the password is basespace


Instructions for Windows Users

Install Virtual Box and Import the Native Apps Virtual Machine on Windows

  1. Install VirtualBox

    a. Select the appropriate VirtualBox installer for your OS

    b. Click through the dialogs and install VirtualBox

    c. Restart your computer

    d. Find "Oracle VM VirtualBox" in your applications and open the program

  2. In the VirtualBox window, select File -> Import Appliance...

  3. Click Open Appliance...

  4. Select the BaseSpace Native App VM.ova

  5. Click Next

  6. Ensure that the Reinitialize the MAC address of all selected network cards option is selected

  7. Click Import, this will start importing the VM and the process may take several minutes

  8. Verify that the Virtual Machine was successfully imported by selecting the new VM from the list in Virtual Box and clicking Start. A new window will open and begin loading your VM, if you are prompted to enter a password you have completed setting up your local dev environment.

SSH into your local Virtual Machine

  1. Download and install an SSH client if you do not already have one. Some examples of popular SSH clients: PuTTY, OpenSSH, and WinSCP

  2. Open your SSH client, the following example will be using PuTTY

  3. Fill in the following information:

    Host Name (or IP Address): basespace@localhost Port: 2222

  4. Click Open

  5. A terminal will open and you will be prompted for a password, the password is basespace


Successfully Login to the Virtual Machine

Upon successful login to the Virtual Machine, you should see the terminal change to:

vagrant@precise64:~$

Creating your Native Application

Congratulations, you've finished setting up your local computer as a development environment for your Native Application for BaseSpace!

You can now use the other documentation in the developer portal to begin building your Docker image for your Native App.

For more general information about native apps, please refer to the Native App Overview

For more information on how to build your Docker image for your application, please refer to Managing your Docker Image.

For more information about the conventions of Native Apps including automatic download, upload, authentication and error reporting, please refer to the Native App Conventions.

For information on how to debug and test your application locally, please refer to Testing and Debugging your Native App Locally


Using a machine you have configured

There are two ways to configure your machine to be a Native Apps Development environment.

Only 64-bit Linux operating systems are supported.

  • Using the SpaceDock docker image
    • Install Docker
    • Install SpaceDock docker image (also includes BSFS and Mono)
  • Using manual installation of SpaceDock, Mono, and BSFS
    • Install Docker
    • Install Mono Xamarin package
    • Install SpaceDock debian package
    • Install BSFS

For this environment setup, we are using the following versions:

  • Docker 1.6.2
  • Mono 3.12.0
  • Ubuntu 15

Using the SpaceDock docker image

Download and Install Docker

In your development machine, type

sudo apt-get update

Download the Docker repository key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

Setup the Docker debian package

sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

Update debian packages

sudo apt-get update

Install Docker 1.6.2 (currently supported with latest SpaceDock version)

sudo apt-get install lxc-docker-1.6.2

We'll have to enable communication over HTTP for your Docker service, this can be done by first opening the Docker config file:

sudo vim /etc/default/docker

Replace and uncomment DOCKER_OPTS with the following:

DOCKER_OPTS="-d -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"

To run docker with the new changes, restart the docker service

sudo service docker restart

For Ubuntu 15, in order to have the Docker service start on machine startupm run the following command:

systemctl enable docker

Download and Install SpaceDock

Download the SpaceDock docker image

sudo docker pull basespace/spacedock

This Docker image already contains BSFS, Mono, and SpaceDock.

Run the SpaceDock Docker image

Now, we will have to run this SpaceDock Docker image with the following command

Use --follow to see the log as its running

sudo docker run -d --net=host --privileged=true -v /data:/data -v /genomes:/genomes basespace/spacedock spacedock -a <agent_id> -m <mission_control_uri> 

This command will return a container_id which is the docker container id for the spacedock docker image that you just ran.

You can see what is going on within the spacedock docker container (to view jobs as they are processing) by using the container_id from above:

sudo docker attach <container_id>

The machine is now ready to receive Native apps jobs from BaseSpace.

Using manual installation of SpaceDock, Mono, and BSFS

Any Linux machine can be configured as a Native Apps development machine by simply installing the following packages:

  1. Mono: Mono is a package that allows C# code to be run in an environment, this is essential for the Native apps process. Mono is available online and can be installed using a Xamarin package which will be covered later
  2. Docker: The Docker engine is also available in the apt-get repository. Docker needs to be installed in order for the machine to run Docker images which are light-weight containers that the developer configures for their BaseSpace app. More information about Docker is available here.
  3. SpaceDock: can be found in the apt-get repository, the package is called spacedock. This is developed by the BaseSpace team and manages a lot of functionality within the native app. More information about SpaceDock can be found in the Native App Conventions documentation.

To make things easier, we would recommend using sudo su to work as the root user on your machine, this will make the following commands simpler. If there is a permissions issue with any of the requests, you will need to use root access by either doing the above or adding sudo appropriately in each command.

Download and Install Docker

In your development machine, type

sudo apt-get update

Download the Docker repository key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

Setup the Docker debian package

sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

Update debian packages

sudo apt-get update

Install Docker 1.6.2 (currently supported with latest SpaceDock version)

sudo apt-get install lxc-docker-1.6.2

We'll have to enable communication over HTTP for your Docker service, this can be done by first opening the Docker config file:

sudo vim /etc/default/docker

Replace and uncomment DOCKER_OPTS with the following:

DOCKER_OPTS="-d -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"

To run docker with the new changes, restart the docker service

sudo service docker restart

For Ubuntu 15, in order to have the Docker service start on machine startupm run the following command:

systemctl enable docker

Docker is now set up in your environment!

Download and install Mono

Add the mono debian key

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

Specify the mono version that you would like to install, for Native App development we are currently using mono version 3.12.0

echo "deb http://download.mono-project.com/repo/debian wheezy/snapshots/3.12.0 main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list

Update debian package repositories

sudo apt-get update

Install the specified mono version

sudo apt-get install -y mono-complete

Now you should have mono properly setup and ready to go!

Download and install SpaceDock

Add the SpaceDock debian repository

echo deb http://basespace-apt.s3.amazonaws.com spacedock main | sudo tee -a /etc/apt/sources.list; sudo apt-get update; sudo apt-cache policy spacedock

Update debian packages

sudo apt-get update

Install SpaceDock

sudo apt-get install spacedock

SpaceDock should now be ready to use in your Native Apps development environment!

Download and install BSFS

Add BSFS debian repository

sudo add-apt-repository "deb http://basespace-apt.s3.amazonaws.com bsfs main"
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test

Update and install the BSFS package

sudo apt-get update
sudo apt-cache policy bsfs
sudo apt-get -y --force-yes install bsfs

Note: In some cases, your 64-bit OS may not be detected. In this case, you will see an error when you try apt-get update so to fix this you will have to open the file /etc/apt/sources.list and add [arch=amd64] to the bsfs repository so it looks like deb [arch=amd64] http://basespace-apt.s3.amazonaws.com bsfs main.

Create dependent directories

In the root folder on your machine, make the /data and /genomes folders. These are necessary in order for BaseSpace to run the app.

sudo mkdir /data

sudo mkdir /genomes

Run SpaceDock

Before your machine can receive jobs from BaseSpace, you will have to run SpaceDock using the following command

sudo spacedock -a <agent_id> -m <mission_control_url>

Now you are ready to build and test your BaseSpace Native app!


Data Management on Development Machine

There are a few practices that will help clean up the development environment. As you run your Native apps through the development environment, the data can remain on the machine after the job is executed. In order to clean up the host development machine, we would recommend:

  • Removing all unnecessary Docker images and containers, more information can be found here
  • Removing all data in the /data directory
    • Each time you run an analysis on the host machine, a new folder is created within the /data directory on the host machine
    • You can also use data in this folder to run your docker image locally to test it without using BaseSpace
  • Removing all data in the root directory
    • Each time you run an analysis on the host machine, new data may be created in the root directoy