BaseSpace Sequence Hub CLI

You can work with your BaseSpace Sequence Hub data using the command line interface (CLI). The BaseSpace Sequence Hub CLI supports scripting and programmatic access to BaseSpace Sequence Hub for automation, bulk operations, and other routine functions. It can be used independently or in conjunction with BaseMount.

The BaseSpace Sequence Hub CLI has been updated to support automation features in the latest BaseSpace Sequence Hub.

To see the overall capabilities of the CLI and to help you get started post-installation, we recommend you use the Worked Examples.

Install BaseSpace Sequence Hub CLI

The BaseSpace Sequence Hub CLI is supported on Linux, Mac, and Windows (32 and 64 bit). It carries a small memory footprint, but for safety it should be installed on a machine with at least 1GB RAM.

Install:

These install instructions assume installation into the $HOME/bin directory. You can create this directory with mkdir $HOME/bin.

# Linux
$ wget "https://api.bintray.com/content/basespace/BaseSpaceCLI-EarlyAccess-BIN/latest/\$latest/amd64-linux/bs?bt_package=latest" -O $HOME/bin/bs
# Mac
$ wget "https://api.bintray.com/content/basespace/BaseSpaceCLI-EarlyAccess-BIN/latest/\$latest/amd64-osx/bs?bt_package=latest" -O $HOME/bin/bs
# Windows
$ wget  "https://api.bintray.com/content/basespace/BaseSpaceCLI-EarlyAccess-BIN/latest/\$latest/amd64-windows/bs.exe?bt_package=latest" -O bs.exe

You can change the installation location (for example, to preserve a previous installation) by changing the option after -O, e.g.

$ wget "https://api.bintray.com/content/basespace/BaseSpaceCLI-EarlyAccess-BIN/latest/\$latest/amd64-linux/bs?bt_package=latest" -O $HOME/bin/bs_v2

Note that your operating system might require that you change the file permissions to make the downloaded binary executable:

$ chmod u+x $HOME/bin/bs

Installing bscp

The copy command bscp is currently installed as a separate binary. Copy the file into a location in your $PATH with a bs- prefix to make it available via the bs cp command.

# Linux                                                              
$ wget https://api.bintray.com/content/basespace/BaseSpace-Copy-BIN/\$latest/linux/bscp?bt_package=develop -O $HOME/bin/bs-cp                                                   
# Mac                                                                
$ wget https://api.bintray.com/content/basespace/BaseSpace-Copy-BIN/\$latest/osx/bscp?bt_package=develop -O $HOME/bin/bs-cp                                                   
# Windows                                                            
$ wget https://api.bintray.com/content/basespace/BaseSpace-Copy-BIN/\$latest/windows/bscp.exe?bt_package=develop -O bs-cp.exe                                                           

Authenticate

$ bs auth

Note this command will authenticate to a default server located in AWS Virginia (https://api.basespace.illumina.com/)for localised authentication, please see Authenticate section.

Command Structure

The BaseSpace Sequence Hub CLI is based on a pattern of a command prefix and a tree of subcommands. Just as in docker there is docker run, docker build, docker exec and in git there is git pull, git push,git commit, the the BaseSpace Sequence Hub CLI has bs list, bs create and bs authenticate. Some commands have subcommands, such as bs list appsession and bs create project.

Each subcommand has its own set of options which can be accessed with --help or -h, as in the following example

$ bs auth --help                                                   
Usage:                                                              
    bs [OPTIONS] authenticate [authenticate-OPTIONS]               

Application Options:                                                  
    -V, --version   Display version                                         

Help Options:                                                         
    -h, --help      Show this help message                                     

[authenticate command options]                                      
-c, --config= Output to this BaseSpaceCLI configuration               
--force Overwrite existing configuration file, if it already exists   
--api-server= BaseSpace API hostname (default:                         
https://api.basespace.illumina.com) [$BASESPACE_API_SERVER]      
--scopes= List of scopes to authenticate with (default: READ GLOBAL, CREATE GLOBAL)                                                        
--timeout= Time in seconds before giving up (default: 120)  

Command Summary

CLI commands can use the following patterns:

  • Standalone commands—Single commands, e.g. auth. See Standalone Commands.

  • Actions on entities—Actions on entities, e.g. list appsession, import dataset, get biosample. These can be used in reverse, for example create project is equivalent to project create. See Entity Reference.

  • Actions on sub-entities—Actions on sub-entities, e.g. export workflow threshold or set project property. See Sub-Entity Reference.

The following table summarizes the CLI functions and commands. For detailed descriptions of commands, see Command Reference.

Function Available Commands
List and explore entities bs list, bs get, bs headers
List and download files within entities bs contents, bs download
Kill running apps bs kill appsession
Delete entities bs delete
Create biosamples and upload fastq files for them bs create biosample, bs upload dataset
Create analysis and configure analysis workflows from apps bs create workflow, bs workflow threshold ..., bs workflow dependency ...

Some command combinations have specific options. Review the usage message for more information. For example:

$ bs list appsession --help                                          
...                                                                   
[list command options]                                              
--exec-status= Filter by execution status                             
--input-biosamples= Filter by Input BioSample                         
...                                                                    

Specify API server and Access Token

Most BaseSpace Sequence Hub CLI commands require that you specify an API server to contact and an access token to authenticate against BaseSpace Sequence Hub. These can be specified in the following ways.

  • Config—Use -c / --config to specify the config name to use e.g. bs -c virginia to use the virginia.cfg configuration file. Configuration files are generated by bs auth and stored in $HOME/.basespace. The file format used here is shared with other BSSH command line tools such as BaseMount, so if you have a BaseMount config file you do not need to derive another one.
  • Command Line—Use --api-server / --access-token to specify the API server and access token on the command line. Warning: This method can store details in your command history and threaten the security of your token.
  • Environment variables—Provide access credentials by specifying the environment variables $BASESPACE_ACCESS_TOKEN and $BASESPACE_API_SERVER. These can also be loaded from a config using bs config load. This is useful for injecting access credentials into Docker containers.

For example:

# list projects based on the config with the name "myworkgroup"
$ bs -c myworkgroup list projects
# list projects by using environment variables
$ export BASESPACE_API_SERVER="https://api.euc1.sh.basespace.illumina.com/"
$ export BASESPACE_ACCESS_TOKEN="XXXX"
$ bs list projects
# list projects providing auth details on command line (not recommended!)
$ bs list projects --api-server=https://api.euc1.sh.basespace.illumina.com/ --access-token=XXXXX

Command Discovery

The CLI has been designed to help you discover commands. By typing the main bs command with --help you can see all the entities that are supported and all the actions that can be performed:

$ bs --help
...
Available actions:
  add           Add lane or workflow settings
  authenticate  Make an authentication request to BaseSpace (aliases: auth)
  clear         Clear lane or workflow settings
  content       Show all files in an entity (aliases: contents, dir)
  create        Create an entity
  delete        Move entities to your trash (aliases: rm, move-to-trash)
  download      Download files from an entity
  export        Export lane or workflow settings
  get           Get detailed information about an entity (aliases: info, inspect)
  import        Import lane or workflow settings
  kill          Abort entities
  list          List and filter entities (aliases: filter, list-all)
  set           Set properties in an entity
  upload        Upload files to an entity
  whoami        Get information about selected user/configuration

Available entities:
  application  View and manage applications (aliases: applications)
  appresult    View and manage appresults (aliases: appresults)
  appsession   View and manage appsessions (aliases: appsessions)
  biosample    View and manage biosamples (aliases: biosamples)
  config       View and manage installed confgurations (aliases: configs, configuration)
  dataset      View and manage datasets (aliases: datasets)
  lane         View and manage automated lane QC thresholds (aliases: lanes)
  project      View and manage projects (aliases: projects)
  run          View and manage runs (aliases: runs)
  workflow     View and manage workflow applications (aliases: workflows)

You can type the action by itself to see the entities to which that action can be applied:

$ bs list
Please specify one command of: application, appresult, appsession, attribute, biosample, config, dataset, project, property or run
$ bs import
Please specify one command of: lane or workflow

Similarly, you can specify an entity and see which actions are available:

$ bs application
Please specify one command of: get, header, launch or list
$ bs workflow
Please specify one command of: create, dependency or threshold

For each command combination, you can always run --help to see which sub-options are available and to get more details on the sub-commands:

$ bs appsession --help
...
Available commands:
  delete    Move an appsession to your trash (aliases: rm, move-to-trash)
  get       Get detailed information about an appsession (aliases: info, inspect)
  header    List all available fields for an appsession (aliases: headers)
  kill      Abort an appsession
  list      List and filter all appsessions (aliases: filter, list-all)
  property  Manage properties on an appsession (aliases: properties)

$ bs appsession kill --help
...
[kill command arguments]
  ID:                 IDs of the appsession to kill (multiple positional arguments)

bs set property --help
Available commands:
  appresult   set properties of an appresult (aliases: appresults)
  appsession  set properties of an appsession (aliases: appsessions)
  biosample   set properties of a biosample (aliases: biosamples)
  dataset     set properties of a dataset (aliases: datasets)
  project     set properties of a project (aliases: projects)
  run         set properties of a run (aliases: runs)

Command Reference

The following sections list the standalone commands and action commands grouped by entity, sub-entity, and action. For more information about how the commands work in practice, see the Worked Examples.

  • The Entities table lists the actions that can be applied to the entity, with some examples.
  • The Sub-Entities table lists the owning entity.
  • The Actions table lists the entities and sub-entities the action applies to, with some examples.

Entities and sub-entities

Some entities have sub-entities that have separate operations you can perform. These include:

  • Core Entities like projects, appsessions, datasets and runs which support properties on those entities.
  • The workflow entity has thresholds and dependencies with their own operations.
  • The biosample entity has requeue sub-entity used in communication with LIMS for high-throughput lab operations use-cases.

To use sub-entities, you need to provide switches to select the main entity (e.g. the project) as well as the sub-entity (e.g. the property name).

We recommend that you access sub-entities noun-first in this pattern bs <entity> <sub-entity> <verb> e.g.

# -n refers to the name of the project
$ bs project property all -n "MyProject"
# -n refers to the name of the project and --property-name refers to the name of the property within that project
$ bs project property get -n "My Project" --property-name="MyProperty"

You can also use this pattern to help explore the available commands, e.g.

$ bs project property
Please specify one command of: all, delete, get or set

For more information on using sub-entities, refer to the worked examples:

Standalone Commands

Authenticate

Authenticates against an instance of BaseSpace Sequence Hub, storing the API server used and the derived access token for use in future commands.

Example:

# by default, this will use the config name "default" and the API server https://api.basespace.illumina.com
$ bs auth                                                            
# authenticate against Frankfurt BaseSpace Sequence Hub instance and into config called "frankfurt"                                                         
$ bs auth -c frankfurt --api-server https://api.euc1.sh.basespace.illumina.com/   
# authenticate against Frankfurt BaseSpace Sequence Hub instance with custom scopes and timeout
$ bs auth -c frankfurt --api-server https://api.euc1.sh.basespace.illumina.com/ --scopes "READ GLOBAL,BROWSE GLOBAL" --timeout 10   

Additional options:

  • --scopes, to specify the scopes you obtain for your token.

  • --timeout, the amount of time to wait before giving up

Scopes

The following scopes are commonly used:

Actions

Scopes

Accessing data

READ GLOBAL
BROWSE GLOBAL

Creating data

CREATE GLOBAL

Deleting data

MOVETOTRASH GLOBAL
EMPTY TRASH

Working with apps MANAGE APPLICATIONS START APPLICATIONS

More information about scopes is available in the API documentation

For example:

# get a powerful token with lots of capabilities
$ bs auth --scopes "BROWSE GLOBAL,READ GLOBAL,CREATE GLOBAL,MOVETOTRASH GLOBAL,START APPLICATIONS,MANAGE APPLICATIONS"

Whoami

Returns information about the current user and token.

Example:

$bs -c configname whoami
+----------------+-------------------------------------------------+
| Name           | BaseSpace User                                  |
| Id             | 1234567                                         |
| Email          | basespaceuser@illumina.com                      |
| DateCreated    | 2015-01-16 15:31:22 +0000 UTC                   |
| DateLastActive | 2017-06-01 12:59:24 +0000 UTC                   |
| Host           | https://api.basespace.illumina.com              |
| Scopes         | ["READ GLOBAL" "CREATE GLOBAL" "BROWSE GLOBAL"] |
+----------------+-------------------------------------------------+

Entity Reference

Entity

Description

Available Actions

Sub-entities

Other notes

Example

Application

BaseSpace app

Get
Header
List

A token with MANAGE APPLICATIONS scope is needed for some operations

$ bs list applications --category=workflow

AppResult

Output from app

Content
Delete
Get
Header
List
Property

# get metadata about appresult with ID 123
$ bs get appresult -i 123
# show files in appresult with ID 123
$ bs contents appresults -i 123

AppSession

Representing an app during or after it has executed

Delete
Get
Header
Kill
List
Property

# get all the appsessions for a particular biosample
$ bs list appsessions --input-biosample=LP1000123

Biosample

Data and metadata for a biological sample

Get
Header
Import
List
Property

See [Worked Examples](cli-examples)

Config

Configuration files stored in $HOME/.basespace

List Load

Not a BSSH entity, but used purely by BSSHCLI

# Show all configs
$ bs config list ... # print config in form to be imported # as environment variables $ bs config load default

Dataset

Output from app that captures typed output

Delete
Get
Header
Import
List
Property
$ bs dataset get --id=ds.123
...
$ bs dataset list --input-biosample=LP123
...
$ bs dataset delete --preserve-metadata -n "DatasetName"
...

Project

Collection of other BSSH entities

Create
Delete
Get
Header
List
Property

# create a project by name and with a description
$ bs create project -n MyProject -d "This is a new project"
# list all project in csv format
$ bs list projects -f csv

Run

Sequencing run, sometimes called a flow cell

Content
Delete
Get
Header
List
Property

# get metadata about a specific run
$ bs run get -i 123
# get all the files in a run
$ bs run contents -i 123

Workflow

A type of BSSH app that can be configured with auto-launch dependencies and QC thresholds

Create
Threshold
Dependency

See [Worked Examples](cli-examples) for more details

Sub-Entity Reference

Entity

Description

Owning Entity

Example

Dependency Workflow
# export in json format the dependencies for workflow with ID 123
$ bs workflow dependency export -i 123
See [Worked Examples](cli-examples) for more details.
Lab Requeues The lab requeues that have been requested for a biosample Biosample
$ bs biosample requeue --requeue-type=NewLibrary -i 32853857
Property Key/value properties on entities AppResult AppSession Biosample Dataset Project
# show all properties for project with ID 123
$ bs project properties all -i 123
# get property with name PName from dataset with ID ds.123
$ bs dataset properties get -i ds.123 --property-name "Pname"

Threshold Workflow
# export in csv format the QC thresholds for workflow with ID 123
$ bs workflow threshold export -i 123

Action Reference

Action

Description

Available Entities

Notes

Example

All Get all properties for an entity Property
# get all properties for run with ID 123
$ bs run property all -i 123

Clear Used to remove workflow settings Workflow dependency
Workflow threshold
# remove all the workflow dependencies set for workflow 123
$ bs workflow dependency clear -i 123

Content Show the contents of a BSSH entity AppResult
Dataset
Run
# show all bam files from appresult with ID 123
$ bs appresult contents -i 123 --extension=bam

Delete Move an entity to trash AppResult
AppSession
Dataset
Project
Run
Items are not deleted until the trash is emptied
# delete project with name MyProject
$ bs delete project --name MyProject
# delete dataset with ID ds.123
$ bs delete dataset --id ds.123

Export Download BSSH data AppResult
Dataset
# download a tgz file from appresult 123
$ bs appresult export --id 291294 --extension=gz -o outfile.gz

Get Application
AppResult
Biosample
Dataset
Project
Property
Run
# get metadata about project with ID 123
$ bs get project --id 123

Headers Show available headers for an entity Application
AppResult
Biosample
Dataset
Project
Run
Useful to choose outputs for `bs list`
# Show available fields for an appsession
$ bs appsession headers

Import Create a new entity within BSSH Biosample
Dataset
Used to upload fastq files with
bs import dataset
This is equivalent to `bs upload sample` in the older version of the CLI
Kill Kill a running appsession Appsession
$ bs kill appsession 123

List List all entities Application
AppResult
Biosample
Dataset
Project
Property
Run
Has options to customize output:
  • Format (-f) which can be csv, yaml, json
  • Ability to specify headers with -F
  • Count items with --count
  • Richer templated output with --template
  • # list projects in yaml format
    $ bs list projects -f yaml
    # list appsessions with custom headers selected
    $ bs list appsessions -F Id -F ExecutionStatus
    # get the number of appresults $ bs list appresults --count
    # get data sets with just their IDs $ bs list data sets --terse

    Load Allow config variables to be loaded into environment Config
    Set Set a property or other setting Property

    CLI for the new data model

    What's New

    The updated CLI features has been designed to be faster, portable, and easier to maintain. We have added additional commands to support new features released in BaseSpace 5.0. These features include:

    • Workflows—BaseSpace Sequence Hub now includes support for apps that can be launched automatically and have an automated QC step, called analysis workflows. The new CLI includes commands to create, list and modify these workflows.

    • Datasets—BaseSpace Sequence Hub apps have the ability to create datasets, a new kind of entity that stores data and metadata in way that's easier to work with and query. The new CLI includes features to list and filter datasets.

    • Biosamples—The biosample is a new entity that groups together metadata and FASTQ data for a biological sample, including attaching analysis workflows to a biosample. The new CLI allows the listing and creation of biosamples and their metadata, including the creation of analysis workflows and inspecting lab requeues.

    The new CLI includes the following additional enhancements.

    • Richer options for listing entities and customising output

    • New get command to provide more details on a specific entity

    • Support for deleting appresults, appsessions, datasets, projects and runs.

    • Ability to inspect files within BaseSpace Sequence Hub entities using the contents command

    • Support for downloading data using the export command

    • More flexible install process. The CLI can be installed by downloading a single binary, which enables you to install the CLI in an environment where you do not have sudo privileges. Using a single binary file also allows simpler distribution of updated versions.

    • Support for all Linux platforms in addition to Mac and Windows.

    What's Not Included

    The new CLI does not include the following feature from the old CLI.

    • The command to wait for an app to finish (bs wait).
    • Options to ignore some validation rules on fastq upload (--allow-invalid-readnames)

    How Can I Get Support for CLI?

    CLI continues to be in alpha status and support is only available through the BaseSpace Sequence Hub Google Group.

    How Can I Run Both Versions of the CLI?

    If you want to try out the new CLI but retain the old CLI (e.g. for compatibility with existing scripts or because you need app launch) you can download the new CLI into a different name (e.g. bs2) and run both side-by-side. As an example (Linux):

    $ wget "https://bintray.com/basespace/BaseSpaceCLI-EarlyAccess-BIN/download_file?file_path=latest%2F0.9.4.459%2Famd64-linux%2Fbs" -O bs2