Storage

Overview

The storage group of commands provide Workbench users the ability to add read-only storage accounts (i.e. buckets and containers) to Workbench that can be used for powering an interactive file picker, identifying samples originating from a specific platform, and identifying sequencing instruments.

Managing Storage Accounts

Storage Add

Add a new storage account that Workbench can use to list and view blobs, buckets, etc. The configuration needed is specific to each cloud, so we add a sub command to simplify the configuration from the CLI’s perspective and limit the choices that are presented to the end user.

If successful the command prints the storage account JSON to the console.

For AWS:

# AWS Flavor
omics workbench storage add aws STORAGE_ID
  --name
  --access-key-id 
  --secret-access-key
  --region
  --bucket
  [--platform-type pacbio|element]
  [--path /path-in-bucket]

For GCP:

# GCP Flavor
omics workbench storage add gcp STORAGE_ID
  --name
  --service-account /path/to/file
  [--project-id ]
  --bucket
  --region
  [--platform-type pacbio|element]
  [--path /path-in-bucket]

For Azure:

# Azure Flavor 
omics workbench storage add azure STORAGE_ID
  --name
  [--sas | --access-key | (--tenant-id | --client-id | --client-secret)
  --container
  --storage-account-name
  --region
  [--platform-type pacbio|element]
  [--path /path-in-bucket]

Positional Parameters

Positional Params
Description
Example

STORAGE_ID

The user defined ID of the storage account to add.

my-gcp-account

Flag Parameters

Flag
Description
Example

--name

A human readable name for the account.

My GCP Account

--platform-type

An optional parameter that indicates the storage account should be indexed as it contains data for a specific sequencing platform

--platform-type pacbio

--path

An optional path within the current bucket to index

AWS

--access-key-id

The access Key ID to authenticate to AWS with.

AKIA5USG4GPGUBM7FPOW

--secret-access-key

The secret value of the access key.

…

--region

The AWS region the access key is for.

us-east-1

--bucket

The name of the bucket in S3.

s3://bucket_name

GCP

--service-account

The path to the service account that has the appropriate permissions to list and read from buckets.

/path/to/service-account.json

--project-id

Optionally set the project ID to use for API operations. This is typically used for billing purposes.

If this is not defined, then the project from the service account will be used.

striking-effort-817

--bucket

The name of the bucket in GCP.

gs://bucket_name

Azure

--sas

An azure shared access signature that grants access to a limited set of resources.

If the SAS is defined, no other access method is used.

…

--access-key

The access key for the given storage account.

If the access key is defined, no other access method is used.

…

--tenant-id

If using OAuth2, provide a tenant ID to authenticate against.

If the TenantID is defined, OAuth2 will be used and no other access method is used.

D54BD769-8734-482E-B331-F8E9A787FF9D

--client-id

If using OAuth2, provide the Client ID to use for authentication.

If the Tenant ID is defined, OAuth2 will be used and no other access method is used.

D54BD769-8734-482E-B331-F8E9A787FF9D

--client-secret

If using OAuth2, provide the Client ID to use for authentication.

If the Tenant ID is defined, OAuth2 will be used and no other access method is used.

…

--container

The name of the container in Azure.

<container>

--storage-account

The name of the storage account in Azure

<storage-account>

--region

The region the Storage account is located in

Storage List

List all of the storage accounts that are connected to Workbench for a given namespace.

omics workbench storage list
  [--max-results]
  [--page]
  [--page-size]
  [--sort]
  [--provider]

Flag Parameters

Flag

Description

Example

--max-results

Optional flag to limit the total number of results to return.

--max-results 10

--page

Optional flag to set the page of results to retrieve from the server. If not defined then the first page will be retrieved.

Unless used in conjunction with --max-results, all results will be listed following the requested page.

--page 4

--page-size

The number of results to return in a single page. Modifying this value may have impacts on performance.

Set a small page size if you have a lower connection or a larger page size (up to 1000) if you have a fast connection.

The default is 100.

--page-size 50

--sort

An optional flag to define how results are sorted by allowing the user to specify one or more columns and directions to sort by.

The value should be in the form:

  • column(:direction)

The user can define multiple columns to sort by (if they are supported) simply by separating the sorts with a semicolon.

The Direction can be defined by the values: asc AND desc

--sort provider:desc

--provider

An optional flag to filter the results by the given cloud provider.

--cloud gcp

Storage Update

Update an existing storage account configuration:

# AWS Flavor
omics workbench storage update aws STORAGE_ID
  --name
  --access-key-id 
  --secret-access-key
  --region
  --bucket

# GCP Flavor
omics workbench storage update gcp STORAGE_ID
  --name
  --service-account /path/to/file
  --bucket
  --region
  [--project-id ]

# Azure Flavor of bucket
omics workbench storage update azure STORAGE_ID
  --name
  --container
  --region
  [--sas | --access-key | (--tenant-id | --client-id | --client-secret)

Positional Parameters

Positional Params

Description

Example

STORAGE_ID

The user defined the ID of the storage account to update.

my-gcp-account

Flag Parameters

Flag

Description

Example

--name

A human readable name for the account.

My GCP Account

AWS

--access-key-id

The access Key ID to authenticate to AWS with.

AKIA5USG4GPGUBM7FPOW

--secret-access-key

The secret value of the access key.

…

--region

The AWS region the access key is for.

us-east-1

--bucket

The name of the bucket in S3.

s3://bucket_name

GCP

--service-account

The path to the service account that has the appropriate permissions to list and read from buckets.

/path/to/service-account.json

--project-id

Optionally set the project ID to use for API operations. This is typically used for billing purposes.

If this is not defined, then the project from the service account will be used.

striking-effort-817

--region

The region of the bucket

--bucket

The name of the bucket in GCP.

gs://bucket_name

Azure

--sas

An azure shared access signature that grants access to a limited set of resources.

If the SAS is defined, no other access method is used.

…

--access-key

The access key for the given storage account.

If the access key is defined, no other access method is used.

…

--tenant-id

If using OAuth2, provide a Tenant ID to authenticate against.

If the Tenant ID is defined, OAuth2 will be used and no other access method is used.

D54BD769-8734-482E-B331-F8E9A787FF9D

--client-id

If using OAuth2, provide the Client ID to use for authentication.

If the Tenant ID is defined, OAuth2 will be used and no other access method is used.

D54BD769-8734-482E-B331-F8E9A787FF9D

--client-secret

If using OAuth2, provide the Client ID to use for authentication.

If the Tenant ID is defined, OAuth2 will be used and no other access method is used.

…

--container

The name of the container in Azure.

https://<storage-account-name>.blob.core.windows.net/<container-name>/<blob-name>

--region

The region of the container

Storage Delete

Delete a storage account from the API

omics workbench storage delete STORAGE_ID

Positional Parameters

Positional Params

Description

Example

STORAGE_ID

The user defined ID of the storage account to delete

my-gcp-account

Platform Management

Storage Platforms Add

Annotate a bucket with a sequencing platform, adding semantic meaning to the files that exist within it.

omics workbench storage platforms add PLATFORM_ID
  --name
  --storage-id
  --platform [PACBIO]
  --path

Positional Parameters

Parameter

Description

Example

PLATFORM_ID

The user-defined ID of the platform to add.

platform-123

Flag Parameters

Flag

Description

Example

--name

A human readable name for the platform.

My Platform

--storage-id

The ID of the storage account associated with the platform.

my-gcp-account

--platform

The sequencing platform associated with the platform.

pacbio

--path

The path in the storage account where platform data is located within a specific bucket.

prefix/path/in/bucketgs://my-platform-data/

Storage Platforms List

List all of the annotated platforms.

omics workbench storage platforms list
  [--max-results]
  [--page]
  [--page-size]
  [--sort]
  [--storage-id]

Flag Parameters

Flag
Description
Example

--max-results

Limit the total number of results returned.

--max-results 10

--page

Specify the page of results to retrieve.

--page 2

--page-size

The number of results to return in a single page.

--page-size 50

--sort

Sort results by column and direction (ASC or DESC).

--sort name:ASC

–storage-id

Required: the storage ID to retrieve platforms from

–storage-id

Storage Platforms Delete

Delete the annotated platform.

omics workbench storage platforms delete <PLATFORM_ID>

Positional Parameters

Parameter

Description

Example

PLATFORM_ID

The ID of the platform to delete.

platform-123

Storage Platforms Describe

Describe one or more platform connections.

omics workbench storage platforms describe <PLATFORM_ID> [...PLATFORM_IDS]

Positional Parameters

Parameter

Description

Example

PLATFORM_ID

The ID of the platform to describe.

platform-123

PreviousInstrumentsNextSamples

Last updated