Adding a Workflow

Adding a Workflow

Workbench provides thousands of community authored workflows through its integrations with external workflow repositories that you can run with your Workbench account. However, there are many situations where the existing workflows are not sufficient.

Adding a private workflow provides a number of key benefits:

1. Custom Logic Definition:

   • Allows you to define custom logic that may not be available in publicly accessible repositories.

2. Rapid Iteration:

   • Enables rapid iteration on your workflows before making them publicly available.

3. Compliance History:

   • Helps meet compliance needs by maintaining a history of all past versions of a workflow.

4. Secure Storage:

   • Provides a secure platform to store workflows, especially those containing potentially sensitive information.

   • (Note: Workflows should never contain PHI - Protected Health Information.)

5. Proprietary Development:

   • Facilitates the development of proprietary workflows without exposing the source code.

Steps to add a Workflow

To begin, navigate to the Workflows page from the Workbench user interface and click on the "Add Workflow" button in the top right corner.

workflows-list

You will be navigated to a new page where you can upload the relevant workflow files and folders by dragging and dropping them onto the screen, or by opening a file picker. When you select a single file, only the filename will be preserved and the preceding path will be stripped away. This approach works well if you have only a single file to upload or if all the files are located in the same directory.

When you have a more complex workflow with a nested directory structure, you should choose to upload your files as a directory, or combine all files into a single zip archive and upload the zip file. This approach will ensure workflow imports resolve appropriately by preserving the relative path structure.

Once you have finished selecting and adding all files, click the "Upload" button in the top right hand corner. This will take you to the next screen where you can edit workflow metadata.

workflows-file-upload

When adding a workflow, there is some required metadata that will help make your interaction with it in Workbench easier.

1. Name:

   • Provide your workflow with a meaningful name so that you can easily locate it in the future.

2. Version:

   • Set the name for the first version of your workflow. We recommend using semantic versioning, however any meaningful name may be used. The name will be converted into an ID by removing illegal characters and spaces.

3. Entrypoint:

   • The entrypoint is the start of the workflow and should point to one of the Workflow Description Language (WDL) files that you uploaded. When the workflow is executed, the entrypoint represents the start of execution and should have a single WDL workflow in it.

   • Simply start typing in the entrypoint input and you will be presented with a dropdown showing you a filtered list of the files that you uploaded.

4. Description:

   • The description is an optional field that provides additional helpful instructions, notes, or any other content you would like. It supports markdown and is editable after the workflow has been created.

Once you have filled in all of the metadata for the workflow, click the "Save" button in the top right hand corner. You will be navigated to the workflow version page, which displays all the versions of the workflow you just created.

wofklow-save

Additional Workflow Metadata

Workflow Naming

When adding a workflow through the user interface, a workflow name is mandatory. However, when using the command line interface (CLI), the name can be omitted. In cases where the workflow name is not provided in the CLI, it will be extracted directly from the WDL file.

For instance, consider the following workflow:

workflow hello_world {

}

the value hello_world will be extracted and set as the workflow name, if no other name was provided.

Workflow Authors

Adding Authors to Your Workflow

When adding a workflow in Workbench, you have the option to specify authors directly within the workflow definition, ensuring that essential metadata remains associated with your workflow. In the main workflow's meta section, you can provide either a single author or a list of authors.

Here's how you can add a single author:

meta {
  author: "George Clement"
}

And for multiple authors:

meta {
  authors [
    "George Clement",
    "Suzy Shrier"
  ]  
}

If no author is provided directly in the workflow, Workbench will automatically set you as the author.

Troubleshooting Workflow Creation

Workflow syntax is not correct

Only workflows which are syntactically correct will be saved to your account. Prior to saving a workflow, it is recommended that you use a tool to perform static type checking and syntax validation. One such tool is MiniWDL, a command line utility and execution engine for WDL. If running the miniwdl check command succeeds, and you are still not seeing your workflow in Workbench, please contact DNAstack support.

miniwdl check path/to/your/workflow.wdl

Missing files

When adding files in the User Interface of Workbench, make sure all WDL files referenced in import statements are properly uploaded with the appropriate paths. If you have a complex workflow structure, you should upload your workflow files as a zip folder to ensure the relative paths are preserved.

Wrong entrypoint

The entrypoint tells Workbench which file contains the main workflow code. If the entrypoint is wrong, validation will like fail, or the resulting workflow that is submitted to an execution engine will be incorrect.