Engines

In Workbench, an engine is an external service that executes workflows written in the Workflow Description Language (WDL). The engine takes a WDL and generates individual task definitions and subsequently dispatches them to an execution service that performs the actual running of the tasks.

Workbench supports multiple engines, each with its own configuration and capabilities. The engine you choose depends on your requirements, such as the cloud provider you use, the tools you need to run, and the scale of your workflows. You can connect an engine running in your own environment to your Workbench account to use it for workflow execution Workbench to an engine running in their own environment, and then Workbench will use the engine to submit and monitor workflow runs.

A Workflow Engine is an application that accepts a workflow script written in WDL, generates individual task definitions, and dispatches them to an execution service.

WDL is an open, community-driven workflow language stewarded by the OpenWDL organization. It is designed to be easy to read and write, and to be able to scale from simple, single tool command lines to complex workflows with chained tools and parallel execution. Workflow engines that support WDL, like Cromwell and miniWDL, are capable of interpreting the WDL script, scheduling the tasks for execution, handling task dependencies, and managing the resources required for the workflow execution.

Supported clouds and On-Premises Environments

Workbench can run workflows on engines in many environments, including on major commercial clouds and on-premises. We provide detailed instructions on setting up engines for:

• Amazon Web Services

• Google Cloud Platform

• Microsoft Azure

• On Premises

Viewing Engines

To view the engines that are currently available in Workbench, navigate to the Settings Page or list them using the Command Line Interface. From the settings page, you can search for engines by using the search bar at the top of the page

• Click on a specific row to view more details about the engine, including the engine state, engine configuration, and engine parameters.

The Default engine

The first time you create an engine in Workbench, it will be set as the default engine. The default engine is prepopulated whenever you submit a workflow run, but you can change the engine for a specific run at any time. Additionally, if you submit a workflow run from the CLI, the default engine will be used if you omit the --engine flag.

To change the default engine:

1. Navigate to the Settings Page and hover over the engine you want to set as the default

2. On the right side of the engine row, action items will appear. Click on the Set as Default star button to set the engine as the default.

You cannot delete the default engine. If you want to delete the default engine, you must first set another engine as the default.

Actions

When you view the list of engines in Workbench, hovering over any row will review a list of actions that can be taken

1. Edit Engine:

   • Clicking on the Edit Engine pencil icon button will allow you to update the engine configuration, engine name, and engine parameters

2. Set as Default:

   • Clicking on the Set as Default star icon button will set the engine as the default engine in Workbench

3. Copy Engine Id:

   • Clicking on the Copy Engine Id duplicate icon button will copy the engine ID to your clipboard. This is useful when you need to reference the engine ID in the CLI or API

4. Delete Engine:

   • Clicking on the Delete Engine trashcan icon will remove the engine from Workbench. This action is irreversible.

   • If the engine is the default engine, you will be prompted to set another engine as the default before deleting the engine

   • If the engine is currently running workflows, they will be marked as failed in workbench however the workflows will continue to run until they are completed

   • Deleting an engine does NOT delete the underlying infrastructure or resources that the engine is connected to. You must manually delete these resources in the cloud provider console

   • Previous runs for the engine will still be visible in the Workbench UI, but you will not be able to submit new runs to the engine, and you will not be able to view log information for the runs

Engine Status

Engines that are connected to Workbench are continuously monitored to ensure that they are available for running workflows. The state of the engine is determined by checking various used by workbench when interacting with the engine, such as:

• Connectivity: The ability to reach the API endpoint

• Credentials: The ability to authenticate with the credentials provided

• Permissions: Check that the credentials provided have the necessary permissions to run workflows or communicate with the engine

• Storage: The ability to read and write (if applicable) data to the storage location specified in the engine configuration

• Logs: The ability to stream logs from the engine through workbench

Each engine will only run the necessary checks that make sense for the engine type and configuration. If any of these checks fail, the engine state will be set to ERROR and the engine will not be available for running workflows until the issue is resolved. If all checks pass, the engine state will be set to ONLINE and the engine will be available for running workflows.

You can view the state of the engine in the Workbench UI by looking at the engine list on the settings page. An engine in the ONLINE state will have a green dot next to the engine name, while an engine in the ERROR state will have a red. For more information on troubleshooting engine issues refer to each engine's configuration guide.