Google Cloud Storage (GCS) Permissions

Concepts

• Service Account (SA): This is an identity generated in Google Cloud that can be used to interact with GCP services. Each SA has a unique email (identity) and one or more JSON keys.

• Data Project: A project that houses the target dataset.

• Quota project: A project that you would like to bill BigQuery to and consume quota resources.

• Permission: Granular actions that a user can perform on a given resource.

• Role: A collection of permissions typically needed for specific interactions against one or more resources: “Data Viewer,” “Storage Reader,” etc.

Minimal Configuration

This setup represents the minimal permissions needed to interact with a SPECIFIC BigQuery DataSource without creating custom roles. It conceptually segregates the Quota Project and Data Project, but they can be the same project.

Creating Credentials

Create a new Service Account (SA) in any project.

For more information on creating a new service account within GCS, please refer to Google's IAM Guide.

gcloud iam service-accounts create publisher-connectors \ 
    --display-name="Publisher Data Connections"

Generate a new JSON key and download it.

gcloud iam service-accounts keys create \
  publisher-connectors-key.json \
--iam-account=publisher-connectors@${PROJECT}.iam.gserviceaccount.com

Creating a Custom Role in Data Project

Create a role that has permission to list buckets:

gcloud iam roles create "StorageBucketLister" \
  --project=${PROJECT} \
  --title="Storage Bucket Lister" \
  --description="Required for Publisher to list buckets" \
  --permissions="storage.buckets.list,storage.buckets.get" \
  --stage=GA

Assign that role to the SA you created in "Creating Credentials":

gcloud projects add-iam-policy-binding ${PROJECT} \ 
--member="serviceAccount:publisher-connectors@${PROJECT}.iam.gserviceaccount.com" \
--role="projects/${PROJECT}/roles/StorageBucketLister"

Data Project

For each bucket that you want to connect to Publisher, assign the Storage Object Viewer policy to the service account:

gcloud storage buckets add-iam-policy-binding gs://${BUCKET} \
--member="serviceAccount:publisher-connectors@${PROJECT}.iam.gserviceaccount.com" \
--role=roles/storage.objectViewer \
--project=${PROJECT}