Google Cloud Storage (GCS) Permissions

Configuring Google Cloud Storage (GCS) permissions establishes secure access between Publisher and your cloud storage resources. These permissions define how Publisher can interact with your buckets and objects, using service accounts and custom roles to ensure controlled access.

This guide outlines the minimum permissions needed to connect Publisher to specific GCS resources, including credential creation and bucket-level access configuration.

Concepts

• Service Account (SA): This is an identity generated in Google Cloud that can be used to interact with GCP services. Each SA has a unique email (identity) and one or more JSON keys.

• Data Project: A project that houses the target dataset.

• Quota project: A project that you would like to bill BigQuery to and consume quota resources.

• Permission: Granular actions that a user can perform on a given resource.

• Role: A collection of permissions typically needed for specific interactions against one or more resources: “Data Viewer,” “Storage Reader,” etc.

Minimal Configuration

This setup represents the minimal permissions needed to interact with a SPECIFIC BigQuery DataSource without creating custom roles. It conceptually segregates the Quota Project and Data Project, but they can be the same project.

Creating Credentials

Create a new Service Account (SA) in any project.

For more information on creating a new service account within GCS, please refer to Google's IAM Guide.

gcloud iam service-accounts create publisher-connectors \ 
    --display-name="Publisher Data Connections"

Generate a new JSON key and download it.

gcloud iam service-accounts keys create \
  publisher-connectors-key.json \
--iam-account=publisher-connectors@${PROJECT}.iam.gserviceaccount.com

Creating a Custom Role in Data Project

Create a role that has permission to list buckets:

gcloud iam roles create "StorageBucketLister" \
  --project=${PROJECT} \
  --title="Storage Bucket Lister" \
  --description="Required for Publisher to list buckets" \
  --permissions="storage.buckets.list,storage.buckets.get" \
  --stage=GA

Assign that role to the SA you created in "Creating Credentials":

gcloud projects add-iam-policy-binding ${PROJECT} \ 
--member="serviceAccount:publisher-connectors@${PROJECT}.iam.gserviceaccount.com" \
--role="projects/${PROJECT}/roles/StorageBucketLister"

Data Project

For each bucket that you want to connect to Publisher, assign the Storage Object Viewer policy to the service account:

gcloud storage buckets add-iam-policy-binding gs://${BUCKET} \
--member="serviceAccount:publisher-connectors@${PROJECT}.iam.gserviceaccount.com" \
--role=roles/storage.objectViewer \
--project=${PROJECT}