How to Authenticate gsutil Using JSON Service Account Key for Google Cloud Storage Access


1 views

When working with Google Cloud Storage (GCS), service account JSON keys provide a more secure and manageable authentication method compared to traditional access keys. The JSON key contains all necessary credentials in a structured format:

{
  "type": "service_account",
  "project_id": "your-project",
  "private_key_id": "unique_id",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...",
  "client_email": "service-account@project.iam.gserviceaccount.com",
  "client_id": "client-id-number",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account%40project.iam.gserviceaccount.com"
}

There are two primary methods to configure gsutil with your JSON key:

Method 1: Using Environment Variable

The most straightforward approach is setting the GOOGLE_APPLICATION_CREDENTIALS environment variable:

# Linux/macOS
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"

# Windows
set GOOGLE_APPLICATION_CREDENTIALS="C:\path\to\key.json"

After setting the variable, gsutil commands will automatically use this credential:

gsutil ls gs://your-bucket-name

Method 2: Configuring .boto File

For more persistent configuration, edit the .boto configuration file:

[Credentials]
gs_service_key_file = /path/to/key.json
gs_service_account_name = service-account@project.iam.gserviceaccount.com

When using JSON keys with gsutil, you might encounter:

  • Permission Errors: Ensure the service account has proper IAM roles (Storage Admin or Storage Object Admin)
  • Key Format Issues: Verify the JSON structure and that the private key contains proper line breaks
  • Expired Keys: Service account keys don't expire by default but can be rotated

For scripts requiring temporary access, generate short-lived credentials:

gcloud auth activate-service-account --key-file=key.json
gcloud config set project your-project-id

This creates a temporary authentication token valid for one hour.

When working with JSON service account keys:

  • Restrict file permissions (chmod 400 key.json)
  • Never commit keys to version control
  • Consider using Workload Identity Federation for production environments
  • Rotate keys regularly using gcloud iam service-accounts keys

When working with Google Cloud Storage buckets via gsutil, the standard documentation often focuses on traditional key/secret pairs. However, many modern Google Cloud setups use JSON key files for service account authentication. Here's how to properly configure this:

First, ensure you have the Google Cloud SDK installed and initialized:

# Install Google Cloud SDK
curl https://sdk.cloud.google.com | bash
exec -l $SHELL
gcloud init

There are two primary methods to use your JSON key:

Method 1: Set Default Application Credentials

Export the environment variable pointing to your key file:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
gsutil ls gs://your-bucket-name

Method 2: Configure gcloud with the Service Account

gcloud auth activate-service-account --key-file=key.json
gcloud config set account client_email_from_json_file@project.iam.gserviceaccount.com
gsutil ls gs://your-bucket-name

Check your active authentication:

gcloud auth list
gsutil config -a

If you encounter permission errors, ensure:

  • The service account has proper IAM permissions (Storage Admin or similar)
  • The JSON file hasn't expired or been revoked
  • There are no extra characters or formatting issues in the JSON

For projects requiring multiple service accounts:

gcloud auth activate-service-account account1@project.iam.gserviceaccount.com --key-file=key1.json
gsutil -i account1@project.iam.gserviceaccount.com ls gs://bucket1

gcloud auth activate-service-account account2@project.iam.gserviceaccount.com --key-file=key2.json
gsutil -i account2@project.iam.gserviceaccount.com ls gs://bucket2