gcp-data-pipe

gcp data pipeline — terragrunt-managed ingest and transform with dataproc, bigquery, pub/sub.

draft infraterraformgcp

year

2025

role

engineer

stack

hcl · terragrunt · gcp · bigquery · dataproc · pub/sub

link

open

production-ready data pipeline on google cloud using terragrunt for infrastructure as code. modular structure with numbered directories — iam, storage, bigquery, pub/sub, dataproc.

architecture

source project with read-only bigquery access
processing project with all compute resources and data writes
components: dataproc cluster, bigquery, pub/sub, gcs, data catalog, service accounts

structure

01-iam/ — service accounts and iam bindings
02-storage/ — gcs buckets
03-bigquery/ — datasets and tables
04-pubsub/ — topics and subscriptions
05-dataproc/ — cluster config

all sensitive values in .env (git-ignored). no pii or secrets in code.