Data Streaming on Google Cloud Platform Training
Level
IntermediateDuration
24h / 3 daysDate
Individually arrangedPrice
Individually arrangedData Streaming on Google Cloud Platform Training
Discover a practical approach to designing, building, and maintaining real-time data pipelines on Google Cloud Platform. During the training, you will learn how to make informed decisions about choosing between batch processing, micro-batching, and streaming, and you will learn the entire process of building a solution – from preparing the data source and CDC, through streaming and data processing in Dataflow, to monitoring, SLOs, and operational procedures. The training is practical and based on a project that participants develop step by step during subsequent blocks. You will learn the key topics related to CDC, Datastream, Dataflow, and Apache Beam, and you will also learn how to design pipelines that are resilient to duplicates, retries, schema changes, and late data. After completing the training, you will be able to consciously design data flow architecture, build streaming pipelines, and prepare them for stable operation and monitoring in a production environment.
Data Engineers-
Data Engineers working with Google Cloud Platform
-
Developers working with data processing and integration systems
-
Data Architects
What You Will Learn
- Making decisions regarding the choice of batch processing, micro-batching, and streaming
- Designing data pipeline architecture and defining SLOs
- Preparing data sources for Change Data Capture
- Configuring and using Datastream
- Building streaming pipelines using Dataflow and Apache Beam
- Working with windows, watermarks, and late data
- Implementing data deduplication and enrichment
- Testing pipelines and preparing them for production
- Monitoring pipelines and creating runbooks and operational procedures
Training Program
Day 1
Module I: Fundamentals + architectural decisions + project kickoff
Objective: establish a common language, selection criteria, and a plan for “what we are building and why.”
- Batch vs streaming: trade-offs (latency/cost/complexity/maintenance) + when not to use streaming
- Delivery models: CDC vs event sourcing vs micro-batch vs near real-time (in practice for DE)
- GCP “building blocks” and selection criteria (without going into tool details): Pub/Sub, Dataflow, Cloud Run, Datastream, (Debezium/Kafka as context)
- Project (start): case selection + SLO definition (lag/latency, e.g. p95), assumptions, risks, cost budget, “done” criteria
Output: architecture card (variants + rationale) + initial SLOs and risks.
Module II: CDC from the source side – “for DBA” checklists + database lab
Objective: ensure that CDC is feasible and safe for production.
- CDC mechanics on sources: MySQL binlog / SQL Server CDC/CT / Oracle redo logs – what needs to be enabled and the consequences
- Permissions and organizational requirements: “what the admin/DBA must do” checklist + typical blockers
- Practical issues: schema changes, missing PK, type collisions, slow OLTP (“we don’t break production”)
- Lab: preparing the CDC source (MySQL or SQL Server) + “does CDC work?” test + what we log/monitor on the source side
Output: ready-to-use implementation checklist + “source constraints” findings.
Day 2
Module III: Datastream – essentials only + proper landing + SLO measurement
Objective: use Datastream as a familiar component and set up a “clean input” for downstream.
- Datastream in a nutshell: what it does, what it does not do; how to read statuses, lag, throughput
- Minimal configuration: connection profiles, object selection, pitfalls (most common causes of lag/errors)
- Datastream → GCS/BQ: landing (formats/naming), “bronze layer” principles for downstream
- Lab: launching the stream + first data in GCS/BQ + baseline SLO (lag/latency) + monitoring metrics
Output: working stream + landing principles + baseline SLO.
Module IV: Dataflow/Beam streaming – windows, watermarks, late data (practically)
Objective: build a stable streaming pipeline and understand what breaks without windows/watermarks.
- Beam streaming fundamentals: unbounded sources, continuous processing model
- Windows, watermarks, late data: examples of errors (duplicates, “gaps”, incorrect aggregations) and fixes
- In-flight transformations: mapping to the BigQuery schema, basics of schema evolution “without surprises”
- Lab: input (GCS/PubSub – depending on the variant) → Dataflow → BQ: simple transformation + latency control
Output: streaming → BQ pipeline, working on project data.
Day 3
Module V: Dataflow patterns – deduplication, enrichment, micro-batch + testability
Objective: deliver “production readiness”: resilience to retries, repetitions, and real-world data.
- Deduplication and exactly-once-ish: retries, repetitions, lack of PK – key and window strategies
- Enrichment: lookup to BigQuery/HTTP/cache – when it makes sense, when it kills costs/latency
- Micro-batch as a cost/latency compromise (near real-time without a platform)
- Testability: mini test/regression suite for the pipeline (what is really worth testing)
- Lab: deduplication + enrichment + writing to BQ + tests/regressions (minimal, sensible set)
Output: “hardened” pipeline + tests + micro-batch decision (yes/no) with rationale.
Module VI: Operations/SLO + monitoring + runbook + alternatives and project completion
Objective: close the topic operationally (SLOs, alerts, runbook), plus consciously choose alternatives/hybrid approaches.
- Orchestration: Composer (Airflow) – running Dataflow, health checks, batch+stream dependencies
- Monitoring and alerting: lag/throughput, CDC errors, costs (Datastream+Dataflow), “source vs GCP issue”
- Runbooks: safe restarts, escalation procedures, definition of “done”, emergency checklists
- Alternatives/hybrid (briefly, decision-oriented): Pub/Sub + Cloud Run + Dataflow; API ingestion; Debezium/Kafka/Confluent – when it makes sense vs overkill
- Project finale: end-to-end demo + dashboards + runbook for scenarios (lag increases / schema changes / retry storm)
Output: ready-to-use runbook + monitoring/SLO + architectural decisions “what we are leaving out”.
