EdX

Serverless Data Processing with Dataflow: Develop Pipelines (edX)

Offered by Google Cloud,
Serverless Data Processing with Dataflow: Develop Pipelines (edX)

In this second installment of the Dataflow course series, we are going to be diving deeper on developing pipelines using the Beam SDK. In this second installment of the Dataflow course series, we are going to be diving deeper on developing pipelines using the Beam SDK. We start with a review of Apache Beam concepts.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

Next, we discuss processing streaming data using windows, watermarks and triggers. We then cover options for sources and sinks in your pipelines, schemas to express your structured data, and how to do stateful transformations using State and Timer APIs. We move onto reviewing best practices that help maximize your pipeline performance. Towards the end of the course, we introduce SQL and Dataframes to represent your business logic in Beam and how to iteratively develop pipelines using Beam notebooks.
This course is part of the Google Cloud Data Engineer Learning Path Professional Certificate.

What you'll learn

  • Review main Apache Beam concepts covered in DE (Pipeline, PCollections, PTransforms, Runner; reading/writing, Utility PTransforms, side inputs, bundles & DoFn Lifecycle)
  • Review core streaming concepts covered in DE (unbounded PCollections, windows, watermarks, and triggers)
  • Select & tune the I/O of your choice for your Dataflow pipeline
  • Use schemas to simplify your Beam code & improve the performance of your pipeline
  • Implement best practices for Dataflow pipelines
  • Develop a Beam pipeline using SQL & DataFrames

Syllabus

  1. Introduction

This module introduces the course and course outline.

  1. Beam Concepts Review

Review main concepts of Apache Beam, and how to apply them to write your own data processing pipelines.

  1. Windows, Watermarks Triggers

In this module, you will learn about how to process data in streaming with Dataflow. For that, there are three main concepts that you need to learn: how to group data in windows, the importance of watermark to know when the window is ready to produce results, and how you can control when and how many times the window will emit output.

  1. Sources & Sinks

In this module, you will learn about what makes sources and sinks in Google Cloud Dataflow. The module will go over some examples of Text IO, FileIO, BigQueryIO, PubSub IO, KafKa IO, BigTable IO, Avro IO, and Splittable DoFn. The module will also point out some useful features associated with each IO.

  1. Schemas

This module will introduce schemas, which give developers a way to express structured data in their Beam pipelines.

  1. State and Timers

This module covers State and Timers, two powerful features that you can use in your DoFn to implement stateful transformations.

  1. Best Practices

This module will discuss best practices and review common patterns that maximize performance for your Dataflow pipelines.

  1. Dataflow SQL & DataFrames

This modules introduces two new APIs to represent your business logic in Beam: SQL and Dataframes.

  1. Beam Notebooks

This module will cover Beam notebooks, an interface for Python developers to onboard onto the Beam SDK and develop their pipelines iteratively in a Jupyter notebook environment.

  1. Summary

This module provides a recap of the course.

Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Advanced Data Engineering (Coursera) Coursera
Duke University

Advanced Data Engineering (Coursera)

In this advanced course, you will gain practical expertise in scaling data engineering systems using cutting-edge tools and techniques. This course is designed for data scientists, data engineers, and anyone with a foundational understanding of data handling who desires to escalate their skills to handle larger, more complex datasets efficiently.

Nov 30th 2026
4 Weeks
Serverless Data Processing with Dataflow: Foundations (Coursera) Coursera
Google Cloud

Serverless Data Processing with Dataflow: Foundations (Coursera)

This course is part 1 of a 3-course series on Serverless Data Processing with Dataflow. In this first course, we start with a refresher of what Apache Beam is and its relationship with Dataflow. Next, we talk about the Apache Beam vision and the benefits of the Beam Portability framework. The Beam Portability framework achieves the vision that a developer can use their favorite programming language with their preferred execution backend.

Dec 14th 2026
2 Weeks
AI Skills for Engineers: Data Engineering and Data Pipelines (edX) EdX
Delft University of Technology,DelftX

AI Skills for Engineers: Data Engineering and Data Pipelines (edX)

Good data is central to effective AI applications. This course teaches the basics of data for AI, covering what data is needed, how to extract data from existing databases and basic data skills including setup of a Python notebook environment, basic data exploration and simple data visualizations.

Self Paced
Self-Paced
Google Cloud Big Data and Machine Learning Fundamentals (edX) EdX
Google Cloud

Google Cloud Big Data and Machine Learning Fundamentals (edX)

Data Analysts, Data Engineers, Data Scientists, and ML Engineers who are getting started with Google Cloud. This course introduces the Google Cloud big data and machine learning products and services that support the data-to-AI lifecycle. It explores the processes, challenges, and benefits of building a big data pipeline and machine learning models with Vertex AI on Google Cloud.

Self Paced
Self-Paced
Machine Learning Operations 2 (MLOps2-AWS): Data Pipeline Automation & Optimization using Amazon Web Services (AWS) (edX) EdX
Statistics.comX,Statistics.com

Machine Learning Operations 2 (MLOps2-AWS): Data Pipeline Automation & Optimization using Amazon Web Services (AWS) (edX)

Most data science projects fail. There are various reasons why, but one of the primary reasons is the challenge of deployment. One piece to the deployment puzzle is understanding how to automate your pipeline’s functions and continuously optimize its performance, which is why we developed this course - MLOp2s: Data Pipeline Automation & Optimization using Amazon Web Services (AWS).

Self Paced
Self-Paced
Parallel Programming in Java (Coursera) Coursera
Rice University

Parallel Programming in Java (Coursera)

This course teaches learners (industry professionals and students) the fundamental concepts of parallel programming in the context of Java 8. Parallel programming enables developers to use multicore computers to make their applications run faster by using multiple processors at the same time. By the end of this course, you will learn how to use popular parallel Java frameworks (such as ForkJoin, Stream, and Phaser) to write parallel programs for a wide range of multicore platforms including servers, desktops, or mobile devices, while also learning about their theoretical foundations including computation graphs, ideal parallelism, parallel speedup, Amdahl's Law, data races, and determinism.

Nov 23rd 2026
4 Weeks
Network Analysis in Systems Biology (Coursera) Coursera
Icahn School of Medicine at Mount Sinai

Network Analysis in Systems Biology (Coursera)

An introduction to data integration and statistical methods used in contemporary Systems Biology, Bioinformatics and Systems Pharmacology research. The course covers methods to process raw data from genome-wide mRNA expression studies (microarrays and RNA-seq) including data normalization, differential expression, clustering, enrichment analysis and network construction. The course contains practical tutorials for using tools and setting up pipelines, but it also covers the mathematics behind the methods applied within the tools.

Nov 23rd 2026
5-12 Weeks