EdX

Serverless Data Processing with Dataflow: Operations (edX)

Offered by Google Cloud,
Serverless Data Processing with Dataflow: Operations (edX)

In the last installment of the Dataflow course series, we will introduce the components of the Dataflow operational model. In the last installment of the Dataflow course series, we will introduce the components of the Dataflow operational model. We will examine tools and techniques for troubleshooting and optimizing pipeline performance.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

We will then review testing, deployment, and reliability best practices for Dataflow pipelines. We will conclude with a review of Templates, which makes it easy to scale Dataflow pipelines to organizations with hundreds of users. These lessons will help ensure that your data platform is stable and resilient to unanticipated circumstances.
This course is part of the Google Cloud Data Engineer Learning Path Professional Certificate.

What you'll learn

  • Perform monitoring, troubleshooting, testing and CI/CD on Dataflow pipelines.
  • Deploy Dataflow pipelines with reliability in mind to maximize stability for your data processing platform.

Syllabus

  1. Introduction

This module covers the course outline

  1. Monitoring

In this module, we learn how to use the Jobs List page to filter for jobs that we want to monitor or investigate. We look at how the Job Graph, Job Info, and Job Metrics tabs collectively provide a comprehensive summary of your Dataflow job. Lastly, we learn how we can use Dataflow’s integration with Metrics Explorer to create alerting policies for Dataflow metrics.

  1. Logging and Error Reporting

In this module, we learn how to use the Log panel at the bottom of both the Job Graph and Job Metrics pages, and learn about the centralized Error Reporting page.

  1. Troubleshooting and Debug

In this module, we learn how to troubleshoot and debug Dataflow pipelines. We will also review the four common modes of failure for Dataflow: failure to build the pipeline, failure to start the pipeline on Dataflow, failure during pipeline execution, and performance issues.

  1. Performance

In this module, we will discuss performance considerations we should be aware of while developing batch and streaming pipelines in Dataflow.

  1. Testing and CI/CD

This module will discuss unit testing your Dataflow pipelines. We also introduce frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.

  1. Reliability

In this module we will discuss methods for building systems that are resilient to corrupted data and data center outages.

  1. Flex Templates

This module covers Flex Templates, a feature that helps data engineering teams standardize and reuse Dataflow pipeline code. Many operational challenges can be solved with Flex Templates.

  1. Summary

This module reviews the topics covered in the course.

Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Assembling and Testing a Quadcopter (edX) EdX
MathWorks

Assembling and Testing a Quadcopter (edX)

Learn the crucial skills of system-level integration, comprehensive testing, and collaboration to design and evaluate quadcopter systems effectively. Welcome to Integration and Testing of Quadcopter Systems, the final course in the Engineering Design and Simulation Program. This course is focused on integrating all the quadcopter subsystems into a cohesive system-level model and conducting thorough testing to ensure optimal performance.

Self Paced
Self-Paced
Introduction to Computer Science and Programming Using Python (edX) EdX
MIT,MITx

Introduction to Computer Science and Programming Using Python (edX)

An introduction to computer science as a tool to solve real-world analytical problems using Python 3.5. This course is the first of a two-course sequence: Introduction to Computer Science and Programming Using Python, and Introduction to Computational Thinking and Data Science. Together, they are designed to help people with no prior exposure to computer science or programming learn to think computationally and write programs to tackle useful problems.

Jan 24th 2024
5-12 Weeks
Automated Software Testing: Model and State-based Testing (edX) EdX
Delft University of Technology,DelftX

Automated Software Testing: Model and State-based Testing (edX)

Learn the advanced software testing techniques, tools, and best practices required to deliver high-quality software. Software testing gets a bad rap for being difficult, time-consuming, redundant, and above all - boring. But in fact, it is a proven way to ensure that your software will work flawlessly and can meet release schedules.

Self Paced
Self-Paced
Data Storage and Processing (edX) EdX
ITMO University,ITMOx

Data Storage and Processing (edX)

Master the culture of data representation, interpretation and outcomes evaluation. Learn the fundamentals of relational and NoSQL database management systems. Want to learn data processing and interpreting the result you’ve got? This course is for you! Get acquainted with preparing and analyzing large amount of data, as well as data storage fundamentals.

No sessions available
5-12 Weeks
Microservices and Serverless (edX) EdX
IBM

Microservices and Serverless (edX)

Design, develop, deploy, manage and secure applications and solutions on public, private or hybrid cloud platforms. This course will introduce you to 12-factor apps and microservices, concepts that emerged to help organizations work better and faster in a cloud-native manner. You’ll then learn about serverless computing—how it works, what value it brings, and what are specific serverless technologies. You’ll get hands-on with IBM Cloud Functions, a serverless platform on IBM Cloud that lets you develop serverless apps with ease. Finally, you will learn to build and deploy applications using container images on the code engine.

Self Paced
Self-Paced
Microservices, Serverless, OpenShift (edX) EdX
IBM

Microservices, Serverless, OpenShift (edX)

Learn about Microservices architecture and Serverless computing. Understand their benefits and the process for deployment. Practice using multiple tools in hands-on labs. Create a serverless web application and deploy as a Microservice on OpenShift and as static files on Cloud Object Storage. The demand for serverless is accelerating as organizations look to scale more quickly and efficiently. With the increase in cloud adoption, Microservices within the serverless stack are becoming more popular with faster deployments and greater flexibility.

Self Paced
Self-Paced
Synthetic Aperture Radar: Ecosystems (edX) EdX
University of Alaska Fairbanks,AlaskaX

Synthetic Aperture Radar: Ecosystems (edX)

This course will introduce the contributions of Synthetic Aperture Radar (SAR) remote sensing to the monitoring of Earth’s ecosystems. Learn how the weather-independence of SAR combined with its ability to penetrate into vegetation canopies make SAR an excellent information source to characterize vegetation structure, measure above-ground biomass, and analyze the change of vegetation long term and throughout the seasons.

Self Paced
Self-Paced