EdX

Spark, Hadoop, and Snowflake for Data Engineering (edX)

Offered by AI (Pragmatic AI Labs),
Spark, Hadoop, and Snowflake for Data Engineering (edX)

Gain the skills for building efficient and scalable data pipelines. Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) and learn how to optimize them using Python, PySpark, and MLflow.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

In this course, you will:

  • Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) and learn how to optimize and manage them
  • Delve into Databricks, a powerful platform for executing data analytics and machine learning tasks
  • Hone your Python data science skills with PySpark
  • Discover the key concepts of MLflow, an open-source platform for managing the end-to-end machine learning lifecycle, and learn how to integrate it with Databricks
  • Gain methodologies to help you improve your project management and workflow skills for data engineering, including applying Kaizen, DevOps, and Data Ops best practices

This course is designed for learners who want to pursue or advance their career in data science or data engineering, or for software developers or engineers who want to grow their data management skill set. With quizzes to test your knowledge throughout, this comprehensive course will help guide your learning journey to become a proficient data engineer, ready to tackle the challenges of today's data-driven world.
This course is part of the Data Engineering Foundations Professional Certificate.

What you'll learn

  • Optimize and manage Hadoop, Spark, and Snowflake platforms
  • Execute data analytics and machine learning tasks using Databricks
  • Enhance Python data science skills with PySpark
  • Manage end-to-end machine learning lifecycle with MLflow
  • Apply Kaizen, DevOps, and DataOps methodologies for data engineering

Syllabus

Module 1: Overview and Introduction to PySpark (7 hours)

  • 10 videos (Total 25 minutes)
  • Meet your Co-Instructor: Kennedy Behrman (0 minutes, Preview module)
  • Meet your Co-Instructor: Noah Gift (1 minute)
  • Overview of Big Data Platforms (1 minute)
  • Getting Started with Hadoop (1 minute)
  • Getting Started with Spark (1 minute)
  • Introduction to Resilient Distributed Datasets (RDD) (2 minutes)
  • Resilient Distributed Datasets (RDD) Demo (4 minutes)
  • Introduction to Spark SQL (1 minute)
  • PySpark Dataframe Demo: Part 1 (3 minutes)
  • PySpark Dataframe Demo: Part 2 (7 minutes)
  • 9 readings (Total 90 minutes)
  • Welcome to Data Engineering Platforms with Python! (10 minutes)
  • What is Apache Hadoop? (10 minutes)
  • What is Apache Spark? (10 minutes)
  • Use Apache Spark in Azure Databricks (optional) (10 minutes)
  • Choosing between Hadoop and Spark (10 minutes)
  • What are RDDs? (10 minutes)
  • Getting Started: Creating RDD's with PySpark (10 minutes)
  • Spark SQL, Dataframes and Datasets (10 minutes)
  • PySpark and Spark SQL (10 minutes)
  • 7 quizzes (Total 210 minutes)
  • PySpark (30 minutes)
  • Big Data Platforms (30 minutes)
  • Apache Hadoop Concepts (30 minutes)
  • Apache Spark Concepts (30 minutes)
  • RDD Concepts (30 minutes)
  • Spark SQL Concepts (30 minutes)
  • PySpark Dataframe Concepts (30 minutes)
  • 2 discussion prompts (Total 20 minutes)
  • Meet and Greet (optional) (10 minutes)
  • Let Us Know if Something's Not Working (10 minutes)
  • 2 ungraded labs (Total 120 minutes)
  • Practice: Creating RDD's with PySpark (60 minutes)
  • Practice: Reading Data into Dataframes (60 minutes)

Module 2: Snowflake (4 hours)

  • 8 videos (Total 27 minutes)
  • What is Snowflake? (2 minutes, Preview module)
  • Snowflake Layers (2 minutes)
  • Snowflake Web UI (3 minutes)
  • Navigating Snowflake (3 minutes)
  • Creating a Table in Snowflake (5 minutes)
  • Snowflake Warehouses (3 minutes)
  • Writing to Snowflake (3 minutes)
  • Reading from Snowflake (2 minutes)
  • 5 readings (Total 50 minutes)
  • Accessing Snowflake (10 minutes)
  • Detailed View Inside Snowflake (10 minutes)
  • Snowsight: The Snowflake Web Interface (10 minutes)
  • Working with Warehouses (10 minutes)
  • Python Connector Documentation (10 minutes)
  • 6 quizzes (Total 180 minutes)
  • Snowflake (30 minutes)
  • Snowflake Architecture (30 minutes)
  • Snowflake Layers (30 minutes)
  • Navigating Snowflake (30 minutes)
  • Creating a Table (30 minutes)
  • Writing to Snowflake (30 minutes)

Module 3: Azure Databricks and MLFlow (5 hours)

  • 16 videos (Total 71 minutes)
  • Accessing Databricks (0 minutes, Preview module)
  • Spark Notebooks with Databricks (4 minutes)
  • Using Data with Databricks (4 minutes)
  • Working with Workspaces in Databricks (3 minutes)
  • Advanced Capabilities of Databricks (1 minute)
  • PySpark Introduction on Databricks (7 minutes)
  • Exploring Databricks Azure Features (3 minutes)
  • Using the DBFS to AutoML Workflow (4 minutes)
  • Load, Register and Deploy ML Models (2 minutes)
  • Databricks Model Registry (2 minutes)
  • Model Serving on Databricks (2 minutes)
  • What is MLOps? (12 minutes)
  • Exploring Open-Source MLFlow Frameworks (5 minutes)
  • Running MLFlow with Databricks (6 minutes)
  • End to End Databricks MLFlow (4 minutes)
  • Databricks Autologging with MLFlow (4 minutes)
  • 7 readings (Total 70 minutes)
  • What is Azure Databricks? (10 minutes)
  • Introduction to Databricks Machine Learning (10 minutes)
  • What is the Databricks File System (DBFS)? (10 minutes)
  • Serverless Compute with Databricks (10 minutes)
  • MLOps Workflow on Azure Databricks (10 minutes)
  • Run MLFlow Projects on Azure Databricks (10 minutes)
  • Databricks Autologging (10 minutes)
  • 4 quizzes (Total 120 minutes)
  • DataBricks (30 minutes)
  • PySpark SQL (30 minutes)
  • PySpark DataFrames (30 minutes)
  • MLFlow with Databricks (30 minutes)
  • 1 ungraded lab (Total 60 minutes)
  • ETL-Part-1: Keyword Extractor Tool to HashTag Tool (60 minutes)

Module 4: DataOps and Operations Methodologies (12 hours)

  • 21 videos (Total 502 minutes)
  • Kaizen Methodology for Data (4 minutes, Preview module)
  • Introducing GitHub CodeSpaces (9 minutes)
  • Compiling Python in GitHub Codespaces (18 minutes)
  • Walking through Sagemaker Studio Lab (28 minutes)
  • Pytest Master Class (Optional) (166 minutes)
  • What is DevOps? (2 minutes)
  • DevOps Key Concepts (35 minutes)
  • Continuous Integration Overview (32 minutes)
  • Build an NLP in Cloud9 with Python (43 minutes)
  • Build a Continuously Deployed Containerized FastAPI Microservice (43 minutes)
  • Hugo Continuous Deploy on AWS (18 minutes)
  • Container Based Continuous Delivery (8 minutes)
  • What is DataOps? (1 minute)
  • DataOps and MLOps with Snowflake (61 minutes)
  • Building Cloud Pipelines with Step Functions and Lambda (16 minutes)
  • What is a Data Lake? (2 minutes)
  • Data Warehouse vs. Feature Store (2 minutes)
  • Big Data Challenges (1 minute)
  • Types of Big Data Processing (1 minute)
  • Real-World Data Engineering Pipeline (2 minutes)
  • Data Feedback Loop (0 minutes)
  • 6 readings (Total 60 minutes)
  • GitHub Codespaces Overview (10 minutes)
  • Getting Started with Amazon SageMaker Studio Lab (10 minutes)
  • Teaching MLOps at Scale with GitHub (Optional) (10 minutes)
  • Getting Started with DevOps and Cloud Computing (10 minutes)
  • Benefits of Serverless ETL Technologies (10 minutes)
  • Next Steps (10 minutes)
  • 4 quizzes (Total 120 minutes)
  • DataOps and Operations Methodologies (30 minutes)
  • Kaizen Methodology (30 minutes)
  • DevOps (30 minutes)
  • DataOps (30 minutes)
  • 1 ungraded lab (Total 60 minutes)
  • ETL-Part2: SQLite ETL Destination (60 minutes)
Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Python for Data Engineering Project (edX) EdX
IBM

Python for Data Engineering Project (edX)

Statistics & Data Analysis

An opportunity to apply your foundational Python skills via a project, using various techniques to collect and work with data. Journey into the realm of becoming a Data Engineer and apply your basic Python knowledge of working with data. You will exercise various techniques in Python to extract data in multiple file formats from different sources, transform it into specific datatypes, and then prepare it for loading it into a database.

Self Paced
Self-Paced
Python APIs Jupyter Notebooks
Details
Data Science and Machine Learning Capstone Project (edX) EdX
IBM

Data Science and Machine Learning Capstone Project (edX)

Statistics & Data Analysis Data Science

Create a project that you can use to showcase your Data Science skills to prospective employers. Apply various data science and machine learning techniques to analyze and visualize a data set involving a real life business scenario and build a predictive model. Now that you've taken several courses on data science and machine learning, it’s time to put your learning to work on a data problem involving a real life scenario. Employers really care about how well you can apply your knowledge and skills to solve real world problems, and the work you do in this capstone project will make you stand out in the job market.

Self Paced
Self-Paced
Python Machine Learning Data Science
Details
CS50's Introduction to Programming with Python (edX) EdX
HarvardX,Harvard University

CS50's Introduction to Programming with Python (edX)

CS: Programming

An introduction to programming using Python, a popular language for general-purpose programming, data science, web programming, and more. An introduction to programming using a language called Python. Learn how to read and write code as well as how to test and "debug" it. Designed for students with and without prior programming experience who'd like to learn Python specifically.

Self Paced
Self-Paced
Programming Python Python Programming
Details
Visualizing Data with Python (edX) EdX
IBM

Visualizing Data with Python (edX)

Statistics & Data Analysis CS: Programming

Data visualization is the graphical representation of data in order to interactively and efficiently convey insights to clients, customers, and stakeholders in general. "A picture is worth a thousand words." We are all familiar with this expression. It especially applies when trying to explain the insights obtained from the analysis of increasingly large datasets. Data visualization plays an essential role in the representation of both small and large-scale data. One of the key skills of a data scientist is the ability to tell a compelling story, visualizing data and findings in an approachable and stimulating way.

Self Paced
Self-Paced
Python Maps Charts
Details
Python Data Structures (edX) EdX
University of Michigan,MichiganX

Python Data Structures (edX)

CS: Programming

The second course in Python for Everybody explores variables that contain collections of data like string, lists, dictionaries, and tuples. Learning how to store and represent and manipulate data collections while a program is running is an important part of learning how to program.

Self Paced
Self-Paced
Programming Python Data Structures
Details
Python for Data Science (edX) EdX
University of California, San Diego,UC San DiegoX

Python for Data Science (edX)

Statistics & Data Analysis CS: Programming

Learn to use powerful, open-source, Python tools, including Pandas, Git and Matplotlib, to manipulate, analyze, and visualize complex datasets. In the information age, data is all around us. Within this data are answers to compelling questions across many societal domains (politics, business, science, etc.). But if you had access to a large dataset, would you be able to find the answers you seek?

Self Paced
Self-Paced
Programming Python Machine Learning
Details