EdX

Spark, Hadoop, and Snowflake for Data Engineering (edX)

Offered by AI (Pragmatic AI Labs),
Spark, Hadoop, and Snowflake for Data Engineering (edX)

Gain the skills for building efficient and scalable data pipelines. Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) and learn how to optimize them using Python, PySpark, and MLflow.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

In this course, you will:

  • Explore essential data engineering platforms (Hadoop, Spark, and Snowflake) and learn how to optimize and manage them
  • Delve into Databricks, a powerful platform for executing data analytics and machine learning tasks
  • Hone your Python data science skills with PySpark
  • Discover the key concepts of MLflow, an open-source platform for managing the end-to-end machine learning lifecycle, and learn how to integrate it with Databricks
  • Gain methodologies to help you improve your project management and workflow skills for data engineering, including applying Kaizen, DevOps, and Data Ops best practices

This course is designed for learners who want to pursue or advance their career in data science or data engineering, or for software developers or engineers who want to grow their data management skill set. With quizzes to test your knowledge throughout, this comprehensive course will help guide your learning journey to become a proficient data engineer, ready to tackle the challenges of today's data-driven world.
This course is part of the Data Engineering Foundations Professional Certificate.

What you'll learn

  • Optimize and manage Hadoop, Spark, and Snowflake platforms
  • Execute data analytics and machine learning tasks using Databricks
  • Enhance Python data science skills with PySpark
  • Manage end-to-end machine learning lifecycle with MLflow
  • Apply Kaizen, DevOps, and DataOps methodologies for data engineering

Syllabus

Module 1: Overview and Introduction to PySpark (7 hours)

  • 10 videos (Total 25 minutes)
  • Meet your Co-Instructor: Kennedy Behrman (0 minutes, Preview module)
  • Meet your Co-Instructor: Noah Gift (1 minute)
  • Overview of Big Data Platforms (1 minute)
  • Getting Started with Hadoop (1 minute)
  • Getting Started with Spark (1 minute)
  • Introduction to Resilient Distributed Datasets (RDD) (2 minutes)
  • Resilient Distributed Datasets (RDD) Demo (4 minutes)
  • Introduction to Spark SQL (1 minute)
  • PySpark Dataframe Demo: Part 1 (3 minutes)
  • PySpark Dataframe Demo: Part 2 (7 minutes)
  • 9 readings (Total 90 minutes)
  • Welcome to Data Engineering Platforms with Python! (10 minutes)
  • What is Apache Hadoop? (10 minutes)
  • What is Apache Spark? (10 minutes)
  • Use Apache Spark in Azure Databricks (optional) (10 minutes)
  • Choosing between Hadoop and Spark (10 minutes)
  • What are RDDs? (10 minutes)
  • Getting Started: Creating RDD's with PySpark (10 minutes)
  • Spark SQL, Dataframes and Datasets (10 minutes)
  • PySpark and Spark SQL (10 minutes)
  • 7 quizzes (Total 210 minutes)
  • PySpark (30 minutes)
  • Big Data Platforms (30 minutes)
  • Apache Hadoop Concepts (30 minutes)
  • Apache Spark Concepts (30 minutes)
  • RDD Concepts (30 minutes)
  • Spark SQL Concepts (30 minutes)
  • PySpark Dataframe Concepts (30 minutes)
  • 2 discussion prompts (Total 20 minutes)
  • Meet and Greet (optional) (10 minutes)
  • Let Us Know if Something's Not Working (10 minutes)
  • 2 ungraded labs (Total 120 minutes)
  • Practice: Creating RDD's with PySpark (60 minutes)
  • Practice: Reading Data into Dataframes (60 minutes)

Module 2: Snowflake (4 hours)

  • 8 videos (Total 27 minutes)
  • What is Snowflake? (2 minutes, Preview module)
  • Snowflake Layers (2 minutes)
  • Snowflake Web UI (3 minutes)
  • Navigating Snowflake (3 minutes)
  • Creating a Table in Snowflake (5 minutes)
  • Snowflake Warehouses (3 minutes)
  • Writing to Snowflake (3 minutes)
  • Reading from Snowflake (2 minutes)
  • 5 readings (Total 50 minutes)
  • Accessing Snowflake (10 minutes)
  • Detailed View Inside Snowflake (10 minutes)
  • Snowsight: The Snowflake Web Interface (10 minutes)
  • Working with Warehouses (10 minutes)
  • Python Connector Documentation (10 minutes)
  • 6 quizzes (Total 180 minutes)
  • Snowflake (30 minutes)
  • Snowflake Architecture (30 minutes)
  • Snowflake Layers (30 minutes)
  • Navigating Snowflake (30 minutes)
  • Creating a Table (30 minutes)
  • Writing to Snowflake (30 minutes)

Module 3: Azure Databricks and MLFlow (5 hours)

  • 16 videos (Total 71 minutes)
  • Accessing Databricks (0 minutes, Preview module)
  • Spark Notebooks with Databricks (4 minutes)
  • Using Data with Databricks (4 minutes)
  • Working with Workspaces in Databricks (3 minutes)
  • Advanced Capabilities of Databricks (1 minute)
  • PySpark Introduction on Databricks (7 minutes)
  • Exploring Databricks Azure Features (3 minutes)
  • Using the DBFS to AutoML Workflow (4 minutes)
  • Load, Register and Deploy ML Models (2 minutes)
  • Databricks Model Registry (2 minutes)
  • Model Serving on Databricks (2 minutes)
  • What is MLOps? (12 minutes)
  • Exploring Open-Source MLFlow Frameworks (5 minutes)
  • Running MLFlow with Databricks (6 minutes)
  • End to End Databricks MLFlow (4 minutes)
  • Databricks Autologging with MLFlow (4 minutes)
  • 7 readings (Total 70 minutes)
  • What is Azure Databricks? (10 minutes)
  • Introduction to Databricks Machine Learning (10 minutes)
  • What is the Databricks File System (DBFS)? (10 minutes)
  • Serverless Compute with Databricks (10 minutes)
  • MLOps Workflow on Azure Databricks (10 minutes)
  • Run MLFlow Projects on Azure Databricks (10 minutes)
  • Databricks Autologging (10 minutes)
  • 4 quizzes (Total 120 minutes)
  • DataBricks (30 minutes)
  • PySpark SQL (30 minutes)
  • PySpark DataFrames (30 minutes)
  • MLFlow with Databricks (30 minutes)
  • 1 ungraded lab (Total 60 minutes)
  • ETL-Part-1: Keyword Extractor Tool to HashTag Tool (60 minutes)

Module 4: DataOps and Operations Methodologies (12 hours)

  • 21 videos (Total 502 minutes)
  • Kaizen Methodology for Data (4 minutes, Preview module)
  • Introducing GitHub CodeSpaces (9 minutes)
  • Compiling Python in GitHub Codespaces (18 minutes)
  • Walking through Sagemaker Studio Lab (28 minutes)
  • Pytest Master Class (Optional) (166 minutes)
  • What is DevOps? (2 minutes)
  • DevOps Key Concepts (35 minutes)
  • Continuous Integration Overview (32 minutes)
  • Build an NLP in Cloud9 with Python (43 minutes)
  • Build a Continuously Deployed Containerized FastAPI Microservice (43 minutes)
  • Hugo Continuous Deploy on AWS (18 minutes)
  • Container Based Continuous Delivery (8 minutes)
  • What is DataOps? (1 minute)
  • DataOps and MLOps with Snowflake (61 minutes)
  • Building Cloud Pipelines with Step Functions and Lambda (16 minutes)
  • What is a Data Lake? (2 minutes)
  • Data Warehouse vs. Feature Store (2 minutes)
  • Big Data Challenges (1 minute)
  • Types of Big Data Processing (1 minute)
  • Real-World Data Engineering Pipeline (2 minutes)
  • Data Feedback Loop (0 minutes)
  • 6 readings (Total 60 minutes)
  • GitHub Codespaces Overview (10 minutes)
  • Getting Started with Amazon SageMaker Studio Lab (10 minutes)
  • Teaching MLOps at Scale with GitHub (Optional) (10 minutes)
  • Getting Started with DevOps and Cloud Computing (10 minutes)
  • Benefits of Serverless ETL Technologies (10 minutes)
  • Next Steps (10 minutes)
  • 4 quizzes (Total 120 minutes)
  • DataOps and Operations Methodologies (30 minutes)
  • Kaizen Methodology (30 minutes)
  • DevOps (30 minutes)
  • DataOps (30 minutes)
  • 1 ungraded lab (Total 60 minutes)
  • ETL-Part2: SQLite ETL Destination (60 minutes)
Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Big Data, Hadoop, and Spark Basics (edX) EdX
IBM

Big Data, Hadoop, and Spark Basics (edX)

Computer Science

This course provides foundational big data practitioner knowledge and analytical skills using popular big data tools, including Hadoop and Spark. Learn and practice your big data skills hands-on. Organizations need skilled, forward-thinking Big Data practitioners who can apply their business and technical skills to unstructured data such as tweets, posts, pictures, audio files, videos, sensor data, and satellite imagery, and more, to identify behaviors and preferences of prospects, clients, competitors, and others. ****

Self Paced
Self-Paced
Big Data Hadoop Apache Spark
Details
CS50's Introduction to Programming with Python (edX) EdX
HarvardX,Harvard University

CS50's Introduction to Programming with Python (edX)

CS: Programming

An introduction to programming using Python, a popular language for general-purpose programming, data science, web programming, and more. An introduction to programming using a language called Python. Learn how to read and write code as well as how to test and "debug" it. Designed for students with and without prior programming experience who'd like to learn Python specifically.

Self Paced
Self-Paced
Programming Python Python Programming
Details
Python for Data Science (edX) EdX
University of California, San Diego,UC San DiegoX

Python for Data Science (edX)

Statistics & Data Analysis CS: Programming

Learn to use powerful, open-source, Python tools, including Pandas, Git and Matplotlib, to manipulate, analyze, and visualize complex datasets. In the information age, data is all around us. Within this data are answers to compelling questions across many societal domains (politics, business, science, etc.). But if you had access to a large dataset, would you be able to find the answers you seek?

Self Paced
Self-Paced
Programming Python Machine Learning
Details
Data Science and Machine Learning Capstone Project (edX) EdX
IBM

Data Science and Machine Learning Capstone Project (edX)

Statistics & Data Analysis Data Science

Create a project that you can use to showcase your Data Science skills to prospective employers. Apply various data science and machine learning techniques to analyze and visualize a data set involving a real life business scenario and build a predictive model. Now that you've taken several courses on data science and machine learning, it’s time to put your learning to work on a data problem involving a real life scenario. Employers really care about how well you can apply your knowledge and skills to solve real world problems, and the work you do in this capstone project will make you stand out in the job market.

Self Paced
Self-Paced
Python Machine Learning Data Science
Details
CS50's Introduction to Artificial Intelligence with Python (edX) EdX
HarvardX,Harvard University

CS50's Introduction to Artificial Intelligence with Python (edX)

Robotics & Computer Vision

Learn to use machine learning in Python in this introductory course on artificial intelligence. AI is transforming how we live, work, and play. By enabling new technologies like self-driving cars and recommendation systems or improving old ones like medical diagnostics and search engines, the demand for expertise in AI and machine learning is growing rapidly. This course will enable you to take the first step toward solving important real-world problems and future-proofing your career.

Self Paced
Self-Paced
Python Artificial Intelligence Machine Learning
Details
Big Data Computing with Spark (edX) EdX
The Hong Kong University of Science and Technology - HKUST,HKUSTx

Big Data Computing with Spark (edX)

Computer Science

Learn the theory and gain hands-on experience of big data systems, using Spark as the exemplary platform. Big data systems such as Hadoop and Spark emerge as enabling technologies in managing massive amounts of data across hundreds or even thousands of computing nodes. Meanwhile, cloud computing platforms have made these technologies easily accessible to individuals as well as large enterprises.

Self Paced
Self-Paced
Big Data Hadoop MapReduce
Details
Python: aprender a programar (edX) EdX
Universitat Politècnica de València,UPValenciaX

Python: aprender a programar (edX)

CS: Programming

Aprende a programar con Python desde cero. Comienza conociendo variables, bucles y funciones y llega a manejar ficheros de texto. Este curso empieza desde 0 para aprender a programar con Python, tratando los fundamentos de programación como son las variables y constantes, las condiciones, los bucles y los módulos y funciones, para acabar introduciendo el tratamiento de cadenas de texto y los ficheros.

Self Paced
Self-Paced
Python Python Programming Programación
Details