EdX

Data Engineering with Databricks (edX)

Offered by AI (Pragmatic AI Labs),
Data Engineering with Databricks (edX)

Become an expert in modern data engineering on Databricks' unified lakehouse platform. Master ETL pipelines, data transformations with Apache Spark, and Delta Lake for reliable data management.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

Master Data Engineering on Databricks Lakehouse Platform

  • Learn Databricks architecture, cluster management & notebook analysis
  • Build reliable ETL pipelines with Delta Lake for data transformation
  • Implement advanced data processing techniques with Apache Spark

Course Highlights:

  • Create & scale Databricks clusters for workloads
  • Load data from diverse sources into notebooks
  • Explore, visualize & profile datasets with notebooks
  • Version control & share notebooks via Git integration
  • Read & ingest data in various file formats
  • Transform data with SQL & DataFrame operations
  • Handle complex data types like arrays, structs, timestamps
  • Deduplicate, join & flatten nested data structures
  • Identify & fix data quality issues with UDFs
  • Load cleansed data into Delta Lake for reliability
  • Build production-ready pipelines with Delta Live Tables
  • Schedule & monitor workloads using Databricks Jobs
  • Secure data access with Unity Catalog

Gain comprehensive skills in data engineering on Databricks through hands-on labs, real-world projects and best practices for the modern data lakehouse.
This course is part of the Large Language Model Operations (LLMOps) Professional Certificate.

What you'll learn

  • Use Databricks for data engineering and ML workloads
  • Create and design ML pipelines
  • Use Llamafile and other local LLMs like Mixtral

Syllabus

Module 1: Databricks Lakehouse Platform Fundamentals

  • Introduction to the Databricks Lakehouse Platform and its architecture
  • Creating, managing, and configuring clusters
  • Setting up and using Databricks with IntelliJ, RStudio, and the Databricks CLI
  • Introduction to notebooks, including execution, sharing, and multi-language support
  • Efficient data transformation with Spark SQL and the Catalog Explorer
  • Creating tables from files and querying external data sources
  • Reliable data pipelines with Delta Lake, ACID transactions, and Z-Ordering optimization

Module 2: Data Transformation and Pipelines
Automated pipelines with Delta Live Tables
Delta Live Tables components
Continuous vs triggered pipelines
Configuring Auto Loader
Querying pipeline events
End-to-end example of Delta Live
Vacuum and garbage collection
Orchestrating workloads with Databricks Jobs
Multi-task workflows and task dependencies
Viewing job history
Using dashboards
Handling failures and configuring retries
Unified data access with Unity Catalog
Catalogs vs metastores
Unity Catalog quickstart in Python
Applying object security
Best practices for catalogs, connections, and business units

Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Distributed Machine Learning with Apache Spark (edX) EdX
University of California, Berkeley,BerkeleyX

Distributed Machine Learning with Apache Spark (edX)

CS: Software Engineering Statistics & Data Analysis

Learn the underlying principles required to develop scalable machine learning pipelines and gain hands-on experience using Apache Spark. Machine learning aims to extract knowledge from data, relying on fundamental concepts in computer science, statistics, probability and optimization.

No sessions available
4 Weeks
Algorithms Machine Learning Exploratory Data Analysis
Details
Data Governance with Databricks (Coursera) Coursera
Whizlabs

Data Governance with Databricks (Coursera)

CS: Software Engineering

Databricks is a cloud-based data engineering tool used to process and transform large amounts of data and explore the data through machine learning models. It combines data warehouses & data lakes into a lakehouse architecture. Data governance is a broad approach that comprises the principles, practices, and tools to manage an organization’s data assets throughout its lifecycle. A data governance strategy allows organizations to make data easily available protecting their data from unauthorized access, and ensuring compliance with regulatory requirements.

Sep 14th 2026
4 Weeks
Data Warehouse Data Governance AWS
Details
Data Storytelling with Power BI (Coursera) Coursera
Edureka

Data Storytelling with Power BI (Coursera)

CS: Information & Technology

Welcome to the Data Storytelling with Power BI course, where you'll embark on a journey to acquire practical expertise in data transformation and visualization. Leverage the potential of Power BI to craft narratives through structured data, leading to the discovery of more profound insights. Throughout this course, you'll explore the industry-specific applications of Power BI and delve into its various features and functionalities.

Sep 14th 2026
5-12 Weeks
Data Management Data Visualization Power BI
Details
Big Data Analysis with Scala and Spark (Coursera) Coursera
École Polytechnique Fédérale de Lausanne

Big Data Analysis with Scala and Spark (Coursera)

CS: Theory CS: Programming

Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout.

Sep 14th 2026
4 Weeks
Programming Algorithms SQL
Details