Linux and Bash for Data Engineering (edX)

In this course, you will:

• Gain proficiency in using Linux for data engineering tasks
• Learn to configure and customize your Bash shell environment
• Understand how to manipulate data using shell variables and standard streams
• Construct powerful Bash scripts and command-line tools to automate workflows
• Master file and data management using Linux utilities and shell scripting

Whether you're a data engineer, software developer, or system administrator, this course will equip you with the Linux and Bash skills necessary to tackle real-world data challenges efficiently and effectively. Enhance your productivity and streamline your data workflows with the power of Linux and Bash scripting.
This course is part of the Data Engineering Foundations Professional Certificate.

What you'll learn

• Navigating and managing Linux filesystems
• Configuring and customizing Bash shell environments
• Manipulating data using shell variables and streams
• Building Bash scripts and command-line tools
• Searching and processing text data in Linux
• Automating data workflows with shell scripting

Syllabus

Module 1: Using Linux (18 hours)
\- Welcome to Linux and Bash for Data Engineering (Video, 0 minutes, Preview module)
\- Meet your Course Instructor: Noah Gift (Video, 0 minutes)
\- Overview of Key Concepts (Video, 4 minutes)
\- Introduction to Working with Linux (Video, 0 minutes)
\- Introduction to the Linux Shell (Video, 2 minutes)
\- Installing and Accessing Linux (Video, 11 minutes)
\- Getting Started with Github Codespaces (Video, 9 minutes)
\- Comparing Linux Shell Environments (Video, 9 minutes)
\- Writing 'Hello World' Bash Script (Video, 6 minutes)
\- Popular Terminal Commands in Linux (Video, 14 minutes)
\- Introduction to Shell Piping (Video, 1 minute)
\- What is Shell Piping? (Video, 3 minutes)
\- Walkthrough of Shell Pipeline Examples (Video, 12 minutes)
\- Introduction to SSH (Video, 1 minute)
\- What is SSH? (Video, 4 minutes)
\- Creating SSH Keys and using with GitHub (Video, 4 minutes)
\- Using SSH to Connect to Remote Server in AWS Cloud (Video, 7 minutes)
\- Walkthrough of SSH Tunneling Example (Video, 4 minutes)
\- Key Terms (Reading, 10 minutes)
\- Meet your Supporting Instructors: Kennedy Behrman and Alfredo Deza (Reading, 10 minutes)
\- Course Structure and Discussion Etiquette (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Amazon CodeWhisperer CLI (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Shell Pipelines (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Securely Connecting to a Linux Instance with SSH (Reading, 10 minutes)
\- Secure Remote Development with VSCode and SSH (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Using Linux (Quiz, 30 minutes)
\- Quiz-Using Linux (Assignment, 180 minutes)
\- Quiz-How Shell Piping Works (Assignment, 180 minutes)
\- Quiz-Using SSH (Assignment, 180 minutes)
\- Meet and Greet (optional) (Discussion Prompt, 10 minutes)
\- Practicing Common Linux Shell Commands (Ungraded Lab, 60 minutes)
\- Using Linux Lab Challenge (Ungraded Lab, 60 minutes)
\- Practicing Common Shell Piping Operations (Ungraded Lab, 60 minutes)
\- Creating SSH Keys Locally (Ungraded Lab, 60 minutes)
\- Creating a Shell Pipeline (Ungraded Lab, 60 minutes)

Module 2: Using Bash (14 hours)
\- Introduction to Working with Bash (Video, 0 minutes, Preview module)
\- Introduction to Configuring your Bash Shell Environment (Video, 2 minutes)
\- What are Shell Configuration Files? (Video, 5 minutes)
\- Configuring .bashrc (Video, 11 minutes)
\- Configuring .zshrc and Third-Party Tool oh-my-zsh (Video, 9 minutes)
\- Introduction to Working with Shell Variables (Video, 1 minute)
\- What are Shell Variables? (Video, 4 minutes)
\- Exporting Shell Variables in an Interactive Prompt (Video, 6 minutes)
\- Using Shell Variables in a Bash Script (Video, 7 minutes)
\- Introduction to Standard Streams (Video, 1 minute)
\- What are Standard Streams? (Video, 2 minutes)
\- Using Standard Out (Video, 5 minutes)
\- Using Standard In (Video, 2 minutes)
\- Using Standard Error (Video, 2 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Using Bash (Quiz, 30 minutes)
\- Quiz-Configuring the Shell Environment (Assignment, 180 minutes)
\- Quiz-Working with Shell Variables (Assignment, 180 minutes)
\- Quiz-Standard Streams: Using Standard In, Standard Out, Standard Error (Assignment, 180 minutes)
\- Configuring the Bash Shell (Ungraded Lab, 60 minutes)
\- Sourcing Shell Variables from a Script (Ungraded Lab, 60 minutes)
\-Practicing with Bash Streams (Ungraded Lab, 60 minutes)

Module 3: Building Bash Scripts (15 hours)
\- Introduction to Building Bash Scripts (Video, 0 minutes, Preview module)
\- Introduction to Shell Logic and Control Flow (Video, 0 minutes)
\- What is Shell Control Flow? (Video, 2 minutes)
\- Using Shell Control Flow to Solve Problems in Bash (Video, 2 minutes)
\- Using Shell Loops in Bash (Video, 6 minutes)
\- Evaluating Conditions in Bash (Video, 5 minutes)
\- Introduction to Manipulating Data in the Bash Shell (Video, 1 minute)
\- Shell Techniques for Data in the Bash Shell (Video, 2 minutes)
\- Truncating Data in Bash (Video, 3 minutes)
\- Filtering Data in Bash (Video, 3 minutes)
\- Searching Data in Bash (Video, 4 minutes)
\- Introduction to Writing Scripts and Command-Line Tools in Bash (Video, 0 minutes)
\- What are Scripts and Command-Line Tools? (Video, 3 minutes)
\- Building a Bash Script (Video, 4 minutes)
\- Building a Bash Function (Video, 5 minutes)
\- Building a Bash Command-Line Tool (Video, 3 minutes)
\- Bash-Adjacent Tools: Makefiles and Dockerfiles (Video, 7 minutes)
\- Data Structures in Bash (Video, 4 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Building Bash Scripts (Quiz, 30 minutes)
\- Quiz-Using Shell Logic and Control Flow (Assignment, 180 minutes)
\- Quiz-Manipulating Data with the Shell (Assignment, 180 minutes)
\- Quiz-Writing Bash Scripts and Command-Line Tools (Assignment, 180 minutes)
\- Writing Loops in Bash (Ungraded Lab, 60 minutes)
\- Truncating Large Files with Bash (Ungraded Lab, 60 minutes)
\- Building a Command-Line Tool for Data Processing (Ungraded Lab, 60 minutes)
\- Building a Bash CLI with Options (Ungraded Lab, 60 minutes)

Module 4: Composing File and Data Solutions (16 hours)
\- Introduction to Composing File and Data Management Solutions in Linux (Vide, 0 minutes, Preview module)
\- Introduction to Searching a Filesystem in Linux (Video, 0 minutes)
\- Methods for Searching a Filesystem in Linux (Video, 4 minutes)
\- Using the Locate Command to Find Files in Linux (Video, 3 minutes)
\- Using the Find Command to Find Files in Linux (Video, 2 minutes)
\- Using xargs to Extend Search Capabilities in Linux (Video, 2 minutes)
\- Using mdfind on OS X (Video, 6 minutes)
\- Introduction to Modifying Files, Directories, Permissions and Archiving in Linux (Video, 0 minutes)
\- Overview of Modifying the Filesystem in Linux (Video, 3 minutes)
\- Moving Files and Directories in Linux (Video, 6 minutes)
\- Setting Permissions on Files and Directories in Linux (Video, 4 minutes)
\- Archiving Data in Linux (Video, 3 minutes)
\- Introduction to Processing Text in Linux (Video, 0 minutes)
\- Benefits of Processing Text in Linux (Video, 1 minute)
\- Using grep, cut, sort and unique in Linux (Video, 8 minutes)
\- Editing with Truncation, awk and sed in Linux (Video, 7 minutes)
\- Using Regular Expressions (regex) in Linux (Video, 5 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Key Terms (Reading, 10 minutes)
\- Lesson Reflection (Reading, 10 minutes)
\- Next Steps (Reading, 10 minutes)
\- Composing File and Data Management Solutions with Linux (Quiz, 30 minutes)
\- Quiz-Searching the Filesystem (Assignment, 180 minutes)
\- Quiz-Modifying Files, Directories, Permissions and Archiving (Assignment, 180 minutes)
\- Quiz-Processing Text (Assignment, 180 minutes)
\- Searching from the Bash Prompt (Ungraded Lab, 60 minutes)
\- Practicing the Basics of Shell Permissions (Ungraded Lab, 60 minutes)
\- Searching for Text with Bash (Ungraded Lab, 60 minutes)
\- Searching the filesystem with find (Ungraded Lab, 60 minutes)
\- Final Course Challenge: Extended Phrase Repeater (Ungraded Lab, 60 minutes)