Sample-based Learning Methods (Coursera)

Sample-based Learning Methods (Coursera)

In this course, you will learn about several algorithms that can learn near optimal policies based on trial and error interaction with the environment---learning from the agent’s own experience. Learning from actual experience is striking because it requires no prior knowledge of the environment’s dynamics, yet can still attain optimal behavior. We will cover intuitively simple but powerful Monte Carlo methods, and temporal difference learning methods including Q-learning. We will wrap up this course investigating how we can get the best of both worlds: algorithms that can combine model-based planning (similar to dynamic programming) and temporal difference updates to radically accelerate learning.

Class Deals by MOOC List - Click here and see Coursera's Active Discounts, Deals, and Promo Codes.

By the end of this course you will be able to:

  • Understand Temporal-Difference learning and Monte Carlo as two strategies for estimating value functions from sampled experience
  • Understand the importance of exploration, when using sampled experience rather than dynamic programming sweeps within a model
  • Understand the connections between Monte Carlo and Dynamic Programming and TD.
  • Implement and apply the TD algorithm, for estimating value functions
  • Implement and apply Expected Sarsa and Q-learning (two TD methods for control)
  • Understand the difference between on-policy and off-policy control
  • Understand planning with simulated experience (as opposed to classic planning strategies)
  • Implement a model-based approach to RL, called Dyna, which uses simulated experience
  • Conduct an empirical study to see the improvements in sample efficiency when using Dyna

Course 2 of 4 in the Reinforcement Learning Specialization.

Syllabus

WEEK 1
Welcome to the Course!
Welcome to the second course in the Reinforcement Learning Specialization: Sample-Based Learning Methods, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!
Monte Carlo Methods for Prediction & Control
This week you will learn how to estimate value functions and optimal policies, using only sampled experience from the environment. This module represents our first step toward incremental learning methods that learn from the agent’s own interaction with the world, rather than a model of the world. You will learn about on-policy and off-policy methods for prediction and control, using Monte Carlo methods---methods that use sampled returns. You will also be reintroduced to the exploration problem, but more generally in RL, beyond bandits.

WEEK 2
Temporal Difference Learning Methods for Prediction
This week, you will learn about one of the most fundamental concepts in reinforcement learning: temporal difference (TD) learning. TD learning combines some of the features of both Monte Carlo and Dynamic Programming (DP) methods. TD methods are similar to Monte Carlo methods in that they can learn from the agent’s interaction with the world, and do not require knowledge of the model. TD methods are similar to DP methods in that they bootstrap, and thus can learn online---no waiting until the end of an episode. You will see how TD can learn more efficiently than Monte Carlo, due to bootstrapping. For this module, we first focus on TD for prediction, and discuss TD for control in the next module. This week, you will implement TD to estimate the value function for a fixed policy, in a simulated domain.

WEEK 3
Temporal Difference Learning Methods for Control
This week, you will learn about using temporal difference learning for control, as a generalized policy iteration strategy. You will see three different algorithms based on bootstrapping and Bellman equations for control: Sarsa, Q-learning and Expected Sarsa. You will see some of the differences between the methods for on-policy and off-policy control, and that Expected Sarsa is a unified algorithm for both. You will implement Expected Sarsa and Q-learning, on Cliff World.

WEEK 4
Planning, Learning & Acting
Up until now, you might think that learning with and without a model are two distinct, and in some ways, competing strategies: planning with Dynamic Programming verses sample-based learning via TD methods. This week we unify these two strategies with the Dyna architecture. You will learn how to estimate the model from data and then use this model to generate hypothetical experience (a bit like dreaming) to dramatically improve sample efficiency compared to sample-based methods like Q-learning. In addition, you will learn how to design learning systems that are robust to inaccurate models.

Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Data Science for Business Innovation (Coursera) Coursera
Politecnico di Milano,EIT Digital

Data Science for Business Innovation (Coursera)

The course is a compendium of the must-have expertise in data science for executive and middle-management to foster data-driven innovation. It consists of introductory lectures spanning big data, machine learning, data valorization and communication. Topics cover the essential concepts and intuitions on data needs, data analysis, machine learning methods, respective pros and cons, and practical applicability issues.

Dec 14th 2026
4 Weeks
Artificial Intelligence: Ethics & Societal Challenges (Coursera) Coursera
Lund University

Artificial Intelligence: Ethics & Societal Challenges (Coursera)

Artificial Intelligence: Ethics & Societal Challenges is a four-week course that explores ethical and societal aspects of the increasing use of artificial intelligent technologies (AI). The aim of the course is to raise awareness of ethical and societal aspects of AI and to stimulate reflection and discussion upon implications of the use of AI in society.

Dec 14th 2026
4 Weeks
Sistemas difusos (Coursera) Coursera
Universidad Nacional de Colombia

Sistemas difusos (Coursera)

Los sistemas difusos permiten efectuar cálculos cuando hay información con incertidumbre, o cuando se debe combinar información tanto cuantitativa como cualitativa. Se trata de una aproximación matemática para modelar esas situaciones. Este curso está diseñado para ayudar a entender y explicar cómo funcionan dichos sistemas. El curso tiene una aproximación teórica y práctica. Los principios matemáticos son de un nivel bajo y están al alcance de un público muy amplio. El curso cuenta con varios laboratorios para aprender a utilizar las herramientas de software que usan esos principios. Este componente práctico requiere una comprensión mínima de programación.

Dec 14th 2026
4 Weeks
Artificial Intelligence (AI) Education for Teachers (Coursera) Coursera
Macquarie University

Artificial Intelligence (AI) Education for Teachers (Coursera)

Today’s learners need to know what artificial intelligence (AI) is, how it works, how to use it in their everyday lives, and how it could potentially be used in their future. Using AI requires skills and values which extend far beyond simply having knowledge about coding and technology. This course is designed by teachers, for teachers, and will bridge the gap between commonly held beliefs about AI, and what it really is. AI can be embedded into all areas of the school curriculum and this course will show you how. This course will appeal to teachers who want to increase their general understanding of AI, including why it is important for learners; and/or to those who want to embed AI into their teaching practice and their students’ learning.

Dec 14th 2026
5-12 Weeks
Foundations of Teaching for Learning: Introduction to Student Assessment (Coursera) Coursera
Commonwealth Education Trust

Foundations of Teaching for Learning: Introduction to Student Assessment (Coursera)

The Foundations of Teaching for Learning programme is for anyone who is teaching, or who would like to teach, in any subject and any context - be it at school, at home or in the workplace. With dynamic lessons taught by established and respected professionals from across the Commonwealth, this eight course programme will see you develop and strengthen your skills in teaching, professionalism, assessment, and more. As you carry on through the programme, you will find yourself strengthening not only your skills, but your connection with colleagues across the globe. A professional development opportunity not to be missed.

Dec 14th 2026
5-12 Weeks
Assessment for Learning (Coursera) Coursera
University of Illinois at Urbana-Champaign

Assessment for Learning (Coursera)

For several decades now, assessment has become an increasingly pressing educational priority. Teacher and school accountability systems have come to be based on analysis of large-scale, standardized summative assessments. As a consequence, assessment now dominates most conversations about reform, particularly as a measure of teacher and school accountability for learner performance. This course is an overview of current debates about testing, and analyses the strengths and weaknesses of a variety of approaches to assessment.

Dec 14th 2026
4 Weeks
Introduction to Machine Learning (Coursera) Coursera
Duke University

Introduction to Machine Learning (Coursera)

This course will provide you a foundational understanding of machine learning models (logistic regression, multilayer perceptrons, convolutional neural networks, natural language processing, etc.) as well as demonstrate how these models can solve complex problems in a variety of industries, from medical diagnostics to image recognition to text prediction.

Dec 14th 2026
5-12 Weeks
Foundations of Teaching for Learning: Being a Teacher (Coursera) Coursera
Commonwealth Education Trust

Foundations of Teaching for Learning: Being a Teacher (Coursera)

The Foundations of Teaching for Learning programme is for anyone who is teaching, or who would like to teach, in any subject and any context - be it at school, at home or in the workplace. With dynamic lessons taught by established and respected professionals from across the Commonwealth, this eight course programme will see you develop and strengthen your skills in teaching, professionalism, assessment, and more. As you carry on through the programme, you will find yourself strengthening not only your skills, but your connection with colleagues across the globe. A professional development opportunity not to be missed.

Dec 14th 2026
5-12 Weeks
Machine Learning for Smart Beta (Coursera) Coursera
Sungkyunkwan University - SKKU

Machine Learning for Smart Beta (Coursera)

In this 4 week course, you will learn about Smart Beta products. Smart betas products have the characteristics of both passive investment(having predetermined rules) and active investments(allows for factor investment). We will walk through the creation mechanisms behind different smart beta products and recreate some of them using R programming.

Dec 14th 2026
4 Weeks
Landing.AI for Beginners: Build Data Visualization AI Models (Coursera) Coursera
Coursera Project Network

Landing.AI for Beginners: Build Data Visualization AI Models (Coursera)

In this 1-hour long project-based course, you'll step into the exciting field of Computer Vision and Generative AI using the LandingLens platform. We'll start by exploring the concept of visual prompting, and initiating a visual prompting project. LandingLens simplifies the model creation, training, and deployment process, making it a user-friendly platform for this endeavor.

Dec 14th 2026
1 Week
Introduction to Digital Transformation (Coursera) Coursera
Siemens

Introduction to Digital Transformation (Coursera)

This course is primarily for professionals, college students, and advanced high school students who are interested in driving the digital transformation by integrating automation, software, and cutting-edge technologies. This course represents a foundational introduction to Digital Transformation, appropriate for learners with a basic familiarity with common business terms and concepts and an interest in digital technology.

Dec 14th 2026
5-12 Weeks
Foundations of Teaching for Learning: Planning for Teaching and Learning (Coursera) Coursera
Commonwealth Education Trust

Foundations of Teaching for Learning: Planning for Teaching and Learning (Coursera)

The Foundations of Teaching for Learning programme is for anyone who is teaching, or who would like to teach, in any subject and any context - be it at school, at home or in the workplace. With dynamic lessons taught by established and respected professionals from across the Commonwealth, this eight course programme will see you develop and strengthen your skills in teaching, professionalism, assessment, and more. As you carry on through the programme, you will find yourself strengthening not only your skills, but your connection with colleagues across the globe. A professional development opportunity not to be missed.

Dec 14th 2026
5-12 Weeks