SPPU · Final Year Computer Engineering · 2019 Course (Revised)
Reinforcement Learning
410653D · Elective VI · Semester VIII · 3 credits · In-semester 30 + end-semester 70
One of the Elective VI options.
Unit 1: Introduction
7 hoursElements, limits and scope, Tic-Tac-Toe; multi-armed bandits, action-value methods, incremental implementation, non-stationary problems, optimistic initial values, UCB, gradient bandits, associative search; finite MDPs, goals, rewards, returns and episodes, policies and value functions, optimality.
Marks: 15 in the in-semester exam
- Reinforcement learning: elements and scope
- Multi-armed bandits
- Action selection: optimistic values, UCB and gradient bandits
- Finite Markov decision processes
- Policies, value functions and optimality
Unit 2: Dynamic Programming and Monte Carlo Methods
7 hoursPolicy evaluation, improvement and iteration, value iteration, asynchronous DP, generalised policy iteration; Monte Carlo prediction and control, off-policy control, importance sampling.
Marks: 15 in the in-semester exam
- Policy evaluation and policy iteration
- Value iteration and generalised policy iteration
- Monte Carlo prediction and control
- Off-policy learning and importance sampling
Unit 3: Temporal Difference Learning and Bootstrapping
7 hoursTD prediction, Sarsa, Q-learning, expected Sarsa, double learning; n-step methods and tree backup; planning with models, prioritised sweeping, trajectory sampling, rollouts and Monte Carlo tree search.
Marks: 17–18 in the end-semester exam
- Temporal-difference prediction
- Sarsa and Q-learning
- Expected Sarsa and double learning
- n-step bootstrapping
- Planning with models, prioritised sweeping and MCTS
Unit 4: Approximate Solution Methods
7 hoursValue-function approximation, stochastic and semi-gradient methods, linear methods, feature construction, nonlinear approximation, LSTD, memory- and kernel-based methods; episodic semi-gradient control; the deadly triad; eligibility traces, TD(λ), Sarsa(λ), Watkins Q(λ).
Marks: 17–18 in the end-semester exam
- Value-function approximation
- Semi-gradient methods and linear function approximation
- On-policy control with approximation
- Off-policy approximation and the deadly triad
- Eligibility traces and TD(λ)
Unit 5: Policy Gradient Methods
7 hoursPolicy parameterisation and the policy gradient theorem; REINFORCE and baselines; actor–critic, the advantage function, A2C and A3C; value-based vs policy-based methods.
Marks: 17–18 in the end-semester exam
- Policy gradient theorem
- REINFORCE and baselines
- Actor–critic methods
Unit 6: Deep Reinforcement Learning
7 hoursDQN, experience replay and target networks, double DQN, prioritised replay, dueling networks; DDPG and PPO; safety and ethics; multi-agent RL, offline RL and RLHF.
Marks: 17–18 in the end-semester exam
- Deep Q-networks
- DQN improvements: double DQN, prioritised replay and dueling
- Continuous actions: DDPG and PPO
- Trends: multi-agent RL, offline RL and RLHF
Unit contents are summarised from the official SPPU syllabus . AiCanCode.org is not affiliated with Savitribai Phule Pune University.