SPPU BE Computer syllabus

SPPU · Final Year Computer Engineering · 2019 Course (Revised)

Reinforcement Learning

410653D · Elective VI · Semester VIII · 3 credits · In-semester 30 + end-semester 70

One of the Elective VI options.

All notes for this subject
  1. Unit 1: Introduction

    7 hours

    Elements, limits and scope, Tic-Tac-Toe; multi-armed bandits, action-value methods, incremental implementation, non-stationary problems, optimistic initial values, UCB, gradient bandits, associative search; finite MDPs, goals, rewards, returns and episodes, policies and value functions, optimality.

    Marks: 15 in the in-semester exam

    • Reinforcement learning: elements and scope
    • Multi-armed bandits
    • Action selection: optimistic values, UCB and gradient bandits
    • Finite Markov decision processes
    • Policies, value functions and optimality
  2. Unit 2: Dynamic Programming and Monte Carlo Methods

    7 hours

    Policy evaluation, improvement and iteration, value iteration, asynchronous DP, generalised policy iteration; Monte Carlo prediction and control, off-policy control, importance sampling.

    Marks: 15 in the in-semester exam

    • Policy evaluation and policy iteration
    • Value iteration and generalised policy iteration
    • Monte Carlo prediction and control
    • Off-policy learning and importance sampling
  3. Unit 3: Temporal Difference Learning and Bootstrapping

    7 hours

    TD prediction, Sarsa, Q-learning, expected Sarsa, double learning; n-step methods and tree backup; planning with models, prioritised sweeping, trajectory sampling, rollouts and Monte Carlo tree search.

    Marks: 17–18 in the end-semester exam

    • Temporal-difference prediction
    • Sarsa and Q-learning
    • Expected Sarsa and double learning
    • n-step bootstrapping
    • Planning with models, prioritised sweeping and MCTS
  4. Unit 4: Approximate Solution Methods

    7 hours

    Value-function approximation, stochastic and semi-gradient methods, linear methods, feature construction, nonlinear approximation, LSTD, memory- and kernel-based methods; episodic semi-gradient control; the deadly triad; eligibility traces, TD(λ), Sarsa(λ), Watkins Q(λ).

    Marks: 17–18 in the end-semester exam

    • Value-function approximation
    • Semi-gradient methods and linear function approximation
    • On-policy control with approximation
    • Off-policy approximation and the deadly triad
    • Eligibility traces and TD(λ)
  5. Unit 5: Policy Gradient Methods

    7 hours

    Policy parameterisation and the policy gradient theorem; REINFORCE and baselines; actor–critic, the advantage function, A2C and A3C; value-based vs policy-based methods.

    Marks: 17–18 in the end-semester exam

    • Policy gradient theorem
    • REINFORCE and baselines
    • Actor–critic methods
  6. Unit 6: Deep Reinforcement Learning

    7 hours

    DQN, experience replay and target networks, double DQN, prioritised replay, dueling networks; DDPG and PPO; safety and ethics; multi-agent RL, offline RL and RLHF.

    Marks: 17–18 in the end-semester exam

    • Deep Q-networks
    • DQN improvements: double DQN, prioritised replay and dueling
    • Continuous actions: DDPG and PPO
    • Trends: multi-agent RL, offline RL and RLHF

Unit contents are summarised from the official SPPU syllabus . AiCanCode.org is not affiliated with Savitribai Phule Pune University.