MP #4 - Deep RL methods
Download [attachment]
Late Policy
- You have free 4 late days in total for all 4 homework assignments.
- You can use late days for any assignments, in whole-day increments (i.e., one day being the minimum unit). A late day extends the deadline by 24 hours.
- Once you have used all 4 late days, the penalty is 10% for each additional late day (until 0 points left).
Bonus Policy
- The highest points you can get in each assignment is capped at 100, i.e., final score = min(score after bonus, 100).
- We will not cap at the question level, but the entire assignment level.
This assignment is designed for you to gain hands‑on experience with deep RL using neural‑network‑based policy and value functions. You will first study DQN and Policy Gradient (PG), then implement and evaluate Double DQN and Actor–Critic (AC). In this assignment, we will use the CartPole‑v1 environment in OpenAI’s gymnasium (https://gymnasium.farama.org/environments/classic_control/cart_pole/).
- Part 1: Basics of deep RL algorithms (60 points)
- Part 2: Fine‑tuning and comparison (40 points)
- Bonus: Prioritized Experience Replay or Proximal Policy Optimization (10 points)
Details are in the starter code’s README.md. The report must be in PDF format, and simply name your submission by your student ID dash assignment1, e.g., “2026000000-assignment4.PDF”.
Part 1: Basics of Deep RL Algorithms (60 points)
The README.md consists of a tutorial that walks through the complete implementation of two baseline algorithms:
- Deep Q‑Network (DQN) in
DQN.py - Policy Gradient (PG / REINFORCE) in
PG.py
Read and understand these baseline implementations first, then build the following on top of them:
- Double DQN (DDQN) — modify the provided DQN with the double Q-learning update
- Actor–Critic (AC) — replacing the return in PG with the advantage
Deliverables:
- (20 points - NN parameterization) Design and optimize the neural networks (QNet and PolicyNet). State clearly the input and output of each network and how they differ from their tabular counterparts in classic MDPs.
- (20 points - Double DQN) Based on the provided
DQNAgent, elaborate the design of DQN, including its optimization objective, replay buffer, and target network. Then explain the double Q-learning modification and paste your implementation. - (20 points - Actor–Critic) Based on the provided
PGAgent, elaborate the deduction of the policy gradient formula and the mathematical form of REINFORCE. Then explain the actor–critic design and how the advantage estimate replaces the return, highlighting its differences from PG. Paste your implementation.
Part 2: Fine-tuning and Analyses (40 points)
Since introducing deep NN for approximating policy / value functions, tuning becomes a key factor influencing outcoming performance.
Deliverables:
- (10 points - Replay Buffer) Finetune the replay buffer settings in DQN/DDQN. Analyze the effect of different replay buffer sizes and batch sizes on final performance.
- (10 points - Tuning) For each algorithm, tune on at least one hyperparameter (hidden size, learning rate, etc). Report their performances in different hyperparameter settings.
- (20 points - Comparison) Plot the rewards of your tuned algorithms. Compare their pros and cons on final performance and convergence speed. Clearly summarize all the experiments you conducted.
Bonus: Prioritized Experience Replay & Proximal Policy Optimization (10 points)
Choose one of the following algorithm to implement (bonus points added on top of the 100-point total):
- Prioritized Experience Replay (PER): replace the uniform sampling in
ReplayBuffer.pywith priority-based sampling, and compare it against the plain replay buffer. - Proximal Policy Optimization (PPO): formalize its training objective, implement it, and compare it with your Actor–Critic. Think about and discuss this question: is it a value-based or policy-based method?
Reference
- OpenAI gymnasium: https://gymnasium.farama.org/
- Arcade learning environment: https://ale.farama.org/environments/
- DQN: https://arxiv.org/pdf/1312.5602
- PER: https://arxiv.org/pdf/1511.05952
- DDQN: https://arxiv.org/pdf/1509.06461
- REINFORCE: https://link.springer.com/article/10.1007/BF00992696
- PPO: https://arxiv.org/pdf/1707.06347