{ "cells": [ { "cell_type": "markdown", "id": "f07761fc", "metadata": {}, "source": [ "# Class 4｜強化學習與模仿學習\n", "\n", "**開啟後只執行第一格「準備工具」**，其他格上課時從上往下逐格做（不要「全部執行」，有些格會先公布答案）。只改標著 🎛️、🎯 的地方。全部在 CPU 上跑，不用 GPU。\n", "\n", "**必做**（課堂約 45 分鐘）：A1、A2、B0、B1、B2、B4、C 冰湖。**選做**：B2b（加上「某種情況才給分」的規則）、D（從示範反推獎勵）。E 反思課後繳交。\n", "\n", "| 投影片 | Colab | 做什麼 | 類別 | 時間 |\n", "|---|---|---|---|---|\n", "| 回報（P24） | A1 | 手算回報：γ 大小有什麼差別 | 必做 | 4 分 |\n", "| Q 學習（P66） | A2 | 手算一次 Q 值更新 | 必做 | 4 分 |\n", "| 獎勵設計（P90） | B0、B1、B2 | 看獎勵怎麼算、三個陷阱，再設計自己的獎勵 | 必做 | 20 分 |\n", "| 規則型獎勵（P89） | B2b | 用選單加上「某種情況才給分」的規則 | 選做 | 8 分 |\n", "| 先模仿再強化（P112） | B4 | 能不能超越不完美的老師？ | 必做 | 6 分 |\n", "| 冰湖（P116） | C | 地板會滑時，AI 走什麼路？ | 必做 | 5 分 |\n", "| 逆強化學習（P106） | D | 看專家的示範，反推它在乎什麼 | 選做 | 8 分 |\n", "| | E | 反思 | 課後 | — |" ] }, { "cell_type": "code", "execution_count": null, "id": "1d2ffe6a", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 準備工具（執行一次，不用改；想看怎麼寫可以展開）\n", "\"\"\"Class 4 實作工具：8×8 坦克戰場（Q 學習、模仿學習）、冰湖。只用 numpy、matplotlib（Colab 已內建），不用 GPU。\"\"\"\n", "\"\"\"Class 4 E1：8×8 小戰場的坦克，用 Q 表學習。學生只改獎勵權重、γ、ε，看坦克學到什麼（包括鑽漏洞）。\"\"\"\n", "import numpy as np\n", "\n", "MAP = [\"........\",\n", " \"..#.....\",\n", " \"..#..##.\",\n", " \"........\",\n", " \".##..#..\",\n", " \".....#..\",\n", " \"...#....\",\n", " \"........\"]\n", "WALL = np.array([[c == \"#\" for c in r] for r in MAP])\n", "N = 8\n", "START, ENEMY = (7, 0), (1, 6) # (列, 欄)：坦克從左下出發，敵人固定在右上\n", "DIRS = [(-1, 0), (0, 1), (1, 0), (0, -1)] # 上 右 下 左\n", "ACTIONS = [\"前進\", \"左轉\", \"右轉\", \"開砲\", \"不動\"]\n", "ENEMY_HP, TANK_HP, MAX_STEPS = 3, 3, 60\n", "ENEMY_RANGE, ENEMY_ACC = 3, 0.35 # 敵人只打 3 格內、同一直線、沒有牆擋的目標，命中率 35%\n", "\n", "REWARD_KEYS = [\"擊殺\", \"命中敵人\", \"被打中\", \"每步存活\", \"開砲\", \"靠近敵人\", \"遠離敵人\", \"陣亡\"]\n", "\n", "\n", "def line_clear(a, b):\n", " \"\"\"a、b 在同一列或同一欄，而且中間沒有牆\"\"\"\n", " if a[0] == b[0]:\n", " lo, hi = sorted((a[1], b[1])); return not WALL[a[0], lo + 1:hi].any()\n", " if a[1] == b[1]:\n", " lo, hi = sorted((a[0], b[0])); return not WALL[lo + 1:hi, a[1]].any()\n", " return False\n", "\n", "\n", "def facing_enemy(pos, d):\n", " dr, dc = DIRS[d]; r, c = pos\n", " while True:\n", " r += dr; c += dc\n", " if not (0 <= r < N and 0 <= c < N) or WALL[r, c]: return False\n", " if (r, c) == ENEMY: return True\n", "\n", "\n", "def dist(p): return abs(p[0] - ENEMY[0]) + abs(p[1] - ENEMY[1])\n", "\n", "\n", "class Arena:\n", " def reset(self, rng):\n", " self.rng = rng; self.pos, self.d, self.hp, self.ehp, self.t = START, 0, TANK_HP, ENEMY_HP, 0\n", " return self.state()\n", "\n", " def state(self):\n", " return ((self.pos[0] * N + self.pos[1]) * 4 + self.d) * TANK_HP * ENEMY_HP + (self.hp - 1) * ENEMY_HP + (self.ehp - 1)\n", "\n", " def step(self, a, R):\n", " ev = dict(fired=False, hit=False, damaged=False, closer=False, farther=False, enemy_fired=False)\n", " before = dist(self.pos)\n", " if a == 0:\n", " r, c = self.pos[0] + DIRS[self.d][0], self.pos[1] + DIRS[self.d][1]\n", " if 0 <= r < N and 0 <= c < N and not WALL[r, c] and (r, c) != ENEMY: self.pos = (r, c)\n", " elif a == 1: self.d = (self.d - 1) % 4\n", " elif a == 2: self.d = (self.d + 1) % 4\n", " elif a == 3:\n", " ev[\"fired\"] = True\n", " if facing_enemy(self.pos, self.d): self.ehp -= 1; ev[\"hit\"] = True\n", " ev[\"closer\"] = dist(self.pos) < before; ev[\"farther\"] = dist(self.pos) > before\n", " if self.ehp > 0 and dist(self.pos) <= ENEMY_RANGE and line_clear(self.pos, ENEMY): # 在射程內：敵人開砲（亂數用法不變）\n", " ev[\"enemy_fired\"] = True\n", " if self.rng.random() < ENEMY_ACC: self.hp -= 1; ev[\"damaged\"] = True\n", " self.t += 1\n", " win, dead = self.ehp <= 0, self.hp <= 0\n", " r = (R[\"擊殺\"] * win + R[\"命中敵人\"] * ev[\"hit\"] + R[\"被打中\"] * ev[\"damaged\"] + R[\"每步存活\"] * (not dead)\n", " + R[\"開砲\"] * ev[\"fired\"] + R[\"靠近敵人\"] * ev[\"closer\"] + R.get(\"遠離敵人\", 0) * ev[\"farther\"] + R[\"陣亡\"] * dead)\n", " if R.get(\"自訂\"): # 學生自己寫的獎勵函數：拿到這一步的前後資訊，回傳要「加」的分數\n", " r += R[\"自訂\"](dict(ev, win=win, dead=dead, before_dist=before, after_dist=dist(self.pos), my_hp=self.hp, enemy_hp=self.ehp, step=self.t, aimed=facing_enemy(self.pos, self.d)))\n", " done = win or dead or self.t >= MAX_STEPS\n", " return self.state(), r, done, dict(ev, win=win, dead=dead)\n", "\n", "\n", "N_STATES = N * N * 4 * TANK_HP * ENEMY_HP\n", "\n", "\n", "def train(R, gamma=0.95, eps=0.05, episodes=8000, seed=0):\n", " \"\"\"Q-learning：Q[s,a] ← Q[s,a] + α（r + γ·max Q[s'] − Q[s,a]）。\n", " 探索：一開始每一步都亂試（ε＝1），前 60% 的局數慢慢降到 eps；學習率 α 從 0.3 慢慢降到 0.05。\n", " 回傳 Q 表與每 200 局的平均回報（學習曲線）\"\"\"\n", " rng = np.random.default_rng(seed); Q = np.zeros((N_STATES, 5)); env = Arena(); curve = []; tot = 0\n", " for ep in range(episodes):\n", " e = max(eps, 1.0 - ep / (episodes * 0.6)); alpha = 0.3 - 0.25 * ep / episodes\n", " s = env.reset(rng); done = False; G = 0\n", " while not done:\n", " a = int(rng.integers(5)) if rng.random() < e else int(np.argmax(Q[s] + rng.random(5) * 1e-6))\n", " s2, r, done, info = env.step(a, R); G += r\n", " end = info[\"win\"] or info[\"dead\"] # 真的結束才不看未來；時間到只是被打斷\n", " Q[s, a] += alpha * (r + (0 if end else gamma * Q[s2].max()) - Q[s, a]); s = s2\n", " tot += G\n", " if (ep + 1) % 200 == 0: curve.append(tot / 200); tot = 0\n", " return Q, curve\n", "\n", "\n", "PREF = np.array([0, 0, 0, 0, 1e-6]) # 分數一樣時偏好「不動」（看得出它其實什麼都沒學）\n", "\n", "\n", "def play(Q, seed=123):\n", " \"\"\"用學好的 Q 表（不再亂試）玩一局，回傳每一步的畫面資料\"\"\"\n", " rng = np.random.default_rng(seed); env = Arena(); s = env.reset(rng); frames = [dict(pos=env.pos, d=env.d, hp=env.hp, ehp=env.ehp, a=None, ev={})]\n", " done = False\n", " while not done:\n", " a = int(np.argmax(Q[s] + PREF)); s, r, done, info = env.step(a, ZERO)\n", " frames.append(dict(pos=env.pos, d=env.d, hp=env.hp, ehp=env.ehp, a=a, ev=info))\n", " return frames\n", "\n", "\n", "def evaluate(Q, n=200, seed=7):\n", " rng = np.random.default_rng(seed); env = Arena(); wins = steps = shots = 0; dead = 0\n", " for _ in range(n):\n", " s = env.reset(rng); done = False\n", " while not done:\n", " a = int(np.argmax(Q[s] + PREF)); s, _, done, info = env.step(a, ZERO); shots += a == 3\n", " wins += info[\"win\"]; dead += info[\"dead\"]; steps += env.t\n", " return dict(win=wins / n, dead=dead / n, timeout=1 - (wins + dead) / n, steps=steps / n, shots=shots / n)\n", "\n", "\n", "ZERO = {k: 0 for k in REWARD_KEYS}\n", "PRESETS = {\n", " \"只獎勵開砲\": dict(ZERO, 開砲=1),\n", " \"只獎勵存活\": dict(ZERO, 每步存活=1),\n", " \"只獎勵靠近敵人\": dict(ZERO, 靠近敵人=1),\n", " \"只看輸贏\": dict(ZERO, 擊殺=10, 陣亡=-10),\n", " \"設計得不錯\": dict(ZERO, 擊殺=10, 命中敵人=1, 被打中=-1, 每步存活=-0.1, 陣亡=-10),\n", "}\n", "\n", "\"\"\"Class 4 補充實驗：Policy Gradient（REINFORCE，有無 baseline）與模仿學習（Behavior Cloning vs DAgger），同一個 8×8 坦克戰場。\"\"\"\n", "import numpy as np\n", "\n", "\n", "def softmax(x): e = np.exp(x - x.max()); return e / e.sum()\n", "\n", "def reinforce(R, episodes=8000, lr=0.1, gamma=0.95, baseline=True, seed=0, log_every=200):\n", " \"\"\"表格式 softmax 策略：θ[s] 是 5 個動作的分數。每局結束後，每一步的 log π(a|s) 乘上 (Gₜ − b) 往上推\"\"\"\n", " rng = np.random.default_rng(seed); th = np.zeros((N_STATES, 5)); env = Arena(); b = 0.0; curve = []; wins = 0\n", " for ep in range(episodes):\n", " s = env.reset(rng); done = False; traj = []\n", " while not done:\n", " p = softmax(th[s]); a = int(rng.choice(5, p=p)); s2, r, done, info = env.step(a, R); traj.append((s, a, r)); s = s2\n", " wins += info['win']; G = 0; rets = []\n", " for (_, _, r) in reversed(traj): G = r + gamma * G; rets.append(G)\n", " rets = rets[::-1]\n", " for (s_, a_, _), Gt in zip(traj, rets):\n", " p = softmax(th[s_]); grad = -p; grad[a_] += 1 # ∂ log π(a|s) / ∂θ[s]\n", " th[s_] += lr * (Gt - (b if baseline else 0)) * grad\n", " if baseline: b += 0.05 * (rets[0] - b) # baseline：最近回報的移動平均\n", " if (ep + 1) % log_every == 0: curve.append(wins / log_every); wins = 0\n", " return th, curve\n", "\n", "def greedy_policy_from(th):\n", " return lambda s: int(np.argmax(th[s]))\n", "\n", "def rollout(policy, n=200, noise=0.0, seed=7, expert=None):\n", " \"\"\"執行策略；noise：每一步有這個機率被「推一下」做隨機動作（模擬沒看過的情況）。expert 不為 None 時，回傳走過的狀態\"\"\"\n", " rng = np.random.default_rng(seed); env = Arena(); wins = 0; visited = []\n", " for _ in range(n):\n", " s = env.reset(rng); done = False\n", " while not done:\n", " visited.append(s)\n", " a = int(rng.integers(5)) if rng.random() < noise else policy(s)\n", " s, _, done, info = env.step(a, ZERO)\n", " wins += info['win']\n", " return wins / n, visited\n", "\n", "def bc_table(expert, states):\n", " \"\"\"Behavior Cloning（表格版）：看過的狀態照抄專家動作，沒看過的狀態只能亂猜（固定一個隨機動作）\"\"\"\n", " rng = np.random.default_rng(1); table = {}\n", " for s in states: table[s] = expert(s)\n", " guess = rng.integers(5, size=N_STATES)\n", " return (lambda s: table.get(s, int(guess[s]))), table\n", "\n", "def train_q_from(R, Q0, episodes, seed, eps_start=1.0, gamma=0.95):\n", " \"\"\"Q 學習，但可以從一張已經有東西的 Q 表開始（模仿起點）\"\"\"\n", " rng = np.random.default_rng(seed); Q = Q0.copy(); env = Arena()\n", " for ep in range(episodes):\n", " e = max(0.05, eps_start - (eps_start - 0.05) * ep / (episodes * 0.6)); a_ = 0.3 - 0.25 * ep / episodes\n", " s = env.reset(rng); done = False\n", " while not done:\n", " a = int(rng.integers(5)) if rng.random() < e else int(np.argmax(Q[s] + rng.random(5) * 1e-6))\n", " s2, r, done, info = env.step(a, R); end = info['win'] or info['dead']\n", " Q[s, a] += a_ * (r + (0 if end else gamma * Q[s2].max()) - Q[s, a]); s = s2\n", " return Q\n", "\n", "def weak_expert(R=None):\n", " \"\"\"「不夠好」的老師：只練 1500 局的 Q 學習（勝率約 68%），像期中不完美的規則策略\"\"\"\n", " R = R or PRESETS['設計得不錯']; Qe, _ = train(R, seed=6, episodes=1500)\n", " return lambda s: int(np.argmax(Qe[s] + PREF)), Qe\n", "\n", "def imitation_then_rl(budgets=(250, 500, 1000, 2000, 4000), seeds=5, demo_games=50):\n", " \"\"\"AlphaGo 的兩階段：先模仿老師（示範過的狀態，老師的動作先給 1 分），再用強化學習繼續練\"\"\"\n", " R = PRESETS['設計得不錯']; expert, Qe = weak_expert(R)\n", " _, demo = rollout(expert, n=demo_games, seed=3); demo = set(demo)\n", " init = np.zeros((N_STATES, 5))\n", " for s in demo: init[s, expert(s)] = 1.0\n", " bc, _ = bc_table(expert, demo)\n", " out = dict(expert=rollout(expert)[0], bc=rollout(bc)[0], budgets=list(budgets), scratch=[], imit=[])\n", " for ep in budgets:\n", " out['scratch'].append(float(np.mean([evaluate(train_q_from(R, np.zeros_like(init), ep, s))['win'] for s in range(seeds)])))\n", " out['imit'].append(float(np.mean([evaluate(train_q_from(R, init, ep, s, eps_start=0.3))['win'] for s in range(seeds)])))\n", " return out\n", "\n", "\"\"\"Class 4 課堂練習：冰湖（會滑倒的格子世界，Q 學習）與 21 點（蒙地卡羅學習什麼時候要牌）。純 numpy，不用安裝套件。\"\"\"\n", "import numpy as np\n", "\n", "# ---------------- 冰湖 ----------------\n", "LAKE = [\"S...\",\n", " \".#.#\",\n", " \"...#\",\n", " \"#..G\"] # S 起點、G 寶藏、# 冰洞（掉下去就結束）\n", "LAKE_DIRS = [(0, -1), (1, 0), (0, 1), (-1, 0)] # 左 下 右 上\n", "LAKE_ARROW = ['←', '↓', '→', '↑']\n", "\n", "def lake_step(s, a, slip, rng):\n", " \"\"\"往 a 走；有 slip 的機率滑到左右兩側（和想走的方向垂直）\"\"\"\n", " if rng.random() < slip: a = (a + rng.choice([-1, 1])) % 4\n", " r, c = divmod(s, 4); dr, dc = LAKE_DIRS[a]\n", " r, c = min(max(r + dr, 0), 3), min(max(c + dc, 0), 3); s2 = r * 4 + c; cell = LAKE[r][c]\n", " return s2, (1.0 if cell == 'G' else 0.0), cell in '#G'\n", "\n", "def lake_train(slip=0.33, gamma=0.95, episodes=4000, step_cost=0.0, seed=0):\n", " rng = np.random.default_rng(seed); Q = np.zeros((16, 4))\n", " for ep in range(episodes):\n", " eps = max(0.05, 1 - ep / (episodes * 0.6)); alpha = 0.3 - 0.25 * ep / episodes; s = 0\n", " for _ in range(100):\n", " a = int(rng.integers(4)) if rng.random() < eps else int(np.argmax(Q[s] + rng.random(4) * 1e-6))\n", " s2, r, done = lake_step(s, a, slip, rng); r -= step_cost\n", " Q[s, a] += alpha * (r + (0 if done else gamma * Q[s2].max()) - Q[s, a]); s = s2\n", " if done: break\n", " return Q\n", "\n", "def lake_eval(Q, slip, n=2000, seed=1):\n", " rng = np.random.default_rng(seed); win = steps = 0\n", " for _ in range(n):\n", " s = 0\n", " for t in range(100):\n", " s, r, done = lake_step(s, int(np.argmax(Q[s])), slip, rng)\n", " if done: break\n", " win += r > 0; steps += t + 1\n", " return win / n, steps / n\n", "\n", "# ---------------- 21 點 ----------------\n", "def draw(rng): return min(int(rng.integers(1, 14)), 10) # A=1、2～10、J Q K=10\n", "\n", "def hand_value(cards):\n", " s = sum(cards); usable = 1 in cards and s + 10 <= 21\n", " return (s + 10 if usable else s), usable\n", "\n", "def bj_episode(policy, rng, explore=0.0):\n", " \"\"\"玩一局：policy(總點數, 莊家明牌, 有沒有可用的 A) → 1 要牌 / 0 停。回傳 (經過的狀態與動作, 輸贏 +1/0/−1)\"\"\"\n", " me = [draw(rng), draw(rng)]; dealer = [draw(rng), draw(rng)]; traj = []\n", " while True:\n", " v, ace = hand_value(me)\n", " if v > 21: return traj, -1\n", " if v < 12: me.append(draw(rng)); continue # 11 點以下一定要牌（不會爆）\n", " st = (v, dealer[0], ace)\n", " a = int(rng.integers(2)) if rng.random() < explore else policy(*st)\n", " traj.append((st, a))\n", " if a == 0: break\n", " me.append(draw(rng))\n", " while hand_value(dealer)[0] < 17: dealer.append(draw(rng)) # 莊家：17 點以下一定要牌\n", " mv, dv = hand_value(me)[0], hand_value(dealer)[0]\n", " return traj, (1 if dv > 21 or mv > dv else 0 if mv == dv else -1)\n", "\n", "def bj_learn(episodes=200000, seed=0):\n", " \"\"\"蒙地卡羅：每個 (狀態, 動作) 的價值＝玩到結束的平均輸贏。用 ε-greedy 一邊玩一邊改策略\"\"\"\n", " rng = np.random.default_rng(seed); Qs = np.zeros((22, 11, 2, 2)); N = np.zeros_like(Qs)\n", " policy = lambda v, d, ace: int(Qs[v, d, int(ace), 1] > Qs[v, d, int(ace), 0])\n", " for ep in range(episodes):\n", " traj, G = bj_episode(policy, rng, explore=0.1)\n", " for (v, d, ace), a in traj:\n", " N[v, d, int(ace), a] += 1; Qs[v, d, int(ace), a] += (G - Qs[v, d, int(ace), a]) / N[v, d, int(ace), a] # 平均（Q 學習那頁 α = 1/n 的版本）\n", " return Qs, policy\n", "\n", "def bj_eval(policy, n=50000, seed=1):\n", " rng = np.random.default_rng(seed); res = [bj_episode(policy, rng)[1] for _ in range(n)]\n", " return float(np.mean([r == 1 for r in res])), float(np.mean([r == 0 for r in res])), float(np.mean(res))\n", "\n", "def bj_threshold_policy(k):\n", " \"\"\"人類常見的簡單規則：總點數 < k 就要牌\"\"\"\n", " return lambda v, d, ace: int(v < k)\n", "\n", "# ======================= Class 4 實作工具（顯示、檢查、排行榜） =======================\n", "import json as _json, time as _time\n", "import numpy as np\n", "import matplotlib, matplotlib.pyplot as plt\n", "from matplotlib import animation, patches\n", "from IPython.display import HTML, display\n", "import logging, os, subprocess, sys\n", "logging.getLogger('matplotlib.font_manager').setLevel(logging.ERROR) # 不要印一大堆找不到字型的警告\n", "def _cjk_font():\n", " \"\"\"Colab 預設沒有中文字型：自動安裝 Noto CJK，圖上的中文才不會變方框\"\"\"\n", " from matplotlib import font_manager\n", " ttc = '/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc'\n", " if 'google.colab' in sys.modules and not os.path.exists(ttc):\n", " subprocess.run(['apt-get', '-qq', 'install', '-y', 'fonts-noto-cjk'], capture_output=True)\n", " if os.path.exists(ttc): font_manager.fontManager.addfont(ttc)\n", "_cjk_font()\n", "plt.rcParams['font.family'] = ['Noto Sans CJK JP', 'Noto Sans CJK TC', 'PingFang TC', 'Heiti TC', 'Arial Unicode MS', 'DejaVu Sans']\n", "plt.rcParams['axes.unicode_minus'] = False\n", "NAVY, ORANGE, GREEN, RED, GRAY = '#073F81', '#ED802C', '#238678', '#C0392B', '#B8C4D0'\n", "\n", "class Stop(Exception):\n", " \"\"\"停下這一格，只顯示一句話。\"\"\"\n", " def _render_traceback_(self): return [str(self)]\n", "\n", "# ---------- 手算 ----------\n", "# ---------- A：手算題（逐步提示、答對或勾「看解答」才公布、記錄進度） ----------\n", "_tries, _done = {}, {}\n", "_ALL = {'A1': ['A1-1', 'A1-2', 'A1-3'], 'A2': ['A2-1', 'A2-2']}\n", "\n", "def _progress():\n", " print('📋 進度：' + '　'.join(f\"{g} \" + ''.join('✅' if _done.get(k) else '⬜' for k in ks) for g, ks in _ALL.items()))\n", "\n", "def _judge(key, answer, truth, hints, solution, show=False, tol=0.01):\n", " \"\"\"answer：數字或算式（例如 0.8**4*10）。答錯依次數給提示；答對或 show=True 才公布 solution\"\"\"\n", " if show:\n", " print('📖 解答：'); [print('　' + l) for l in solution]; _progress(); return None\n", " if answer is None or answer == '':\n", " print('👉 先用紙筆算，把答案（數字或算式）填在 ANSWER，再按 ▶。'); _progress(); return None\n", " try: v = float(answer)\n", " except (TypeError, ValueError): print('⛔ ANSWER 要是數字或算式，例如 4.096 或 0.8**4*10（次方用 **）。'); return None\n", " if abs(v - truth) <= max(tol, abs(truth) * 0.005):\n", " _done[key] = True; print(f'✅ 答對！{v:g}'); [print('　' + l) for l in solution]\n", " else:\n", " n = _tries[key] = _tries.get(key, 0) + 1\n", " print(f'❌ 你寫 {v:g}，' + ('太大了' if v > truth else '太小了') + f'（第 {n} 次）')\n", " print('💡 提示：' + hints[min(n, len(hints)) - 1])\n", " if n >= len(hints): print('　還是卡住？和旁邊的同學討論，或對照投影片的算法再算一次。')\n", " _progress(); return v == truth\n", "\n", "def check_return(rewards, gamma, answer=None, key='A1', show=False):\n", " \"\"\"手算回報 G′ = r₁ + γ·r₂ + γ²·r₃ + …\"\"\"\n", " G = sum(r * gamma ** k for k, r in enumerate(rewards))\n", " terms = ' + '.join(f'{gamma:g}^{k}×{r:g}' if k else f'{r:g}' for k, r in enumerate(rewards))\n", " blanks = ' + '.join(f'γ^{k}×{r:g}' if k else f'{r:g}' for k, r in enumerate(rewards))\n", " print(f'題目：獎勵 {list(rewards)}，γ = {gamma:g}')\n", " return _judge(key, answer, G,\n", " [f'第 1 步不打折，第 2 步乘 γ 一次，第 k 步乘 γ 的 (k−1) 次方（共 {len(rewards)} 步）。',\n", " f'把數字代進去：G′ = {blanks}，γ = {gamma:g}。'],\n", " [f'G′ = {terms}', f'　= {G:.4g}'], show)\n", "\n", "def check_q(q_old, reward, best_next, alpha, gamma, answer=None, key='A2', show=False):\n", " \"\"\"手算一次 Q 值更新：新 Q = 舊 Q + α ×（獎勵 + γ × 下一格最好動作的 Q − 舊 Q）\"\"\"\n", " target = reward + gamma * best_next; new = q_old + alpha * (target - q_old)\n", " print(f'題目：舊 Q = {q_old:g}，這一步獎勵 = {reward:g}，下一格最好動作的 Q = {best_next:g}，α = {alpha:g}，γ = {gamma:g}')\n", " return _judge(key, answer, new,\n", " ['分兩步：先算「目標 = 獎勵 + γ × 下一格最好動作的 Q」，再算「新 Q = 舊 Q + α ×（目標 − 舊 Q）」。',\n", " f'目標 = {reward:g} + {gamma:g} × {best_next:g} = {target:.4g}；接著算 {q_old:g} + {alpha:g} × ({target:.4g} − {q_old:g})。'],\n", " [f'目標 = {reward:g} + {gamma:g} × {best_next:g} = {target:.4g}', f'新 Q = {q_old:g} + {alpha:g} × ({target:.4g} − {q_old:g}) = {new:.4g}'], show)\n", "\n", "# ---------- B2b：選「在什麼情況下」給分（不用寫程式） ----------\n", "RULES = {\n", " '（不用）': None,\n", " '對準敵人的每一步': lambda i: i['aimed'],\n", " '開砲但沒打中': lambda i: i['fired'] and not i['hit'],\n", " '血只剩 1 時靠近敵人': lambda i: i['my_hp'] == 1 and i['after_dist'] < i['before_dist'],\n", " '血只剩 1 時遠離敵人': lambda i: i['my_hp'] == 1 and i['after_dist'] > i['before_dist'],\n", " '離敵人 3 格以內的每一步': lambda i: i['after_dist'] <= 3,\n", " '離敵人超過 3 格時靠近': lambda i: i['before_dist'] > 3 and i['after_dist'] < i['before_dist'],\n", "}\n", "BASES = {'只看輸贏': dict(擊殺=10, 陣亡=-10), '只看輸贏＋位能型提示（靠近 +1、遠離 −1）': dict(擊殺=10, 陣亡=-10, 靠近敵人=1, 遠離敵人=-1),\n", " '設計得不錯（命中、被打中、每步都有分）': dict(擊殺=10, 命中敵人=1, 被打中=-1, 每步存活=-0.1, 陣亡=-10)}\n", "\n", "def rule_reward(base, rules):\n", " \"\"\"base：BASES 的名稱；rules：[(規則名稱, 分數), …]。回傳可以直接給 tank_lab 的獎勵 dict 與說明文字\"\"\"\n", " use = [(n, RULES[n], v) for n, v in rules if RULES.get(n) and v]\n", " def extra(info): return sum(v * fn(info) for _, fn, v in use)\n", " extra.__name__ = '；'.join(f'{n} {v:+g}' for n, _, v in use) or '沒有加規則'\n", " R = {k: 0 for k in [\"擊殺\", \"命中敵人\", \"被打中\", \"每步存活\", \"開砲\", \"靠近敵人\", \"遠離敵人\", \"陣亡\"]}; R.update(BASES[base])\n", " if use: R['自訂'] = extra\n", " return R, f'{base}＋{extra.__name__}'\n", "\n", "# ---------- E1：坦克獎勵設計 ----------\n", "tank_runs = []\n", "\n", "def _draw_arena(ax):\n", " ax.set_xlim(-.5, N - .5); ax.set_ylim(N - .5, -.5); ax.set_aspect('equal'); ax.set_xticks([]); ax.set_yticks([])\n", " ax.add_patch(patches.Rectangle((-.5, -.5), N, N, fc='#F4F7FB', ec=GRAY))\n", " for r in range(N):\n", " for c in range(N):\n", " if WALL[r, c]: ax.add_patch(patches.Rectangle((c - .5, r - .5), 1, 1, fc='#5A6670', ec='white'))\n", " for r, c in [(ENEMY[0] + dr * k, ENEMY[1] + dc * k) for dr, dc in DIRS for k in range(1, ENEMY_RANGE + 1)]:\n", " if 0 <= r < N and 0 <= c < N and not WALL[r, c] and line_clear((r, c), ENEMY):\n", " ax.add_patch(patches.Rectangle((c - .5, r - .5), 1, 1, fc='#F6DFD8', ec='none', zorder=1.5))\n", "\n", "def tank_movie(frames, title=''):\n", " \"\"\"坦克動畫：開砲的那一步多畫 3 格，砲彈從砲口飛出去；敵人開砲時，藍色砲彈飛向坦克\"\"\"\n", " fig, ax = plt.subplots(figsize=(4.6, 4.9)); _draw_arena(ax)\n", " enemy = patches.Rectangle((ENEMY[1] - .35, ENEMY[0] - .35), .7, .7, fc=NAVY, zorder=3); ax.add_patch(enemy)\n", " tank = patches.Polygon([[0, 0]] * 3, closed=True, fc=ORANGE, ec='#8A4A10', zorder=4); ax.add_patch(tank)\n", " my_shell, = ax.plot([], [], 'o', color=RED, ms=9, zorder=6); en_shell, = ax.plot([], [], 'o', color='#2F6FB5', ms=9, zorder=6)\n", " boom, = ax.plot([], [], '*', color='#F2A100', mec=RED, mew=1.5, ms=26, zorder=7); boom2, = ax.plot([], [], '*', color='#F2A100', mec=RED, mew=1.5, ms=26, zorder=7); miss = ax.text(0, 0, '', fontsize=9, ha='center', va='center', color='#536B83', zorder=7)\n", " txt = ax.set_title('')\n", " def shot_end(r, c, dr, dc):\n", " rr, cc = r, c\n", " while True:\n", " nr, nc = rr + dr, cc + dc\n", " if not (0 <= nr < N and 0 <= nc < N) or WALL[nr, nc]: return rr + dr * .45, cc + dc * .45\n", " rr, cc = nr, nc\n", " if (rr, cc) == ENEMY: return rr, cc\n", " seq = [(k, 0, 1) for k in range(1)]\n", " for k in range(1, len(frames)):\n", " ev = frames[k]['ev']; n = 3 if (frames[k]['a'] == 3 or ev.get('enemy_fired')) else 1\n", " seq += [(k, j, n) for j in range(1, n + 1)]\n", " def upd(i):\n", " k, j, n = seq[i]; f = frames[k]; r, c = f['pos']; dr, dc = DIRS[f['d']]; ev = f['ev']; t = j / n\n", " tip = (c + dc * .42, r + dr * .42); back = [(c - dc * .3 + dr * .32, r - dr * .3 - dc * .32), (c - dc * .3 - dr * .32, r - dr * .3 + dc * .32)]\n", " tank.set_xy([tip, *back]); enemy.set_alpha(0.25 if f['ehp'] <= 0 else 1)\n", " my_shell.set_data([], []); en_shell.set_data([], []); boom.set_data([], []); boom2.set_data([], []); miss.set_text('')\n", " if f['a'] == 3: # 我方砲彈：砲口 → 撞牆或打中敵人\n", " er, ec_ = shot_end(r, c, dr, dc); sr, sc = r + dr * .45, c + dc * .45\n", " if t < 1: my_shell.set_data([sc + (ec_ - sc) * t], [sr + (er - sr) * t])\n", " elif ev.get('hit'): boom.set_data([ec_], [er])\n", " if ev.get('enemy_fired'): # 敵人砲彈：敵人 → 坦克\n", " er, ec_ = ENEMY\n", " if t < 1: en_shell.set_data([ec_ + (c - ec_) * t], [er + (r - er) * t])\n", " elif ev.get('damaged'): boom2.set_data([c], [r])\n", " else: miss.set_position((c, r - .55)); miss.set_text('沒打中')\n", " act = '開始' if f['a'] is None else ACTIONS[f['a']]\n", " what = '、'.join(x for x, on in [('打中敵人', ev.get('hit')), ('被打中', ev.get('damaged')), ('敵人開砲沒打中', ev.get('enemy_fired') and not ev.get('damaged'))] if on)\n", " txt.set_text(f'{title}\\n第 {k} 步：{act}' + (f'（{what}）' if what else '') + f'｜我方 {\"♥\" * f[\"hp\"]}｜敵人 {max(f[\"ehp\"], 0)}')\n", " return tank, enemy, my_shell, en_shell, boom, boom2, miss, txt\n", " anim = animation.FuncAnimation(fig, upd, frames=len(seq), interval=150, blit=False)\n", " plt.close(fig); return HTML(anim.to_jshtml(default_mode='once'))\n", "\n", "def _diagnose(e, acts, R, gamma=0.95):\n", " from collections import Counter\n", " top, n = Counter(acts).most_common(1)[0] if acts else ('不動', 0)\n", " msgs = []\n", " if e['win'] >= 0.8: msgs.append(f'🟢 坦克真的學會打贏了：勝率 {e[\"win\"]:.0%}，平均 {e[\"steps\"]:.0f} 步結束。挑戰：讓步數再少一點。')\n", " if e['win'] < 0.2 and top == '開砲' and n > len(acts) * 0.6: msgs.append('🔴 鑽漏洞：它學會「原地一直開砲」刷分，根本不去打敵人。開砲不等於打贏。')\n", " if e['win'] < 0.2 and top == '不動' and n > len(acts) * 0.6:\n", " if R.get('每步存活', 0) > 0: msgs.append('🔴 鑽漏洞：只獎勵活著，最安全的做法就是躲著不出去。')\n", " elif gamma < 0.7: msgs.append(f'🔴 太短視：γ = {gamma:g} 時，十幾步外的勝利打折後幾乎是 0，走過去「不值得」，乾脆不動。')\n", " else: msgs.append('🔴 學不會：獎勵太少見。亂試時幾乎不可能贏，它從來沒拿過獎勵，不知道該往哪走。試試加上「命中敵人」的小獎勵。')\n", " if e['win'] < 0.2 and R.get('靠近敵人', 0) > 0 and top == '前進': msgs.append('🔴 鑽漏洞：它在繞路刷「靠近敵人」的分數——走遠一點再靠近，又能拿分，但不開打。')\n", " if 0.2 <= e['win'] < 0.8: msgs.append(f'🟡 有時贏、有時不贏（勝率 {e[\"win\"]:.0%}）。試試調 γ（看得夠遠嗎？）或改獎勵。')\n", " if not msgs: msgs.append('🟡 看動畫：它在做什麼？是你想要的嗎？')\n", " return msgs\n", "\n", "_R_ORDER = [\"擊殺\", \"陣亡\", \"命中敵人\", \"被打中\", \"每步存活\", \"開砲\", \"靠近敵人\", \"遠離敵人\"]\n", "\n", "def _reward_text(R):\n", " \"\"\"獎勵 dict → 白話：擊殺 +10、陣亡 −10、…（自訂規則接在後面）\"\"\"\n", " parts = [f'{k} {R[k]:+g}'.replace('-', '−') for k in _R_ORDER if R.get(k)]\n", " if callable(R.get('自訂')): parts.append(getattr(R['自訂'], '__name__', '自訂規則').replace('-', '−'))\n", " return '、'.join(parts) or '（全部 0）'\n", "\n", "def tank_lab(note, R, gamma=0.95, eps=0.05, episodes=8000, seeds=3):\n", " t0 = _time.time(); Qs, curves = [], []\n", " for s in range(seeds):\n", " Q, cv = train(R, gamma=gamma, eps=eps, episodes=episodes, seed=s); Qs.append(Q); curves.append(cv)\n", " ev = [evaluate(Q) for Q in Qs]; e = {k: float(np.mean([x[k] for x in ev])) for k in ev[0]}\n", " frames = play(Qs[0]); acts = [ACTIONS[f['a']] for f in frames[1:]]\n", " run = dict(note=note or '—', R=_reward_text(R), gamma=gamma, eps=eps, episodes=episodes, **e); tank_runs.append(run)\n", " print(f'回合 {len(tank_runs) - 1}｜{run[\"note\"]}')\n", " print(f'🎯 勝率 {e[\"win\"]:.0%}｜陣亡 {e[\"dead\"]:.0%}｜時間到還沒結束 {e[\"timeout\"]:.0%}｜平均 {e[\"steps\"]:.1f} 步｜開砲 {e[\"shots\"]:.1f} 次')\n", " for m in _diagnose(e, acts, R, gamma): print(m)\n", " print(f'　獎勵：{run[\"R\"]}')\n", " print(f'　設定：γ {gamma:g}、ε {eps:g}、訓練 {episodes} 局 × {seeds} 個種子（{_time.time() - t0:.1f} 秒）')\n", " display(tank_movie(frames, run['note']))\n", " fig, ax = plt.subplots(figsize=(7, 2.6), layout='constrained')\n", " for k, cv in enumerate(curves): ax.plot(np.arange(1, len(cv) + 1) * 200, cv, color=[ORANGE, NAVY, GREEN][k % 3], lw=1.8, label=f'種子 {k}')\n", " ax.set_xlabel('訓練局數'); ax.set_ylabel('每局平均獎勵'); ax.set_title('學習曲線：坦克拿到的獎勵（你定的分數，不一定是勝率）'); ax.legend(); ax.grid(alpha=.25); plt.show()\n", " tank_board(mark=len(tank_runs) - 1, top=3)\n", " return run\n", "\n", "def tank_board(mark=None, top=10):\n", " if not tank_runs: print('還沒有回合'); return\n", " order = sorted(range(len(tank_runs)), key=lambda i: (-tank_runs[i]['win'], tank_runs[i]['steps']))\n", " print(f'\\n🏆 排行榜前 {min(top, len(order))} 名（勝率高者優先，同勝率比步數少）')\n", " show = order[:top] + ([mark] if mark is not None and mark not in order[:top] else [])\n", " for i in show:\n", " r = tank_runs[i]; m = '　👈 這回合' if i == mark else ''\n", " print(f'{order.index(i) + 1:>2}. 勝率 {r[\"win\"]:.0%}｜{r[\"steps\"]:.1f} 步｜回合 {i}｜{r[\"note\"]}{m}')\n", " if top < len(order): print(f'　（共 {len(order)} 回合；往下找「完整排行榜」那一格可以看全部）')\n", "\n", "# ---------- D：從示範反推獎勵（逆強化學習） ----------\n", "IRL_FEATS = ['命中敵人', '被打中', '開砲', '靠近敵人', '遠離敵人', '每步存活', '擊殺', '陣亡'] # 和 B2 的滑桿同名\n", "IRL_EXPERTS = {'積極的專家（會去打）': '設計得不錯', '躲著的專家（只想活下來）': '只獎勵存活'}\n", "_irl_cache = {}\n", "_irl_last = {}\n", "\n", "def _phi(info): return np.array([info['hit'], info['damaged'], info['fired'], info['closer'], info['farther'], 1, info['win'], info['dead']], float)\n", "\n", "def _irl_features(Q, n=200, seed=7):\n", " rng = np.random.default_rng(seed); env = Arena(); tot = np.zeros(8); wins = 0\n", " for _ in range(n):\n", " s_ = env.reset(rng); done = False\n", " while not done:\n", " a = int(np.argmax(Q[s_] + PREF)); s_, _, done, info = env.step(a, ZERO); tot += _phi(info)\n", " wins += info['win']\n", " return tot / n, wins / n\n", "\n", "def irl_lab(use, rounds=5, seed=0, expert='積極的專家（會去打）'):\n", " \"\"\"use：{事件: True/False} 獎勵可以看哪些事。每一輪：比較專家和 Actor 每件事做幾次 → 專家做得比較多的事調高權重、Actor 做得比較多的調低 → 重新訓練 Actor\"\"\"\n", " t0 = _time.time()\n", " if expert not in _irl_cache: QE, _ = train(PRESETS[IRL_EXPERTS[expert]], seed=seed); _irl_cache[expert] = _irl_features(QE)\n", " fE, winE = _irl_cache[expert]; mask = np.array([bool(use.get(f)) for f in IRL_FEATS], float)\n", " if not mask.any(): print('⛔ 至少要勾一件事，獎勵才看得到東西。'); return\n", " unit = {'命中敵人': '次', '被打中': '次', '開砲': '次', '靠近敵人': '次', '遠離敵人': '次', '每步存活': '步', '擊殺': '', '陣亡': ''}\n", " fmt = lambda f, v: f'{v:.0%}' if f in ('擊殺', '陣亡') else f'{v:.1f} {unit[f]}'.replace('.0 ', ' ')\n", " def acts(f_): return f'走 {f_[5]:.0f} 步、開砲 {f_[2]:.1f} 次、命中 {f_[0]:.1f} 次'\n", " print(f'👀 專家每局平均：{acts(fE)}，勝率 {winE:.0%}（IRL 只看得到這些，看不到專家原本的獎勵）')\n", " off = [f for f, m in zip(IRL_FEATS, mask) if not m]\n", " print(f'　獎勵可以看：{\"、\".join(f for f, m in zip(IRL_FEATS, mask) if m)}')\n", " print(f'　沒勾：{\"、\".join(off)} → 不算分（權重固定 0），但動作還在、照樣可以做；勾法不同，學出來的打法和勝率就不同\\n' if off else '　（全部都勾：每件事都能拿來調獎勵）\\n')\n", " w = np.zeros(8); QA = np.zeros((N_STATES, 5)); wins = []; cards = []\n", " for it in range(rounds + 1):\n", " fA, winA = _irl_features(QA); wins.append(winA)\n", " if it == 0: head = f'第 0 輪｜Actor 還沒學：{acts(fA)} → 勝率 {winA:.0%}'\n", " else:\n", " tag = '✅ 和專家差不多' if winA >= winE - .1 else ('❌ 被鑽了：原地一直開砲，不去打' if fA[2] > 3 * max(fE[2], 1) else ('❌ 學不會：整局不動（獎勵太少見，亂試找不到勝利）' if fA[5] >= 59 and fA[2] < .5 else '🟡 還沒學好'))\n", " head = f'第 {it} 輪｜Actor 用新的獎勵學完：{acts(fA)} → 勝率 {winA:.0%}　{tag}'\n", " print(head)\n", " if it == rounds: cards.append((fA, winA, [])); break\n", " d = (fE - fA) * mask; score = np.abs(d) / np.maximum(np.maximum(fE, fA), 1)\n", " tops = [k for k in np.argsort(-np.abs(d)) if score[k] > .45][:3]; cards.append((fA, winA, [f'{IRL_FEATS[k]} {\"↑\" if d[k] > 0 else \"↓\"}' for k in tops]))\n", " if not tops: print('　→ 每件事的次數都和專家差不多了，獎勵只做小幅微調\\n')\n", " else:\n", " why = '；'.join(f'「{IRL_FEATS[k]}」Actor {fmt(IRL_FEATS[k], fA[k])}、專家 {fmt(IRL_FEATS[k], fE[k])}，' + ('專家比較多 → 調高' if d[k] > 0 else 'Actor 比較多 → 調低') for k in tops)\n", " print(f'　→ 差最多的：{why}\\n')\n", " w = w + (fE - fA) * mask\n", " Rr = dict(ZERO); wr = w / (np.abs(w).max() + 1e-9) * 10\n", " Rr['自訂'] = lambda info, wr=wr: float(wr @ _phi(info)); QA, _ = train(Rr, seed=seed + 1 + it)\n", " wn = w / (np.abs(w).max() + 1e-9) * 10\n", " _irl_last.clear(); _irl_last.update({f: round(float(v), 1) for f, v in zip(IRL_FEATS, wn)})\n", " print('📋 最後反推出的獎勵：' + '、'.join(f'{f} {v:+.1f}'.replace('-', '−') for f, v in sorted(zip(IRL_FEATS, wn), key=lambda x: -abs(x[1])) if abs(v) >= .1))\n", " _irl_cards(cards, fE, winE, mask)\n", " print(f'（{_time.time() - t0:.0f} 秒）')\n", "\n", "def irl_to_b2():\n", " \"\"\"把 D 最後反推出的獎勵，當成 B2 的 8 個滑桿拿去訓練一次，結果加進排行榜\"\"\"\n", " if not _irl_last: print('👉 先執行上一格（逆強化學習），才有反推出的獎勵。'); return None\n", " R = {k: 0 for k in _R_ORDER}; R.update(_irl_last)\n", " print('🎛️ 把反推出的獎勵填進 B2 的滑桿：' + '、'.join(f'{k} {R[k]:+g}'.replace('-', '−') for k in _R_ORDER if R[k]))\n", " run = tank_lab('IRL 反推的獎勵', R)\n", " others = [r['win'] for r in tank_runs[:-1] if r['note'] != 'IRL 反推的獎勵']; best = max(others) if others else None\n", " if best is not None and run['win'] >= best - .1:\n", " print(f'👉 和 B2 最好的回合（勝率 {best:.0%}）差不多：數字很不一樣，卻教出差不多的坦克——能解釋同一組示範的獎勵不只一種。')\n", " else:\n", " print('👉 比 B2 最好的回合低：IRL 只保證「行為像專家」，換個種子重新訓練，不一定每次都學得那麼好。'\n", " '實務上會多訓練幾次、挑最像專家的那一個，或把反推的獎勵和人設計的獎勵合起來用。')\n", " return run\n", "\n", "def _irl_cards(cards, fE, winE, mask=None):\n", " \"\"\"上＝勝率；下＝每個「有勾選」的項目各一張小圖（Actor 實線、專家虛線）\"\"\"\n", " mask = np.ones(8) if mask is None else mask\n", " on = [k for k in range(8) if mask[k]]; off = [IRL_FEATS[k] for k in range(8) if not mask[k]]\n", " n = len(cards); x = np.arange(n); wins = [c[1] for c in cards]\n", " cols = min(4, len(on)); rows = (len(on) + cols - 1) // cols\n", " fig = plt.figure(figsize=(max(8, 2.6 * cols), 2.4 + 2.1 * rows), layout='constrained')\n", " gs = fig.add_gridspec(1 + rows, cols, height_ratios=[1.2] + [1] * rows)\n", " ax = fig.add_subplot(gs[0, :]); ax.plot(x, wins, 'o-', color=GREEN, lw=2.6, ms=8, label='Actor'); ax.axhline(winE, color=NAVY, ls='--', lw=1.6, label=f'專家 {winE:.0%}')\n", " for k, v in enumerate(wins): ax.text(k, v + .1, f'{v:.0%}', ha='center', fontsize=10.5, color=GREEN if v >= winE - .1 else RED, fontweight='bold')\n", " ax.set_ylim(-.1, 1.3); ax.set_yticks([0, .5, 1], ['0%', '50%', '100%']); ax.set_xticks(x, [f'第 {k} 輪' for k in x])\n", " ax.legend(loc='upper left', ncol=2, frameon=False, fontsize=9); ax.spines[['top', 'right']].set_visible(False)\n", " ax.set_title('勝率', loc='left', fontweight='bold', color=NAVY)\n", " for m, k in enumerate(on):\n", " a = fig.add_subplot(gs[1 + m // cols, m % cols]); ys = [c[0][k] for c in cards]\n", " a.plot(x, ys, 'o-', color=ORANGE, lw=2, ms=5); a.axhline(fE[k], color=NAVY, ls='--', lw=1.3)\n", " a.set_title(f'{IRL_FEATS[k]}（專家 {fE[k]:.0%}）' if IRL_FEATS[k] in ('擊殺', '陣亡') else f'{IRL_FEATS[k]}（專家 {fE[k]:.1f}）', fontsize=10, color='#24364A')\n", " a.set_xticks(x, [str(i) for i in x], fontsize=8); a.tick_params(labelsize=8); a.spines[['top', 'right']].set_visible(False)\n", " fig.suptitle('獎勵可以看：' + '、'.join(IRL_FEATS[k] for k in on) + ('\\n沒勾（獎勵看不到）：' + '、'.join(off) if off else ''), fontsize=11, color=NAVY, fontweight='bold')\n", " plt.show()\n", " print('下面每張小圖：橘線＝這一輪的 Actor、藍虛線＝專家；橫軸是第幾輪。沒勾＝不算分、不是不能做：Actor 照樣可以做那件事，只是做多做少都不影響分數。')\n", "\n", "# ---------- E2：GAN 對決 ----------\n", "gan_runs = []\n", "\n", "def gan_lab(note, lr_g, lr_d, d_steps=1, steps=3000, seed=0):\n", " t0 = _time.time(); out, hist = train_gan(steps=steps, lr_g=lr_g, lr_d=lr_d, d_steps=d_steps, seed=seed)\n", " real, _ = real_ring(1000, np.random.default_rng(1)); modes, quality = coverage(out[-1][1])\n", " gan_runs.append(dict(note=note, lr_g=lr_g, lr_d=lr_d, d_steps=d_steps, steps=steps, modes=modes, quality=quality))\n", " fig, axs = plt.subplots(1, len(out), figsize=(2.3 * len(out), 2.6), layout='constrained')\n", " for ax, (t, pts) in zip(axs, out):\n", " ax.scatter(real[:, 0], real[:, 1], s=3, c=GRAY, alpha=.5); ax.scatter(pts[:, 0], pts[:, 1], s=3, c=ORANGE, alpha=.6)\n", " ax.set_xlim(-3, 3); ax.set_ylim(-3, 3); ax.set_aspect('equal'); ax.set_xticks([]); ax.set_yticks([]); ax.set_title(f'第 {t} 步', fontsize=10)\n", " fig.suptitle(f'{note}｜灰＝真資料（8 群）、橘＝生成器畫的', fontsize=11); plt.show()\n", " print(f'回合 {len(gan_runs) - 1}｜生成器學習率 {lr_g:g}、鑑別器學習率 {lr_d:g}、鑑別器每步練 {d_steps} 次、{steps} 步｜{_time.time() - t0:.1f} 秒')\n", " print(f'🎯 涵蓋 {modes} / 8 群｜{quality:.0%} 的點落在真資料附近')\n", " if modes <= 3: print('🔴 mode collapse：生成器找到一兩種騙得過鑑別器的畫法，就只畫那幾種。')\n", " elif modes < 8: print('🟡 漏了幾群（mode dropping）：有學到一部分，但不完整。')\n", " elif quality < .7: print('🟡 8 群都有，但很多點散在中間：品質還不夠。')\n", " else: print('🟢 8 群都學出來，而且大部分的點都在對的位置。')\n", " for i, r in sorted(enumerate(gan_runs), key=lambda x: (-x[1]['modes'], -x[1]['quality']))[:8]:\n", " print(f' 回合 {i}｜{r[\"modes\"]} 群｜品質 {r[\"quality\"]:.0%}｜G {r[\"lr_g\"]:g}、D {r[\"lr_d\"]:g}、D×{r[\"d_steps\"]}｜{r[\"note\"]}')\n", "\n", "# ---------- E3：驗證碼 ----------\n", "QUIZ_SEED = 2026\n", "\n", "def _quiz_items(n=10):\n", " rng = np.random.default_rng(QUIZ_SEED); items = []\n", " for i in range(n):\n", " lv = min(3, i * 4 // n); k = tuple(int(min(3, max(0, lv + v))) for v in rng.integers(-1, 1, 4))\n", " t = random_text(rng); items.append((t, k, make(t, *k, rng=rng)))\n", " return items\n", "\n", "def captcha_quiz():\n", " items = _quiz_items(); fig, axs = plt.subplots(2, 5, figsize=(13, 2.9), layout='constrained')\n", " for k, (ax, (_, _, img)) in enumerate(zip(axs.flat, items)):\n", " ax.imshow(img, cmap='gray', vmin=0, vmax=255); ax.set_xticks([]); ax.set_yticks([]); ax.set_title(f'第 {k + 1} 題', fontsize=10)\n", " plt.show(); print('👉 把 10 題的答案依序填在 ANSWERS（用空白或逗號隔開，大小寫都可以），再執行下一格。')\n", "\n", "def check_quiz(answers):\n", " if '填' in str(answers): print('👉 先把 10 題的答案填在 ANSWERS 再按 ▶（現在還不會公布答案）。'); return None\n", " items = _quiz_items(); got = [a.strip().upper() for a in str(answers).replace(',', ' ').split()]\n", " if len(got) != len(items): raise Stop(f'⛔ 要剛好 {len(items)} 個答案，現在有 {len(got)} 個。')\n", " me = [g == t for g, (t, _, _) in zip(got, items)]\n", " ai = [AI_TABLE[''.join(map(str, k))] for _, k, _ in items]\n", " for k, ((t, kn, _), g, ok, a) in enumerate(zip(items, got, me, ai), 1):\n", " print(f'第 {k:>2} 題　答案 {t}｜你寫 {g:<5}{\"✅\" if ok else \"❌\"}｜這種難度新版 AI 答對率 {a:.0%}')\n", " print(f'\\n你答對 {sum(me)} / {len(me)} 題；AI 在同樣難度的平均答對率 {np.mean(ai):.0%}')\n", " fig, ax = plt.subplots(figsize=(7, 2.6), layout='constrained')\n", " ax.plot(range(1, 11), [int(x) for x in me], 'o-', color=ORANGE, label='你（答對＝1）'); ax.plot(range(1, 11), ai, 's-', color=NAVY, label='AI 答對率')\n", " ax.set_xlabel('題號（越後面越難）'); ax.set_ylim(-.05, 1.05); ax.legend(); ax.grid(alpha=.25); ax.set_title('人 vs AI：難度越高，誰先撐不住？'); plt.show()\n", "\n", "def design_captcha(noise, rotate, overlap, warp, seed=0, n=5):\n", " rng = np.random.default_rng(seed); items = [(t, make(t, noise, rotate, overlap, warp, rng=rng)) for t in [random_text(rng) for _ in range(n)]]\n", " fig, axs = plt.subplots(1, n, figsize=(2.7 * n, 1.4), layout='constrained')\n", " for k, (ax, (_, img)) in enumerate(zip(axs, items)):\n", " ax.imshow(img, cmap='gray', vmin=0, vmax=255); ax.set_xticks([]); ax.set_yticks([]); ax.set_title(f'{k + 1}', fontsize=10)\n", " plt.show(); print(f'👉 請隔壁同學看圖作答，答案填在 NEIGHBOR_ANSWERS（{n} 個）。設計代碼：{noise}{rotate}{overlap}{warp}-{seed}')\n", " return items\n", "\n", "def score_design(noise, rotate, overlap, warp, seed, neighbor_answers):\n", " if '一' in str(neighbor_answers): print('👉 請隔壁同學作答，把答案填在 NEIGHBOR_ANSWERS 再按 ▶。'); return None\n", " rng = np.random.default_rng(seed); n = 5; texts = [random_text(rng) for _ in range(n)] # 和 design_captcha 同樣的亂數順序\n", " got = [a.strip().upper() for a in str(neighbor_answers).replace(',', ' ').split()]\n", " if len(got) != n: raise Stop(f'⛔ 要剛好 {n} 個答案。')\n", " human = np.mean([g == t for g, t in zip(got, texts)]); k = f'{noise}{rotate}{overlap}{warp}'; old, new = AI_OLD[k], AI_TABLE[k]\n", " print(f'正確答案：{\" \".join(texts)}')\n", " print(f'同學答對率 {human:.0%}｜舊版搶票程式（只看過簡單驗證碼）答對率 {old:.0%}｜設計分數＝同學 − 舊版 ＝ {human - old:+.0%}')\n", " if human - old >= .6: print('🟢 達標（≥ +60 個百分點）：人看得懂、舊版程式看不懂，你打敗了舊版！')\n", " elif human < .6: print('🔴 人也看不懂了：這種驗證碼會把真的客人擋在門外。')\n", " else: print('🔴 還沒達標：舊版搶票程式還是看得懂太多（目標：同學 − 舊版 ≥ +60 個百分點）。')\n", " print(f'\\n⚡ 但是……黃牛拿各種難度的驗證碼重新訓練，新版搶票程式在你的設計上答對率 {new:.0%}。')\n", " print('　 這就是對抗：你變強，對手就跟著重新訓練。最後的解法通常不是更難的驗證碼，而是看操作行為、實名制與法律。')\n", "\n", "# ---------- 推導的驗證 ----------\n", "def bellman_demo(episodes=500, alpha=0.1, gamma=0.9, eps=0.5, seed=0, answer=None, show=False):\n", " \"\"\"四格走廊：第 0 格是牆，坦克從第 2 格出發，走進第 4 格得 +10。先手算 Q*(2,→)，答對（或看解答）後再看 Q 學習學到的值\"\"\"\n", " print(f'題目：四格走廊，γ = {gamma:g}。Q*(2,→) 是多少？')\n", " _judge('A3', answer, gamma * 10, [f'從終點往回算：第 3 格往右直接拿 10，所以 Q*(3,→) = 10。', f'Q*(2,→) = 0 + γ × Q*(3,→) = 0 + {gamma:g} × 10。'],\n", " [f'Q*(3,→) = 10', f'Q*(2,→) = 0 + {gamma:g} × 10 = {gamma * 10:g}'], show)\n", " if not (_done.get('A3') or show): return None\n", " print('\\n▶ 接著讓 Q 學習自己學，看會不會學到一樣的數字：')\n", " rng = np.random.default_rng(seed); Q = np.zeros((4, 2)); hist = []\n", " for _ in range(episodes):\n", " s = 2\n", " for _ in range(30):\n", " a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s] + rng.random(2) * 1e-6))\n", " if a == 1 and s == 3: r, s2, done = 10, None, True\n", " else: r, s2, done = 0, (s + 1 if a == 1 else max(s - 1, 1)), False\n", " Q[s, a] += alpha * (r + (0 if done else gamma * Q[s2].max()) - Q[s, a])\n", " if done: break\n", " s = s2\n", " hist.append(Q.copy())\n", " g = gamma; h = np.array(hist)\n", " truth = [('Q(3,→)', 3, 1, 10), ('Q(2,→)', 2, 1, g * 10), ('Q(1,→)', 1, 1, g * g * 10), ('Q(3,←)', 3, 0, g * g * 10), ('Q(2,←)', 2, 0, g ** 3 * 10), ('Q(1,←)', 1, 0, g ** 3 * 10)]\n", " fig, ax = plt.subplots(figsize=(7.5, 3), layout='constrained')\n", " for (name, s_, a_, v), col in zip(truth, [GREEN, NAVY, '#835C98', ORANGE, RED, '#B35A12']):\n", " ax.plot(h[:, s_, a_], color=col, lw=2, label=name); ax.axhline(v, color=col, ls='--', lw=1)\n", " ax.set_xlabel('訓練局數'); ax.set_ylabel('Q 值'); ax.legend(frameon=False, ncol=3, fontsize=9); ax.grid(alpha=.2); ax.set_title(f'實線＝Q 學習，虛線＝貝爾曼方程（γ = {gamma:g}）'); plt.show()\n", " for name, s_, a_, v in truth: print(f'{name}：手算 {v:.3f}｜學到 {h[-1, s_, a_]:.3f}')\n", "\n", "def hack_return(gamma, steps_to_enemy=12, answer=None, show=False):\n", " \"\"\"只獎勵「靠近 +1」時，乖乖走過去 vs 走過去再來回晃 的回報（無限長的近似）\"\"\"\n", " n = steps_to_enemy; go = (1 - gamma ** n) / (1 - gamma); loop = gamma ** (n + 1) / (1 - gamma ** 2)\n", " print(f'題目：γ = {gamma:g}，離敵人 {n} 步。「走過去再來回晃」的總回報是多少？')\n", " _judge('A4', answer, go + loop, [f'分兩段：①乖乖走過去 {n} 步，每步 +1：1 + γ + … + γ^{n - 1} = (1 − γ^{n}) / (1 − γ)。',\n", " f'②之後每 2 步（退一步、再進一步）拿一次 +1，從第 {n + 2} 步開始：γ^{n + 1} + γ^{n + 3} + … = γ^{n + 1} / (1 − γ²)。兩段加起來。'],\n", " [f'乖乖走過去：(1 − γ^{n}) / (1 − γ) = {go:.2f}', f'再來回晃：多 γ^{n + 1} / (1 − γ²) = {loop:.2f}，合計 {go + loop:.2f}', '來回晃的回報比較高：Q 學習會學到刷分。'], show, tol=0.05)\n", "\n", "def optimal_d_demo(mu_fake=2.0, steps=3000, seed=0):\n", " \"\"\"1D：固定生成器分布 q = N(mu_fake, 1)，訓練鑑別器，和公式 D* = p/(p+q) 比較\"\"\"\n", " torch.manual_seed(seed); rng = np.random.default_rng(seed)\n", " D = nn.Sequential(nn.Linear(1, 32), nn.Tanh(), nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 1)); opt = torch.optim.Adam(D.parameters(), 3e-3); bce = nn.BCEWithLogitsLoss()\n", " for _ in range(steps):\n", " xr = torch.tensor(rng.normal(0, 1, (256, 1)), dtype=torch.float32); xf = torch.tensor(rng.normal(mu_fake, 1, (256, 1)), dtype=torch.float32)\n", " loss = bce(D(xr), torch.ones(256, 1)) + bce(D(xf), torch.zeros(256, 1)); opt.zero_grad(); loss.backward(); opt.step()\n", " xs = np.linspace(-3, mu_fake + 3, 300); pdf = lambda x, m: np.exp(-(x - m) ** 2 / 2)\n", " with torch.no_grad(): d = torch.sigmoid(D(torch.tensor(xs[:, None], dtype=torch.float32))).numpy()[:, 0]\n", " star = pdf(xs, 0) / (pdf(xs, 0) + pdf(xs, mu_fake))\n", " fig, ax = plt.subplots(figsize=(7.5, 3), layout='constrained')\n", " ax.plot(xs, star, 'k--', lw=2, label='公式 D* = p/(p+q)'); ax.plot(xs, d, color=NAVY, lw=2.5, alpha=.8, label='訓練出來的鑑別器')\n", " ax.legend(frameon=False); ax.grid(alpha=.2); ax.set_xlabel('x'); ax.set_title(f'真資料 p = N(0,1)、生成器 q = N({mu_fake:g},1)'); plt.show()\n", " print(f'訓練結果和公式的平均差距：{np.abs(d - star).mean():.3f}（越接近 0 越吻合）')\n", "\n", "def baseline_demo(lr=0.02, runs=100, steps=300):\n", " \"\"\"兩台拉霸機（平均 10 vs 8，獎勵都是正的）：Policy Gradient 有沒有 baseline\"\"\"\n", " def one(bl, seed):\n", " rng = np.random.default_rng(seed); th = np.zeros(2); b = 0.; out = []\n", " for _ in range(steps):\n", " e = np.exp(th - th.max()); p = e / e.sum(); a = int(rng.choice(2, p=p)); r = rng.normal((10., 8.)[a], 2.)\n", " g = -p; g[a] += 1; th += lr * (r - (b if bl else 0)) * g\n", " if bl: b += 0.1 * (r - b)\n", " e = np.exp(th - th.max()); out.append((e / e.sum())[0])\n", " return np.array(out)\n", " fig, ax = plt.subplots(figsize=(7.5, 3), layout='constrained')\n", " for bl, col, name in [(False, RED, '沒有 baseline'), (True, GREEN, '有 baseline')]:\n", " ps = np.array([one(bl, s) for s in range(runs)]); ax.plot(ps.mean(0), color=col, lw=2.2, label=name)\n", " print(f'{name}：最後選好拉霸機的機率 {ps[:, -1].mean():.0%}；{(ps[:, -1] < .5).mean():.0%} 的實驗選到比較差的')\n", " ax.set_ylim(0, 1); ax.set_xlabel('拉了幾次'); ax.set_ylabel('選好拉霸機的機率'); ax.legend(frameon=False); ax.grid(alpha=.2); plt.show()\n", "\n", "def imitation_demo(noise=0.1, demo_games=50, dagger_rounds=4):\n", " \"\"\"模仿學習：專家＝Q 學習學好的坦克；BC 只抄示範看過的狀態；DAgger 讓模仿者自己開、走到哪問專家\"\"\"\n", " Q, _ = train(PRESETS['設計得不錯']); expert = lambda s: int(np.argmax(Q[s] + PREF))\n", " _, demo = rollout(expert, n=demo_games, seed=3); bc, _ = bc_table(expert, demo); states = set(demo)\n", " for it in range(dagger_rounds):\n", " pol, _ = bc_table(expert, states); _, vis = rollout(pol, n=50, noise=noise, seed=10 + it); states |= set(vis)\n", " dag, _ = bc_table(expert, states)\n", " print(f'專家示範 {demo_games} 局，走過 {len(set(demo))} 個狀態；DAgger 補到 {len(states)} 個狀態')\n", " for name, pol in [('專家', expert), ('模仿學習 BC', bc), (f'DAgger（補 {dagger_rounds} 輪）', dag)]:\n", " print(f'{name:<14} 照常開 {rollout(pol)[0]:.0%}｜每步 {noise:.0%} 被推一下 {rollout(pol, noise=noise)[0]:.0%}')\n", "\n", "# ---------- B4：先模仿、再強化 ----------\n", "def imitation_rl_demo(demo_games=50, budgets=(250, 500, 1000, 2000), seeds=3):\n", " \"\"\"老師（不完美）示範 demo_games 局 → 比較「從零開始強化」和「先模仿再強化」：練不同局數時的勝率\"\"\"\n", " R = PRESETS['設計得不錯']; expert, _ = weak_expert(R)\n", " _, demo = rollout(expert, n=demo_games, seed=3); demo = set(demo)\n", " init = np.zeros((N_STATES, 5))\n", " for s_ in demo: init[s_, expert(s_)] = 1.0\n", " t0 = _time.time(); w_e = rollout(expert)[0]; w_b = rollout(bc_table(expert, demo)[0])[0]\n", " scratch = [float(np.mean([evaluate(train_q_from(R, np.zeros_like(init), n, s_))['win'] for s_ in range(seeds)])) for n in budgets]\n", " imit = [float(np.mean([evaluate(train_q_from(R, init, n, s_, eps_start=0.3))['win'] for s_ in range(seeds)])) for n in budgets]\n", " fig, ax = plt.subplots(figsize=(7.2, 3.6), layout='constrained'); x = range(len(budgets))\n", " ax.plot(x, imit, 'o-', color=GREEN, lw=2.5, ms=8, label='先模仿、再強化'); ax.plot(x, scratch, 's-', color=RED, lw=2.5, ms=8, label='從零開始強化')\n", " if abs(w_e - w_b) < .02: ax.axhline(w_e, color=NAVY, ls='--', lw=1.5, label=f'老師 {w_e:.0%}（只模仿也是 {w_b:.0%}）')\n", " else: ax.axhline(w_e, color=NAVY, ls='--', lw=1.5, label=f'老師 {w_e:.0%}'); ax.axhline(w_b, color=GRAY, ls=':', lw=1.5, label=f'只模仿（照抄老師）{w_b:.0%}')\n", " for k in x:\n", " hi_i = imit[k] >= scratch[k]\n", " ax.text(k + (.12 if abs(imit[k] - scratch[k]) < .08 else 0), imit[k] + (.05 if hi_i else -.1), f'{imit[k]:.0%}', ha='center', color=GREEN, fontweight='bold')\n", " ax.text(k - (.12 if abs(imit[k] - scratch[k]) < .08 else 0), scratch[k] + (-.1 if hi_i else .05), f'{scratch[k]:.0%}', ha='center', color=RED, fontweight='bold')\n", " ax.set_xticks(list(x), [f'{n} 局' for n in budgets]); ax.set_xlabel('強化學習練了幾局'); ax.set_ylabel('勝率'); ax.set_ylim(-.15, 1.1)\n", " ax.legend(fontsize=9, loc='upper center', bbox_to_anchor=(.5, -.2), ncol=3, frameon=False); ax.grid(alpha=.25); ax.set_title(f'老師示範了 {demo_games} 局（裡面只出現過 {len(demo)} 種局面）'); plt.show()\n", " first = lambda ws: next((n for n, w in zip(budgets, ws) if w > w_e), None)\n", " fi, fs = first(imit), first(scratch)\n", " print('🟢 先模仿再強化：' + (f'練 {fi} 局就超越老師。' if fi else '這次還沒超越老師。') + '　🔴 從零開始：' + (f'要練到 {fs} 局才超越老師。' if fs else f'練到 {budgets[-1]} 局都還沒超越老師。'))\n", " print('👉 示範給了一個好起點：不用從零亂試，很快就能拿到勝利、開始改進。練得夠久，從零開始也會追上。')\n", " print(f'（{_time.time() - t0:.0f} 秒）')\n", "\n", "# ---------- C：冰湖 ----------\n", "def _lake_panel(ax, Q, slip, title):\n", " \"\"\"畫一張冰湖：起點、洞、寶藏、每一格 AI 想走的方向；朝牆走的格子標橘色\"\"\"\n", " walls = []\n", " for s_ in range(16):\n", " r, c = divmod(s_, 4); ch = LAKE[r][c]; a = int(np.argmax(Q[s_])); dr, dc = LAKE_DIRS[a]\n", " toward_wall = ch in 'S.' and not (0 <= r + dr <= 3 and 0 <= c + dc <= 3)\n", " fc = {'#': '#0B3D66', 'G': '#F2C14E'}.get(ch, '#FDE3C8' if toward_wall else '#DDEFFF')\n", " ax.add_patch(patches.Rectangle((c, 3 - r), 1, 1, fc=fc, ec='white', lw=2))\n", " txt = {'#': '洞', 'G': '寶'}.get(ch, LAKE_ARROW[a])\n", " ax.text(c + .5, 3 - r + .5, txt, ha='center', va='center', fontsize=20, color='white' if ch == '#' else '#24364A', fontweight='bold')\n", " if ch == 'S': ax.text(c + .12, 3 - r + .85, '起', fontsize=10, color=ORANGE, fontweight='bold')\n", " if toward_wall: walls.append((r, c, LAKE_ARROW[a]))\n", " ax.set_xlim(0, 4); ax.set_ylim(0, 4); ax.set_aspect('equal'); ax.axis('off'); ax.set_title(title, fontsize=11)\n", " return walls\n", "\n", "def _lake_episode(Q, slip, seed):\n", " \"\"\"照 Q 表走一局，記下每一步「想走的方向」和「實際走的方向」\"\"\"\n", " rng = np.random.default_rng(seed); s_ = 0; steps = []\n", " for _ in range(60):\n", " a = int(np.argmax(Q[s_])); act = a\n", " if rng.random() < slip: act = (a + rng.choice([-1, 1])) % 4\n", " r, c = divmod(s_, 4); dr, dc = LAKE_DIRS[act]; r2, c2 = min(max(r + dr, 0), 3), min(max(c + dc, 0), 3); s2 = r2 * 4 + c2\n", " steps.append((s_, a, act, s2, LAKE[r2][c2])); s_ = s2\n", " if LAKE[r2][c2] in '#G': break\n", " return steps\n", "\n", "def lake_movie(Q, slip):\n", " \"\"\"挑一局有滑倒、30 步內走完的來播：先畫想走的方向，再移動到實際走到的格子\"\"\"\n", " pick = None\n", " for seed in range(300):\n", " ep = _lake_episode(Q, slip, seed)\n", " if len(ep) <= 30 and (slip == 0 or sum(a != b for _, a, b, _, _ in ep) >= 2) and ep[-1][4] == 'G': pick = ep; break\n", " pick = pick or _lake_episode(Q, slip, 0)\n", " fig, ax = plt.subplots(figsize=(4.2, 4.6))\n", " for s_ in range(16):\n", " r, c = divmod(s_, 4); ch = LAKE[r][c]\n", " ax.add_patch(patches.Rectangle((c, 3 - r), 1, 1, fc={'#': '#0B3D66', 'G': '#F2C14E'}.get(ch, '#DDEFFF'), ec='white', lw=2))\n", " if ch in '#G': ax.text(c + .5, 3 - r + .5, {'#': '洞', 'G': '寶'}[ch], ha='center', va='center', fontsize=18, color='white' if ch == '#' else '#24364A', fontweight='bold')\n", " ax.text(.12, 3.85, '起', fontsize=10, color=ORANGE, fontweight='bold')\n", " ax.set_xlim(0, 4); ax.set_ylim(0, 4); ax.set_aspect('equal'); ax.axis('off')\n", " me, = ax.plot([], [], 'o', color=ORANGE, ms=14, mec='#8A4A10', mew=2, zorder=5)\n", " want = ax.annotate('', xy=(0, 0), xytext=(0, 0), arrowprops=dict(arrowstyle='-|>', color=GREEN, lw=3), zorder=6)\n", " real = ax.annotate('', xy=(0, 0), xytext=(0, 0), arrowprops=dict(arrowstyle='-|>', color=RED, lw=2.5, ls='--'), zorder=6)\n", " note = ax.text(2, -.25, '', ha='center', va='center', fontsize=10.5, color=RED, fontweight='bold'); title = ax.set_title('')\n", " seq = [(k, j) for k in range(len(pick)) for j in (0, 1)]\n", " xy = lambda s_: (s_ % 4 + .5, 3 - s_ // 4 + .5)\n", " names = ['左', '下', '右', '上']\n", " def upd(i):\n", " k, j = seq[i]; s_, a, act, s2, cell = pick[k]; x, y = xy(s_)\n", " want.set_visible(False); real.set_visible(False); note.set_text('')\n", " if j == 0:\n", " me.set_data([x], [y]); dc, dr = LAKE_DIRS[a][1], -LAKE_DIRS[a][0]\n", " want.xy = (x + dc * .45, y + dr * .45); want.set_position((x, y)); want.set_visible(True)\n", " if act != a:\n", " rc, rr = LAKE_DIRS[act][1], -LAKE_DIRS[act][0]; real.xy = (x + rc * .45, y + rr * .45); real.set_position((x, y)); real.set_visible(True)\n", " note.set_text(f'滑倒！想往{names[a]}，結果往{names[act]}')\n", " elif s2 == s_: note.set_text(f'往{names[a]}撞牆，留在原地（滑倒也只會往{names[(a + 1) % 4]}或往{names[(a - 1) % 4]}，不會滑向{names[(a + 2) % 4]}邊）')\n", " else:\n", " x2, y2 = xy(s2); me.set_data([x2], [y2])\n", " if cell == 'G': note.set_text('到達寶藏 +1'); note.set_color(GREEN)\n", " elif cell == '#': note.set_text('掉進洞，結束'); note.set_color(RED)\n", " else: note.set_color(RED)\n", " title.set_text(f'實際走一局（滑倒機率 {slip:.0%}）\\n第 {k + 1} 步：綠＝想走的方向、紅虛線＝實際滑去的方向')\n", " return me, want, real, note, title\n", " an = animation.FuncAnimation(fig, upd, frames=len(seq), interval=450, blit=False)\n", " plt.close(fig); return HTML(an.to_jshtml(default_mode='once'))\n", "\n", "def lake_demo(slip=0.33, gamma=0.95, step_cost=0.0):\n", " \"\"\"左：地板不會滑時學到的路線；右：照你設定的滑倒機率學到的路線。橘色格子＝AI 故意朝牆走\"\"\"\n", " Q0 = lake_train(slip=0.0, gamma=gamma, step_cost=step_cost); w0, s0 = lake_eval(Q0, 0.0)\n", " Q = lake_train(slip=slip, gamma=gamma, step_cost=step_cost); win, steps = lake_eval(Q, slip)\n", " fig, axs = plt.subplots(1, 2, figsize=(7.6, 3.9), layout='constrained')\n", " _lake_panel(axs[0], Q0, 0.0, f'地板不會滑\\n走到寶藏 {w0:.0%}，平均 {s0:.1f} 步')\n", " walls = _lake_panel(axs[1], Q, slip, f'滑倒機率 {slip:.0%}（γ = {gamma:g}）\\n走到寶藏 {win:.0%}，平均 {steps:.1f} 步')\n", " plt.show()\n", " print('🎯 目標：從「起」走到「寶」（+1 分），掉進「洞」就結束。箭頭＝AI 在每一格想走的方向。')\n", " print(f'❄️ 地板會滑：想往某個方向走時，有 {slip:.0%} 的機率滑到「左邊或右邊」（和想走的方向垂直）。')\n", " if walls:\n", " print('🟧 橘色格子＝AI 故意朝牆走（撞牆會留在原地）：')\n", " for r, c, a in walls:\n", " print(f'　第 {r + 1} 列第 {c + 1} 格朝 {a}：撞牆不會動；就算滑倒，也只會滑到垂直的兩側，避開旁邊的洞。')\n", " print('👉 寧可多花幾步，也不要冒險：「最短」不一定「最好」。')\n", " else:\n", " print('👉 沒有朝牆走的格子：地板不滑時，直接走最短的路就好。把 SLIP 調大再看一次。')\n", " print('\\n▶ 下面的動畫：用右邊那張路線實際走一局。')\n", " display(lake_movie(Q, slip))\n", "\n", "# ---------- D：21 點 ----------\n", "_BJ = {}\n", "def blackjack_demo(stand_small=13, stand_big=17):\n", " \"\"\"你的策略：莊家明牌 2～6 時，點數 ≥ stand_small 就停；莊家 7～10 或 A 時，點數 ≥ stand_big 就停\"\"\"\n", " if 'pol' not in _BJ: _BJ['Q'], _BJ['pol'] = bj_learn()\n", " ai = _BJ['pol']; mine = lambda v, d, ace: int(v < (stand_small if 2 <= d <= 6 else stand_big))\n", " w1, t1, e1 = bj_eval(mine); w2, t2, e2 = bj_eval(ai)\n", " fig, axs = plt.subplots(1, 2, figsize=(10, 3.6), layout='constrained')\n", " for ax, (name, pol) in zip(axs, [('你的策略', mine), ('AI 學出來的', ai)]):\n", " g = np.array([[pol(v, d, False) for d in range(1, 11)] for v in range(12, 22)])\n", " ax.imshow(g, cmap=matplotlib.colors.ListedColormap([GREEN, ORANGE]), aspect='auto', origin='lower')\n", " for i in range(10):\n", " for j in range(10): ax.text(j, i, '要' if g[i, j] else '停', ha='center', va='center', color='white', fontsize=9)\n", " ax.set_xticks(range(10), ['A'] + [str(k) for k in range(2, 11)]); ax.set_yticks(range(10), [str(v) for v in range(12, 22)])\n", " ax.set_xlabel('莊家的明牌'); ax.set_ylabel('我的點數'); ax.set_title(name)\n", " plt.show()\n", " print(f'你的策略：贏 {w1:.1%}、平手 {t1:.1%}，平均每局 {e1:+.3f}')\n", " print(f'AI 的策略：贏 {w2:.1%}、平手 {t2:.1%}，平均每局 {e2:+.3f}')\n", " print('👉 兩張圖哪裡不一樣？就算 AI 最好的策略，平均還是輸——莊家的規則本身就佔優勢。')\n", "\n", "print(\"工具已載入：坦克戰場 8×8、冰湖\")\n" ] }, { "cell_type": "markdown", "id": "bcc02e36", "metadata": {}, "source": [ "# A｜手算：回報與 Q 值更新（約 8 分鐘）" ] }, { "cell_type": "markdown", "id": "584381a6", "metadata": {}, "source": [ "## A1｜回報：未來的獎勵要打折（P24）\n", "回報 **G′ = r₁ + γ·r₂ + γ²·r₃ + …**：越晚拿到的獎勵，乘越多次 γ。γ 接近 1＝看得遠；γ 小＝只在乎眼前。\n", "\n", "每一格是一題：**先用紙筆算**，把答案填在 `ANSWER`（可以直接寫算式，例如 `0.8**4*10`，次方用 `**`），再按 ▶。答錯會給提示；答對才會公布算式。" ] }, { "cell_type": "code", "execution_count": null, "id": "2e5c36c6", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 A1 挑戰 1：獎勵 0、0、0、0、10，γ = 0.8\n", "#@markdown 第 5 步才贏，現在看起來值多少？\n", "ANSWER = None #@param {type:\"raw\"}\n", "check_return([0, 0, 0, 0, 10], 0.8, ANSWER, key='A1-1')" ] }, { "cell_type": "code", "execution_count": null, "id": "b1e53f21", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 A1 挑戰 2：獎勵 1、1、1、1，γ = 0.5\n", "#@markdown γ 小時，後面的獎勵還剩多少？\n", "ANSWER = None #@param {type:\"raw\"}\n", "check_return([1, 1, 1, 1], 0.5, ANSWER, key='A1-2')" ] }, { "cell_type": "code", "execution_count": null, "id": "14dd6d9a", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 A1 挑戰 3：獎勵 -1、-1、-1、10，γ = 0.9\n", "#@markdown 先吃點苦、最後贏，划算嗎？\n", "ANSWER = None #@param {type:\"raw\"}\n", "check_return([-1, -1, -1, 10], 0.9, ANSWER, key='A1-3')" ] }, { "cell_type": "markdown", "id": "69d002ab", "metadata": {}, "source": [ "## A2｜Q 值更新：走一步，修正一次（P66）\n", "**新 Q ＝ 舊 Q ＋ α ×（這一步的獎勵 ＋ γ × 下一格最好動作的 Q − 舊 Q）**。括號裡是「這次實際體驗到的 − 原本以為的」，α 決定要相信新體驗多少。做法和 A1 一樣：先手算，再填 `ANSWER`。" ] }, { "cell_type": "code", "execution_count": null, "id": "6f8c675c", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 A2 挑戰 1：舊 Q = 1，獎勵 +2，下一格最好動作的 Q = 5，α = 0.5，γ = 0.8\n", "#@markdown 新的 Q 是多少？\n", "ANSWER = None #@param {type:\"raw\"}\n", "check_q(1, 2, 5, 0.5, 0.8, ANSWER, key='A2-1')" ] }, { "cell_type": "code", "execution_count": null, "id": "77b7c744", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 A2 挑戰 2：舊 Q = 3，獎勵 -1，下一格最好動作的 Q = 2，α = 0.2，γ = 0.9\n", "#@markdown 新的 Q 是多少？\n", "ANSWER = None #@param {type:\"raw\"}\n", "check_q(3, -1, 2, 0.2, 0.9, ANSWER, key='A2-2')" ] }, { "cell_type": "markdown", "id": "087efa8f", "metadata": {}, "source": [ "# B｜坦克獎勵設計挑戰（約 25 分鐘）\n", "\n", "8×8 的小戰場：🟧 你的坦克從左下出發，🟦 敵人固定在右上（3 滴血），灰色是牆，淡紅色是敵人的射程（3 格內、同一直線、沒有牆擋，每步 35% 機率打中你）。\n", "坦克只有 5 個動作：前進、左轉、右轉、開砲、不動。**它不知道目標是什麼，只知道你給的獎勵**，用 Q 表自己摸索（每次訓練 8000 局 × 3 個種子，約 5 秒）。\n", "\n", "## B0｜獎勵是怎麼算出來的？（P89）\n", "「靠近敵人 +1」不是一句話，是一段程式。環境每走一步做三件事：\n", "\n", "1. **記下動作前的狀態**：例如和敵人的距離 `before = dist(pos)`\n", "2. **執行動作**，得到新的狀態\n", "3. **比較前後，判斷發生了什麼事**：距離變小 → `closer`；打中 → `hit`；被打中 → `damaged`……\n", "\n", "最後 **獎勵 ＝ Σ（每件事的分數 × 有沒有發生）**。工具裡真正的那一行：\n", "```python\n", "r = R[\"擊殺\"]*win + R[\"命中敵人\"]*hit + R[\"被打中\"]*damaged + R[\"每步存活\"]*(not dead) + R[\"開砲\"]*fired + R[\"靠近敵人\"]*closer + R[\"遠離敵人\"]*farther + R[\"陣亡\"]*dead\n", "```\n", "下面一步一步走給你看（獎勵：贏 +10、死 −10、靠近 +1）。注意最後幾步：**掉頭走回去沒扣分，再走過來又 +1**——這就是「靠近敵人」會被刷分的原因。" ] }, { "cell_type": "code", "execution_count": null, "id": "74e2ceb5", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🔍 B0｜一步一步看獎勵怎麼算（不用改，按 ▶）\n", "env = Arena(); env.reset(np.random.default_rng(0)) # 坦克從左下出發，車頭朝上\n", "R = dict(PRESETS['只看輸贏'], 靠近敵人=1) # 贏 +10、死 −10、靠近 +1\n", "plan = ['前進', '前進', # ① 往上走兩格，靠近敵人\n", " '右轉', '前進', # ② 轉向右邊，再靠近一格\n", " '右轉', '右轉', '前進', # ③ 掉頭（右轉兩次＝轉 180°），往回走一格：離敵人變遠\n", " '右轉', '右轉', '前進'] # ④ 再掉頭，走回來：又靠近一格\n", "EVENT = {'closer': '靠近', 'farther': '遠離', 'fired': '開砲', 'hit': '命中', 'damaged': '被打中', 'win': '打贏', 'dead': '陣亡'}\n", "total = 0\n", "for k, name in enumerate(plan, 1):\n", " before = dist(env.pos) # 動作前：和敵人的距離（走幾格）\n", " s, r, done, info = env.step(ACTIONS.index(name), R) # 執行動作\n", " after = dist(env.pos) # 動作後的距離\n", " events = '、'.join(v for key, v in EVENT.items() if info[key]) or '—'\n", " total += r\n", " print(f'第 {k:>2} 步｜{name}｜距離 {before:>2} → {after:>2}｜發生：{events:<4}｜這一步 {r:+g}｜累計 {total:+g}')\n", "print('👉 看第 7 步：往回走（遠離）沒有扣分；第 10 步走回來又 +1。來回走就能一直刷「靠近」的分數。')" ] }, { "cell_type": "markdown", "id": "76f3eba2", "metadata": {}, "source": [ "## B1｜先看三個陷阱（P90）\n", "依序選每一個 `PRESET` 按 ▶，看動畫：坦克學到了什麼？是你想要的嗎？" ] }, { "cell_type": "code", "execution_count": null, "id": "ed8c77e5", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎛️ 陷阱獎勵\n", "PRESET = '只獎勵開砲' #@param ['只獎勵開砲', '只獎勵存活', '只獎勵靠近敵人', '只看輸贏']\n", "run = tank_lab(PRESET, PRESETS[PRESET])" ] }, { "cell_type": "markdown", "id": "5b7f2494", "metadata": {}, "source": [ "## B2｜🎛️ 設計你的獎勵\n", "預設是「只看輸贏」（贏 +10、死 −10）——上面已經看到它學不會。用滑桿加上其他獎勵，讓坦克**真的打贏**，而且越快越好。\n", "\n", "| 任務 | 改哪裡 | 想一想 |\n", "|---|---|---|\n", "| 1 讓它學得會 | 加上「命中敵人」的小獎勵 | 為什麼只有輸贏時學不會？（獎勵太少見，它從來沒贏過） |\n", "| 2 不要拖 | 「每步存活」改成小負數（例如 −0.1） | 時間成本會讓它變急還是變魯莽？ |\n", "| 3 怕痛 | 「被打中」給負分 | 它會不會為了不被打，乾脆不去打？ |\n", "| 4 看多遠 | `GAMMA` 0.5、0.8、0.99 | γ 太小時，為什麼連路都不走？（P22） |\n", "| 4b 位能型提示 | **先把命中、被打中、每步存活歸 0**（只留擊殺 +10、陣亡 −10），再比較「靠近 +1、遠離 0」和「靠近 +1、遠離 −1」 | 推導過：只有靠近 +1 會被刷；走回來會扣回去就不會（P96） |\n", "| 5 探索 | `EPS` 0.0、0.3 | 不亂試和一直亂試，各會怎樣？（P30） |\n", "\n", "**學習設定是什麼？**\n", "- `GAMMA`（γ，折扣率）：越晚拿到的獎勵打越多折。0.9 大約只看 10 步內、0.99 大約看 100 步（P26）。\n", "- `EPS`（ε，亂試的機率）：訓練時每一步有 ε 的機率**隨便選一個動作**，其他時候選目前分數（Q）最高的動作，這叫 ε-greedy（P31）。訓練一開始 ε＝1（全部亂試），前 60% 的局數慢慢降到你設定的 `EPS`。ε＝0：從不亂試，可能一輩子不知道開砲的好處；ε 太大：一直亂試，學到了也不照著做。訓練完看動畫、算勝率時一律不亂試。\n", "- `EPISODES`：訓練幾局。" ] }, { "cell_type": "code", "execution_count": null, "id": "52780f81", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎛️ 獎勵控制台：改這裡，再按 ▶（每按一次，排行榜多一個回合）\n", "NOTE = '我的獎勵' #@param {type:\"string\"}\n", "#@markdown ### 每件事給幾分\n", "#@markdown **① 結果**：一局結束時才有（打贏、陣亡）\n", "擊殺 = 10 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "陣亡 = -10 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "#@markdown **② 中途提示**：每一步都可能拿到（正＝鼓勵、負＝懲罰）\n", "命中敵人 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "被打中 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "每步存活 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "開砲 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "靠近敵人 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "遠離敵人 = 0 #@param {type:\"slider\", min:-10, max:10, step:0.1}\n", "#@markdown ### 學習設定（說明在上一格：GAMMA＝看多遠、EPS＝亂試的機率、EPISODES＝訓練幾局）\n", "GAMMA = 0.95 #@param [0.5, 0.8, 0.9, 0.95, 0.99] {type:\"raw\"}\n", "EPS = 0.05 #@param [0.0, 0.05, 0.1, 0.3] {type:\"raw\"}\n", "EPISODES = 8000 #@param [2000, 4000, 8000, 16000] {type:\"raw\"}\n", "R = dict(擊殺=擊殺, 命中敵人=命中敵人, 被打中=被打中, 每步存活=每步存活, 開砲=開砲, 靠近敵人=靠近敵人, 遠離敵人=遠離敵人, 陣亡=陣亡)\n", "run = tank_lab(NOTE, R, gamma=GAMMA, eps=EPS, episodes=EPISODES)" ] }, { "cell_type": "markdown", "id": "3654a835", "metadata": {}, "source": [ "## B2b｜🎯 進階：加上「某種情況才給分」的規則（P89）\n", "B2 的滑桿只能決定「每件事給幾分」。這裡可以再加上**最多兩條規則**：從選單選「在什麼情況下」，用拉桿決定給幾分（正＝鼓勵、負＝懲罰），再按 ▶ 訓練。\n", "\n", "| 挑戰 | 基本獎勵 | 規則 | 先猜 |\n", "|---|---|---|---|\n", "| 1 獎勵瞄準 | 只看輸贏 | 「對準敵人的每一步」+0.3 | 它會開砲嗎？為什麼？ |\n", "| 2 血少就逃 | 只看輸贏＋位能型提示 | 「血只剩 1 時靠近敵人」−2、「血只剩 1 時遠離敵人」+2 | 勝率會變高還是變低？ |\n", "| 3 自己想 | 任選 | 任選兩條 | 寫下理由，再看排行榜 |" ] }, { "cell_type": "code", "execution_count": null, "id": "e73457e6", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎯 B2b｜選規則＋拉桿，再按 ▶\n", "BASE = '只看輸贏' #@param ['只看輸贏', '只看輸贏＋位能型提示（靠近 +1、遠離 −1）', '設計得不錯（命中、被打中、每步都有分）']\n", "RULE_1 = '對準敵人的每一步' #@param ['（不用）', '對準敵人的每一步', '開砲但沒打中', '血只剩 1 時靠近敵人', '血只剩 1 時遠離敵人', '離敵人 3 格以內的每一步', '離敵人超過 3 格時靠近']\n", "SCORE_1 = 0.3 #@param {type:\"slider\", min:-5, max:5, step:0.1}\n", "RULE_2 = '（不用）' #@param ['（不用）', '對準敵人的每一步', '開砲但沒打中', '血只剩 1 時靠近敵人', '血只剩 1 時遠離敵人', '離敵人 3 格以內的每一步', '離敵人超過 3 格時靠近']\n", "SCORE_2 = 0.0 #@param {type:\"slider\", min:-5, max:5, step:0.1}\n", "R, NAME = rule_reward(BASE, [(RULE_1, SCORE_1), (RULE_2, SCORE_2)])\n", "run = tank_lab(NAME, R)" ] }, { "cell_type": "markdown", "id": "a06ef91a", "metadata": {}, "source": [ "### 🏆 完整排行榜（B1、B2、B2b 所有回合）" ] }, { "cell_type": "code", "execution_count": null, "id": "5140da86", "metadata": {}, "outputs": [], "source": [ "tank_board(top=30)" ] }, { "cell_type": "markdown", "id": "dd7ccf66", "metadata": {}, "source": [ "## B4｜先模仿、再強化：能超越老師嗎？（P112）\n", "老師是一台**不完美**的坦克（勝率約七成，像期中的規則策略）。按 ▶ 會畫出兩條線，比較「強化學習練了幾局」時的勝率：\n", "\n", "- 🔴 **從零開始強化**：不看示範，自己亂試、照獎勵學。\n", "- 🟢 **先模仿、再強化**：先照抄老師當起點，再自己練（AlphaGo、虛擬果蠅的路線）。\n", "\n", "兩條虛線是**老師**和**只模仿（照抄老師）**的勝率。\n", "\n", "| 任務 | 改哪裡 | 先猜，再按 ▶ |\n", "|---|---|---|\n", "| 1 | 不用改 | 哪一條線先超過老師？從零開始要練幾局？ |\n", "| 2 | `DEMO_GAMES`（老師示範幾局）10 vs 200 | 示範越多，起點越好嗎？ |" ] }, { "cell_type": "code", "execution_count": null, "id": "bc37a27f", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎛️ 先模仿、再強化（每次約 20 秒）\n", "DEMO_GAMES = 50 #@param [10, 50, 200] {type:\"raw\"}\n", "imitation_rl_demo(DEMO_GAMES)" ] }, { "cell_type": "markdown", "id": "d6d77618", "metadata": {}, "source": [ "# C｜冰湖：地板會滑的世界（P116）\n", "**目標**：4×4 的冰湖，從左上角「起」走到右下角「寶」（+1 分），掉進「洞」就結束。AI 用 Q 學習自己摸索。\n", "\n", "**難的地方**：地板會滑。想往某個方向走時，有 `SLIP` 的機率滑到**左邊或右邊**（和想走的方向垂直）。\n", "\n", "按 ▶ 會畫兩張圖：**左邊是地板不會滑時學到的路線，右邊是照你設定的 `SLIP` 學到的路線**。箭頭＝AI 在每一格想走的方向；**橘色格子＝AI 故意朝牆走**，下面會印出原因。最後有一段**動畫**：照右邊的路線實際走一局，綠色箭頭是想走的方向，紅色虛線是滑倒後實際滑去的方向。\n", "\n", "| 任務 | 改哪裡 | 先猜，再按 ▶ |\n", "|---|---|---|\n", "| 1 | `SLIP` 0.33 → 0.5 | 越滑，朝牆走的格子會變多還是變少？平均步數呢？ |\n", "| 2 | `STEP_COST`（每走一步扣的分）0 → 0.05 | 有時間壓力時，它會冒險走近路，還是照樣繞路？ |\n", "| 3 | `GAMMA` 0.5 vs 0.99 | 太短視時，還走得到寶藏嗎？ |" ] }, { "cell_type": "code", "execution_count": null, "id": "3e90d28c", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎛️ 冰湖\n", "SLIP = 0.33 #@param [0.0, 0.33, 0.5] {type:\"raw\"}\n", "GAMMA = 0.95 #@param [0.5, 0.9, 0.95, 0.99] {type:\"raw\"}\n", "STEP_COST = 0.0 #@param [0.0, 0.01, 0.05] {type:\"raw\"}\n", "lake_demo(SLIP, GAMMA, STEP_COST)" ] }, { "cell_type": "markdown", "id": "2558fa31", "metadata": {}, "source": [ "# D｜選做：看專家的示範，反推它在乎什麼（逆強化學習，P106）\n", "**情境**：有一台很會打的專家坦克，我們**看不到它的獎勵**，只看得到它每局平均做了什麼（每局走幾步、開砲幾次、命中敵人幾次……）。能不能從這些行為，**猜出一組獎勵**，教出一台差不多的坦克？\n", "\n", "**每一輪的做法**（P107）：\n", "1. 讓 Actor 用目前的獎勵學完，看它每件事做了幾次。\n", "2. 和專家比：**專家做得比較多的事，把它的獎勵調高；Actor 做得比較多的事，把它的獎勵調低**。\n", "3. 用新的獎勵重新訓練 Actor，回到第 1 步。\n", "\n", "按 ▶ 會一輪一輪印出：Actor 學到什麼、勝率多少、和專家差最多的是哪件事、獎勵怎麼調。打勾決定「獎勵可以看哪些事」：這 8 件事和 **B2 的 8 個滑桿同名**，IRL 反推出來的，就是這 8 個滑桿該設多少。\n", "\n", "> ⚠️ **沒勾＝不算分，不是不能做。** 坦克的動作（前進、轉向、開砲）永遠都在。取消「開砲」只代表「每按一次開砲」不加分也不扣分；Actor 為了拿「命中敵人」「擊殺」的分數，還是會開砲、還是打得死敵人——只是不會為了開砲本身亂打。\n", "\n", "| 挑戰 | 改哪裡 | 先猜 |\n", "|---|---|---|\n", "| 1 | 8 件事全勾 | 第幾輪會被鑽？下一輪會怎樣？ |\n", "| 2 | 取消勾選「開砲」 | 還會被鑽嗎？為什麼？ |\n", "| 3 | 只勾「擊殺」和「陣亡」 | 只看最後結果，反推得穩嗎？ |" ] }, { "cell_type": "code", "execution_count": null, "id": "d15cde8d", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🎛️ 逆強化學習：勾選獎勵可以看的事，再按 ▶（每次約 10 秒）\n", "#@markdown **獎勵可以看哪些事**（打勾＝可以用來反推）\n", "命中敵人 = True #@param {type:\"boolean\"}\n", "被打中 = True #@param {type:\"boolean\"}\n", "開砲 = True #@param {type:\"boolean\"}\n", "靠近敵人 = True #@param {type:\"boolean\"}\n", "遠離敵人 = True #@param {type:\"boolean\"}\n", "每步存活 = True #@param {type:\"boolean\"}\n", "擊殺 = True #@param {type:\"boolean\"}\n", "陣亡 = True #@param {type:\"boolean\"}\n", "ROUNDS = 5 #@param {type:\"slider\", min:2, max:6, step:1}\n", "irl_lab(dict(命中敵人=命中敵人, 被打中=被打中, 開砲=開砲, 靠近敵人=靠近敵人, 遠離敵人=遠離敵人, 每步存活=每步存活, 擊殺=擊殺, 陣亡=陣亡), rounds=ROUNDS)" ] }, { "cell_type": "markdown", "id": "6478ef9e", "metadata": {}, "source": [ "### 🔁 把反推的獎勵拿去 B2 訓練一次\n", "IRL 反推出來的 8 個數字，就是 B2 那 8 個滑桿（擊殺、陣亡、命中敵人、被打中、每步存活、開砲、靠近敵人、遠離敵人）的設定。按 ▶ 會用這組獎勵重新訓練坦克，結果加進 B 的排行榜，和你自己設計的獎勵比一比。" ] }, { "cell_type": "code", "execution_count": null, "id": "cb70e1c0", "metadata": { "cellView": "form" }, "outputs": [], "source": [ "#@title 🔁 用反推的獎勵訓練一次（要先跑上一格）\n", "run = irl_to_b2()\n", "tank_board(top=10)" ] }, { "cell_type": "markdown", "id": "d4d6bd2d", "metadata": {}, "source": [ "# E｜反思與繳交（課後）\n", "\n", "用自己的話回答，引用你在排行榜、動畫、B4 或冰湖看到的數字。繳交可檢視的 Colab 連結。" ] }, { "cell_type": "markdown", "id": "3d3bfce6", "metadata": {}, "source": [ "✍️ **作答方式**：每題下面那一格是文字格，**雙擊**它，把「（在這裡寫你的答案）」換成你的回答，再按 Shift＋Enter。可以換行、可以貼數字。" ] }, { "cell_type": "markdown", "id": "df5eb7f8", "metadata": {}, "source": [ "### ① 三個陷阱裡，坦克各學到什麼？為什麼？" ] }, { "cell_type": "markdown", "id": "86262654", "metadata": {}, "source": [ "（在這裡寫你的答案）" ] }, { "cell_type": "markdown", "id": "b685d28d", "metadata": {}, "source": [ "### ② 你的最佳獎勵設計與理由（引用排行榜的勝率、步數）" ] }, { "cell_type": "markdown", "id": "66ca6402", "metadata": {}, "source": [ "（在這裡寫你的答案）" ] }, { "cell_type": "markdown", "id": "43a9c734", "metadata": {}, "source": [ "### ③ B4：先模仿再強化，為什麼比較快？對期中有什麼啟發？" ] }, { "cell_type": "markdown", "id": "8aaa5b17", "metadata": {}, "source": [ "（在這裡寫你的答案）" ] }, { "cell_type": "markdown", "id": "e4aa8e77", "metadata": {}, "source": [ "### ④ 如果期中計分改成「擊殺×2 − 0.5×被擊殺」（沒有血量分），坦克可能怎麼鑽？" ] }, { "cell_type": "markdown", "id": "9119b307", "metadata": {}, "source": [ "（在這裡寫你的答案）" ] }, { "cell_type": "markdown", "id": "994042bd", "metadata": {}, "source": [ "## 延伸閱讀\n", "- 李宏毅（台灣，國立臺灣大學），ML 2021：Deep Reinforcement Learning（YouTube，2021）\n", "- OpenAI（美國），Faulty Reward Functions in the Wild，2016（賽艇繞圈刷分）\n", "- Ng, Harada & Russell（美國，UC Berkeley），Policy invariance under reward transformations，ICML 1999\n", "- Ho & Ermon（美國，史丹佛大學），Generative Adversarial Imitation Learning，NeurIPS 2016\n", "- DeepSeek-AI（中國），DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning，Nature 2025\n", "- Vaxenburg et al.（美國，HHMI Janelia／Google DeepMind），Whole-body physics simulation of fruit fly locomotion，Nature 2025" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }