Hi everyone. Fourth update on the Diablo I RL project. Parts 1-3 are covered here:
- Part 1 - from random walk to stair-finding
- Part 2 - full exploration of the first dungeon level
- Part 3 - monsters, potions, first real combat
This time the agent is running a full game. Ironman: no town returns, one life, dropped item stays on a floor, all 16 dungeon levels in sequence. Best result so far is dungeon level 9. Death distribution over 97 runs:
level 2: 4 ( 4.1%) ####
level 3: 6 ( 6.2%) ######
level 4: 19 ( 19.6%) ###################
level 5: 19 ( 19.6%) ###################
level 6: 22 ( 22.7%) ######################
level 7: 15 ( 15.5%) ###############
level 8: 9 ( 9.3%) #########
level 9: 3 ( 3.1%) ###
runs=97 min=2 max=9 mean=5.49 std=1.67
The video at the top shows one of those runs - the agent going from level 1 to level 9 where it dies. It is shortened: only a few combats are included, the walking between them is cut. Each clip was recorded from a checkpoint saved at the start of that level, so the fights you see are a live replay, not the original sequence. The agent makes the same decisions but the outcome can differ.
Why Ironman
Partly because it sounds cool, partly because it is actually the easiest mode to implement. No town returns means no town navigation code to write. One life means there is no ambiguity about whether a run succeeded or failed. The hero goes down and keeps going down, and the agent never has to decide whether to backtrack. Fewer decisions to implement means fewer things to get wrong.
One important thing to mention: quests are disabled. In Diablo 1 some quests rewrite the dungeon layout in ways that block progression. For example Chamber of Bone (is on floor 6) or Halls of the Blind (is on floor 7), both generate rooms where a staircase is hidden behind a door you can only open by completing the quest. The Butcher and Skeleton King are in the same category: special rooms that require quest logic to navigate. An algorithm for that is doable, but it adds complexity. Disabling quests keeps the dungeon layout predictable: stairs are always reachable. Well, yeah, no Butcher does make the survival numbers look better - not going to pretend otherwise :)
The hybrid approach
The agent is not a pure RL model. The philosophy is: if something can be written as an algorithm, write it as an algorithm, and only send the genuinely hard parts to the neural network.
The supervisor is a state machine. Most of the time it runs in DUNGEON mode, where the RL model is in control. Once the per-level step budget runs out or enough monsters are killed, it switches to PATHFIND and navigates to the last seen staircase using BFS. The model itself never sees the staircase trigger - it is masked as a wall so the model does not accidentally walk into it mid-fight. The supervisor decides when to descend.
The RL model
The model handles all the dungeon decisions: where to move, when to attack, which unexplored corridor to try next, whether to open a door, a chest, or a sarcophagus, and when to pick up an item. Potions are also the model's call - it observes the current potion count (small, large, rejuvenation) and issues a restore-health or restore-mana command when it decides the moment is right; the engine patch translates that into an actual potion drink. A CNN processes the 21x21 local observation window and an LSTM carries memory across steps, so what the agent saw earlier affects what it does now.
As covered in Part 3, a plain CNN was not enough. Three attention blocks sit on top of it - SelfAttention, MemoryCrossAttention, and FiLM. The ablations from Part 3 showed that FiLM does almost all the work. The other two turned out to be dead weight in the trained model, which was a surprise.
Two separate models were trained: one for floors 1-4 (Cathedral) and one for floors 5-8 (Catacombs). The Cathedral and Catacombs have different geometry, different monster pools, and the hero's stats look different at those floors too. One model covering all of it was worse than two specialized ones. For floors 9 and beyond the agent falls back to the Cathedral model - it gets there in about 3% of runs, not enough to justify a dedicated training run. The model switches on every level transition and the LSTM state resets.
Stat strategy: dexterity rush
Every level-up gives 5 attribute points to spend. The agent uses dex-rush, a 30-year-old Diablo 1 player technique: put Dexterity first, everything else later. Dexterity feeds directly into the attack rating formula that decides whether hits land. On floors 1-4 enemies have low armor class, so stacking Dexterity means the Warrior hits more often. Fights end faster, which means less damage taken. It sounds simple and it is - that's why it works.
I tried strength-first and balanced splits as alternatives. Dex-rush won. The agent does not learn this - it is hardcoded and runs automatically on every level-up.
Gear management
This is the part that went through the most rewrites.
The first version scored items by a weighted sum of their stats - dexterity weighted higher than vitality, and so on. Simple, fast, and it worked okay on early floors. On deeper floors it started failing in ways that were hard to debug, because stat interactions in Diablo are not linear.
So the idea was to take the original code from the DevilutionX engine, where the game actually computes attack rating, damage, and armor class, then translate it all into Python. The result is a function that computes the hero's expected combat efficiency against the average monster on the current floor:
(hero_hp * hero_damage_per_hit * hero_hit_chance) / (monster_hp * monster_damage_per_hit * monster_hit_chance)
Per-floor average monster stats were measured from the actual engine across 200 seeds per floor. When a new item shows up in inventory the agent virtually equips it, recomputes that number, and compares to the current value. If it goes up, the item is worth equipping.
The reason this matters: a +10 Dexterity ring might be excellent or worthless depending on whether the hit chance is already capped at 95%, whether the hero has a shield, and what character level we are at. A weighted sum has no way to know. The full formula just handles it.
Gear degrades. The agent repairs algorithmically when durability drops below a threshold - no model involvement there.
Dropped items are immediately masked in the model's observation so the model does not walk back to pick them up.
Since there is no town and no shop, gold is useless and everything that cannot be equipped right now gets dropped. The inventory stays empty. Unidentified items are a special case though - the agent can still equip them using their visible stats (base armor class, raw damage range). The risk is that the item might be cursed, so when equipping an unidentified item, the previous item is kept as a backup in inventory. If the new item gets identified and reveals some cursed chars, the agent drops it and falls back to the backup, so the hero is never naked in a fight.
Reward shaping
I started with sparse reward - only terminal outcomes fire, the model figures out the rest. That got me pretty far. Then it stalled.
Shaping from there was mostly trial and error. After each training run I dump all the action events from the whole run into a table - attacks per episode, items collected, wasted actions, everything. It sounds simple but it is actually quite useful: if the model is spamming fireball on empty mana eight times per episode, the table tells you immediately and the fix is obvious. If wasted attacks go down after you raise the penalty, you know the shaping worked. If they don't, something else is going on.
Event frequency (86451711 total events, 512 files, 164452 all train episodes):
freq per_ep sum_R label
--------------- ------ ---------- ------------------------------
48.3%(41767256) 253.98 -1253017.7 Movement penalty
21.7%(18780064) 114.20 939003.2 Explore tiles
14.5%(12541833) 76.26 1881274.9 Attack 1 monster(s)
5.0%( 4311508) 26.22 -1221552.1 Damage taken
4.9%( 4204562) 25.57 421077.9 Kill monster
1.7%( 1471523) 8.95 117744.9 Collect item
1.1%( 910983) 5.54 75545.4 Activate object
0.9%( 738467) 4.49 14814.6 Open door
0.8%( 676459) 4.11 -101468.8 Wasted primary
0.5%( 410848) 2.50 20542.4 Correct restore HP
0.2%( 188687) 1.15 -18868.7 No-potion restore HP
...
Some changes were effective, while others were not. Penalizing phasing when no monster was nearby worked. The crowd-fighting amplifier, which scaled damage in dense groups, was tested but eventually discarded. It turns out that reducing the hero's stats during training accomplishes the same goal without needing to adjust rewards: the hero naturally takes more damage and learns to lure effectively.
The challenging part was balancing the movement penalty against the exploration reward. I introduced an exploration bonus in version 16, but the per-step calculations still made standing still slightly optimal. I had to adjust both numbers before the agent actually started moving toward new tiles.
The last working environment version is 17. The event table and per-level outcome stats are what made it possible to iterate without going blind:
Per-level outcomes (164452 all train episodes, success: 50.4%):
Level Goal Death Escape Stuck Timeout
---------------------------------------------------------------------
5 65.3%(26973) 31.1%(12841) 0.4%(148) 2.9%(1217) 0.3%(141)
6 49.6%(20433) 46.8%(19273) 0.3%(109) 3.0%(1242) 0.3%(143)
7 47.5%(19548) 49.6%(20420) 0.3%(134) 2.2%(891) 0.4%(177)
8 39.2%(15959) 56.4%(22982) 0.4%(167) 3.4%(1381) 0.7%(273)
Each floor tracks goal/death/stuck/timeout split independently, so if level 6 drops from 50% clears to 40% after a reward change, something broke there specifically.
Monte Carlo simulation
Getting the training curriculum right required knowing what a real run actually looks like at each floor. How many XP does a Warrior earn on floors 1-4? What character level do they typically enter floor 5 at? What gear is realistically available?
I wrote a simulator (scripts/diablo-sim.py in the repo) that runs a Warrior through all 16 floors in a Markov loop: sample XP from monster kill counts measured from the real engine, apply the level-up formula, track gear carried forward. After enough simulations you get a distribution of hero state at each floor entry.
Training at the wrong character level produces an agent that works in training and falls apart in real runs. The simulator fixed that.
Discovered behaviors
The agent has discovered a solid strategy for staying alive: it lures
everything into a narrow doorway and waits. Just stands there. The
monsters walk over one by one and it picks them off. No reward shaping
for that, no explicit training on this - just happened. Also, when it's being heavily pressured by a gathered crowd in an open space, it tries to flee or to find another doorway to fight back.
Low health with no potions is something else to watch. It gets
genuinely scared. Hesitates in the corridors, takes a few steps
forward, steps back, and if something does wander close it immediately
turns and runs away. Obviously does not want to die :) Which is
technically correct behavior but looks almost funny (poor guy).
Interestingly, while most monsters can usually be lured, meaning the
agent just has to stand and they'll come, there are some that don't
react at all unless you get very close. I watched the agent open a
door, see a room full of Gargoyles, and then... wait. The Gargoyles
were dormant, motionless. This went on for tens of seconds. Nothing
happened on either side. The agent was clearly expecting them to come
over like everyone else does. They did not. It has not seen enough of
these monsters in training and just assumed the usual rules
apply. Fixing this would mean making sure those types show up more
often during training.
The video at the top has most of these behaviors, very interesting to
watch, especially seen on deeper levels.
What is next
Honestly, I am stuck. For the last couple of months I have been trying to push survival on floors 5-8 and the training curve just flattens. Nothing I have tried moves it. I do not know yet whether this is a ceiling for the current architecture or just something I am missing. The Catacombs are the bottleneck - reaching floor 9 in 3% of runs is a consequence of floors 5-8 being hard, not floors 9-12 being impossible.
The obvious workaround is to allow town returns and trading. Better gear and more healing potions would help a lot. But that means writing town navigation and trading code, and it feels like giving up on the Ironman idea. The goal was still to kill Diablo in one unassisted run. Whether that is reachable with this approach, I do not know yet.
The repo is here: https://github.com/rouming/DevilutionX-AI
