I made a beginner-friendly visual explanation of reinforcement learning—the type of machine learning behind game-playing AI, robotics, and parts of chatbot training.

Instead of starting with equations, it uses a robot-in-a-maze example to explain:

• Agents, environments, actions, and rewards

• How trial and error improves performance over episodes

• Exploration vs exploitation

• Delayed rewards and policies

• Reward hacking and other risks

Watch here: https://youtu.be/a_MSTPB5MyA

If you like the explanation, please like, share, and subscribe to my channel for more visual AI and machine learning videos.

A couple of days ago I posted a PPO agent landing Falcon 9 on a droneship. I wanted to extend the simulation to a more complex problem, one that needs two agents that depend on each other. So I built Super Heavy being caught by the tower arms (Mechazilla). One agent flies the booster, the other controls the arms and carriage.

Setup: PPO in JAX on a single GPU, heavy domain randomization (wind, gusts, engine-outs, failed relights, sensor noise and delay, mass and thrust variation, limited propellant), and every attempt starts a few km up with the engines off.

How it went

I quickly found out I can’t train both from scratch at the same time. The tower has nothing to learn from if the booster never shows up, and the booster never gets rewarded for a catch if the arms never close. So I did it in steps.

1. Booster first, with a simple rule-based tower. The tower just closed the arms when the booster got close, which let the booster learn the whole approach. Even this took a while. With short rollouts (1.6 s) it kept forgetting how to catch, because a catch takes several seconds of slow, steady descent while the arms close. Longer rollouts (6.4 s, λ = 0.99) fixed that.
2. Then the tower, with the booster frozen. Trained from scratch, at first it just swung the arms around and kept hitting the booster. I also tried fading the rule-based tower out slowly, but it fell apart the moment the help was gone. After a few fixes, the from-scratch tower won, and it ended up catching more reliably than the rules did.
3. Then the booster again, next to the learned tower. This one surprised me. It got worse fast. It started coming in slightly too fast and slid off the arms. A much smaller learning rate fixed it, and the two together now do clearly better than with the rule-based tower.

Everything runs live in the browser, so you can watch both agents: https://qforge-dev.github.io/hoverslam

There is also a playground for my previous Falcon simulation.

Happy to answer questions. I’m not an ML expert, so feedback on the multi-agent part is very welcome.

This is a follow-up to my previous post where I introduced a high-parallel Rocket League simulator for reinforcement learning. I said in the comments there that I was working on a write-up with benchmarks, and the link is embedded in this post!

In the blog post I go over the throughput gains on PPO workloads, cover the interface (as always it's just a Gymnasium wrapper), and detail some of the changes to the reference CPU simulation, RocketSim, which make CARL better suited for execution on the GPU.

jwmccarthy.github.io will also contain posts going forward about explorations into RL for Rocket League if anyone is interested in seeing where this goes. Thanks for reading :D

Hi,

I would like to announce that I have released Metis-Core v0.4.1-alpha. This version includes some bug fixes and a new example: DQN_Pong. Do you remember the game of Pong?

https://github.com/felixromo314/METIS-Core/

In this example, using the DQN of Metis-Core, the AI paddle learns to play Pong.

For the graphics I used the raylib library (https://www.raylib.com/), created by Ramon Santamaria (u/raysan5). It is very easy to use and powerful: with just a few functions you have the window, the paddles and the ball on the screen, and you can watch the AI learning in real time.

In case you are planning to make a similar game, or a proof of concept with DQN, I would like to invite you to have a look at this step-by-step tutorial that explains how to implement DQN with Metis-Core:

https://github.com/felixromo314/METIS-Core/blob/main/tutorial/Tutorial-DQN.md

I hope you like it!

Best,
Felix

https://github.com/vahmoh25/NeuroMario

This repository demonstrates how neuroevolution trains neural networks to master Super Mario. Watch the AI learn and play in a 3D environment, and explore its neural network through an interactive 3D visualization. Super cool!

While working on a research project I kept writing the same preprocessing code over and over which pulled EHR tables, line up vitals/labs by the hour, work out what counts as an action, attach a reward and then get it all into a shape d3rlpy would accept. I tried looking for a library that already did this and couldn’t find one. Healthcare ML tools are mostly built for supervised prediction and offline RL libraries assume you already have a clean state/action/reward dataset. So I built the piece in between.

What it does:

• Builds hourly patient trajectories (states, actions, rewards, terminals) from MIMIC-IV v3.1 through BigQuery. Every query gets a dry run first and a hard maximum_bytes_billed cap, so a bad cohort query can’t quietly run up a bill.
• Builds actions from ICU input events: vasopressor and fluid doses, binned with edges you choose.
• Treats rewards as swappable: mortality, per-step SOFA change, 30-day readmission, or a weighted mix of these.
• Estimates the observed clinician (behavior) policy from the data, which you’ll want for off-policy evaluation.
• Exports straight to a d3rlpy dataset, with an optional JSON sidecar recording where the data came from.

You don’t need MIMIC access to try it. There’s a synthetic MIMIC-style data generator.

Docs: https://pinktownscavenger.github.io/ehr2rl/
GitHub: https://github.com/pinktownscavenger/ehr2rl

I’d really like feedback from anyone who has done offline RL on clinical data, especially on reward design and how I’ve discretized the actions. It’s a research tool and isn’t meant for making treatment decisions.

Hi all, I’m a founder exploring a company that collects egocentric (first-person) video of real manual work for robot learning. Before building anything, I want to learn from people who actually train on this data or choose data vendors.

Looking for: ML/research engineers or data leads at robotics, humanoid or embodied-AI teams (labs or startups).

What we’d talk about (30 min):

Details:

To sign up: DM me your role and company type (e.g. “ML engineer, humanoid startup”). I’ll confirm through LinkedIn or a work email, whichever you prefer. That’s used only to verify the role, never shared.

Update since last post, skip to 20s to see the final training trends, and the deployment is in this post.

Some lessons since last time:

• Jiggling in GUI during training is because of parameter exploration
• Always test with 0 action first to see if the implementation is correct, especially with all those link indices flying around
• Even then, there are still dangerous hidden bugs, like adding observation noise accidentally affecting the control, which needs a lot of logging to debug

Features added since last time:

• Adding domain randomizations on observation noise, initial link positions, and especially randomizing horizontal forces up to 5N (for a 3 kg robot) indicated by the red arrows in the video
• Curriculum scheduling from only penalty_joint_pos_init, to expanding to increasing the domain randomizations and making reward/penalty tighter over training iterations
• Adding a penalty on torque distribution for the robot to use all 4 legs
• Adding a penalty on feet slipping on the ground
• Simple low-pass filter on action
• Adding ent_coef to enhance exploration after the curriculum scheduling maxes out
• Skipping the first few steps for training because the simulation needs to initialize first (like the robot clipping into the ground)
• Logging the cause of termination. Showing 20% from pitch > 45 deg because of the force pushing it on the left/right-hand side
• Logging the curriculum schedule values
• Adding .csv for logging; can be thrown to LLM for it to analyze the trend
• Export to ONNX file for deployment. Adding TRAIN_CONFIG in ONNX file to record all the training configuration (too bad it is hard to do the same for vector normalization, which needs an extra file)

GitHub: configuration file, training code, and deployment code
Music credit

An RL agent learning to land a Falcon 9 on a droneship. Everything is simulated: rocket physics, waves, wind, fuel slosh, sensor noise. No real flight data, just trial and error.

It’s a hoverslam, so the booster can’t hover and has to hit zero speed right at the deck.

PPO, ~80k parameter policy, 10 controls (throttle, gimbal, grid fins, thrusters, legs, ignition), 67 world parameters randomized every attempt.

Out of 59M training flights: 34.7M splashed, 10.8M exploded, 13.2M landed.

Still weak in strong wind and high waves, working on that now. Happy to answer questions.

MARL has been shown a lot in video games and simplified simulation environments, but can it be deployed in actual robots for complex real-world industrial applications?

We recently investigated decentralized task assignment and navigation for multi-robot machine tending, introducing FMAPPO, which combines 2D LiDAR observations with task-specific state information.

The work uses Isaac Sim/Lab for training in simulation, then deploys to two physical robots with ROS 2. One aspect we investigate is how policy performance changes with the command update frequency during deployment.

Paper: https://arxiv.org/abs/2609.14567

Videos of the real deployment are here.

I'm interested in discussing the practical challenges of sim-to-real RL, particularly policy robustness under real sensing and control constraints.

The implementation isn't publicly available yet, but I plan to release it after the paper is accepted.

I'm fine-tuning Qwen3.5-0.8B to play a Game Boy boss fight using structured state inputs and 18 discrete button combinations.

After behavior cloning, I ran 15 PPO iterations using 171 training episodes. The selected policy improved from 39/50 to 47/50 wins on the original encounter in separate greedy evaluations. This is one training run, with the checkpoint selected after evaluation.

What puzzled me was the critic. Held-out explained variance stayed inconsistent and often negative, ending around 0.04. It's a small MLP over detached policy features, evaluated on held-out whole episodes. Each iteration collects roughly 512 decisions across 10–13 episodes. Rewards include boss hits, completion, damage and elapsed time.

What would you check first to determine whether the critic is limiting further learning?

I'm considering GRPO, but I'd like to understand whether removing the critic addresses the problem before switching algorithms.

The plot shows critic fit alongside five sampled-action development games per checkpoint, not the separate 50-game evaluations above. The held-out episodes and return targets change between iterations.

Methods, implementation and evaluation results: https://jintaili.github.io/kittie-game/training/

It was a project of mine to make an AI for the game Sandtris. Now that I have a decent bot, I wanted to share it so here it is!
I built a this as a small web game. You can play it immediately in your browser: www.sandtrisduel.lol. Fully free.
I would love to get feedback on it, and actually have some other people play around with the bot.
Also if you are curious about it, I will be sharing how the model is exactly working soon.

I work on Flame, an open-source distributed engine for elastic workloads. We added a Search-R1-style training example that teaches Qwen2.5-3B-Instruct to search a BM25 corpus while answering HotpotQA questions.

The training loop runs as Python services: vLLM executors generate trajectories, PyTorch trainer replicas update the policy using group-relative advantages, and Flame’s object cache carries batches and weight updates.

The interesting infrastructure pieces:

• Warm executors: models, retrieval indexes, and optimizer state stay loaded between calls.
• Elastic rollouts: generation capacity can grow with pending work.
• Gang-scheduled trainers: the trainer group receives its minimum resources together, avoiding partially scheduled collective participants.
• Incremental weight sync: after the first full export, existing rollout executors fetch only changed bf16 values.

In a 100-step run using 14 rollout GPUs and two trainer GPUs, held-out exact match improved from 14.5% to 47%, averaging 24.3 seconds per step after the first two steps. Steady-state weight publication was approximately 165 MB per step, versus a 6.17 GB full model.

Caveats: one seed, 200 evaluation questions, and a corpus built from the selected questions’ context paragraphs. These scores aren’t directly comparable with published Search-R1 benchmarks. The synchronous loop also leaves substantial GPU idle time.

• Blog and results
• Runnable example

For others building agent-training systems: how are you handling weight synchronization and the tradeoff between synchronous updates and policy lag?

I'm building Bitbots, a browser game where players assemble robots from parts and train them to move using reinforcement learning.

This video shows the game's “brain view”: the policy's inputs, selected hidden-layer activations, and motor outputs, synchronized with the robot's movement. These are recorded values from the actual policy, not a decorative animation.

The idea is to let players see more than just whether their robot walks or falls over. They can watch how the network's activity changes alongside its movement. It isn't an explanation of what individual neurons “mean,” but I wanted to give players a way to look inside the controller.

The current simulation uses MuJoCo, with GPU training through MJX's Warp backend. Players choose the robot's body and joint configuration; the policy learns how to control it. During races, the robot acts autonomously.

For people working on locomotion across different morphologies: how would you support creative, unconventional robot designs under a limited training budget? I’d like players to experiment with weird bodies and joint configurations, rather than feel pushed toward a few designs known to work. How would you approach this without making the reward so prescriptive that every robot ends up moving the same way?

I'm the developer. Bitbots is currently in closed alpha, and free invites are available by comment or DM if you'd like to experiment with your own robot designs.

Project page: https://bitbots.endresgames.com

EDIT: EDIT: Wow, thank you all for the interest, it means a lot! Bitbots is a passion project, and I'd love your ideas and feedback. Join the Discord for updates and alpha invites: https://discord.gg/zgxpuhgB8U 💡

How can agents adapt when the interactions between them keep changing? Here, σ controls the severity of these changes. PACT tracks how agents affect one another through shared roads, power networks or robotic systems, then adjusts their actions to compensate. These demos show PACT helping existing reinforcement-learning agents handle changing interactions in collective urban fleet routing, power-grid control, cooperative lifting and robot locomotion.

During my MSc in Digital Agriculture at UNICAMP, I studied reinforcement learning for fraud prevention in crop insurance. The key modeling choice was to include the remaining budget in the state: approving one claim changes what can be funded later.

I built an inspectable Gym-style environment with approve/reject actions, claim context, budget use, approval history, and a statistical signal. The public repository documents subsequent DQN and random-policy experiments.

Walkthrough and short browser demo: https://nandevers.github.io/research/masters/

Code: https://github.com/nandevers/gym-insurance

The browser demo is synthetic; no trained agent or fraud prediction is running there. Benford distance is a statistical proxy, not a fraud label. Confirmed labels and independent evaluation would be needed to claim fraud-detection performance.

For a finite budget, would you model it as an explicit state/constraint or use a reward penalty? What baseline or ablation would you want before comparing policies?

AI helped edit this post; the research and code are mine.

I built a real-time war game where an LLM commands one nation against 42 others. Nations make pacts and break them, and there is a nuclear doomsday clock: every nuke moves it closer to midnight, and whoever's strike reaches midnight is wiped out. So the move that ends the world always loses for the one who makes it.

I wanted to see what different rewards teach. All of this is small, 10 sealed wars per arm, so read it as a first look.

Imitation first (Qwen 7B, left side of the chart, the experiment from my earlier post here): trained only on DeepSeek wars where nobody struck midnight, it still struck midnight in 4 of 10 (the untrained model: 0). Most of the "good" wars still had nuke orders in them. Cutting the nuke turns out of the data got it to 0 of 10.

Then reward (gpt-oss-20b, 3 rounds). Each round it played a batch of wars, I kept its 6 best by the reward and fine-tuned on those turns with LoRA. So it is filtered self-imitation, not PPO.

Untrained: struck midnight 4 of 10, won 0, broke 3 pacts

Rewarded to win: struck midnight 6 of 10, won 0, broke 7 pacts

Rewarded to win with no nukes: struck midnight 0 of 10, won 0, broke 6 pacts

So rewarding the win made it more dangerous and no better at winning, and banning nukes removed midnight but it just moved the aggression into breaking pacts.

My question for people who know this better than me: how would you reward restraint without rewarding passivity, or pushing the bad behaviour somewhere else? Pacts broken looks like the obvious next term, but I suspect it just moves again.

The environment is on Prime Intellect's Environments Hub if anyone wants to try their own reward (prime env install clouthier-labs/second-strike). Rewards come from the game's own record, no LLM judge, and a seed always gives the same war.

I’ve been training a whole-body recovery policy for the Asimov 1 humanoid in MuJoCo/mjlab. Each episode starts from a randomized supine, prone, or lateral pose with additional joint perturbations. The objective is to reach an upright configuration and remain stable.

The first policy achieved a high reward, but rollout inspection exposed several problems: some initial states intersected the floor, knee-standing received too much reward, and the robot occasionally used joint limits for support. We corrected the reset distribution using forward kinematics, rejected invalid configurations, separated genuine recovery episodes from standing initialization, and revised the reward around pelvis height, torso orientation, posture, flat foot contact, smooth actions, and ankle-limit avoidance.

We also replaced idealized actuators with DC motor torque-speed models, added randomized control delay of up to 25 ms, and applied external pushes during training. Under the earlier actuator model, the policy recovered and maintained its stance in roughly 97% of randomized evaluation episodes. That number is no longer directly comparable after changing the actuator dynamics, so the current policy is being retrained and evaluated under the harder model.

The video shows several rollouts at normal speed, followed by the same recoveries at 0.25×. Code and training setup:
https://github.com/TenPaths/cyclotron/tree/feature/mjlab-standup

For people working on humanoid recovery: which perturbations and evaluation metrics would you add before attempting sim-to-real transfer?

Hi everyone. Fourth update on the Diablo I RL project. Parts 1-3 are covered here:

• Part 1 - from random walk to stair-finding
• Part 2 - full exploration of the first dungeon level
• Part 3 - monsters, potions, first real combat

This time the agent is running a full game. Ironman: no town returns, one life, dropped item stays on a floor, all 16 dungeon levels in sequence. Best result so far is dungeon level 9. Death distribution over 97 runs:

  level  2:   4 (  4.1%)  ####
  level  3:   6 (  6.2%)  ######
  level  4:  19 ( 19.6%)  ###################
  level  5:  19 ( 19.6%)  ###################
  level  6:  22 ( 22.7%)  ######################
  level  7:  15 ( 15.5%)  ###############
  level  8:   9 (  9.3%)  #########
  level  9:   3 (  3.1%)  ###

runs=97  min=2  max=9  mean=5.49  std=1.67

The video at the top shows one of those runs - the agent going from level 1 to level 9 where it dies. It is shortened: only a few combats are included, the walking between them is cut. Each clip was recorded from a checkpoint saved at the start of that level, so the fights you see are a live replay, not the original sequence. The agent makes the same decisions but the outcome can differ.

Why Ironman

Partly because it sounds cool, partly because it is actually the easiest mode to implement. No town returns means no town navigation code to write. One life means there is no ambiguity about whether a run succeeded or failed. The hero goes down and keeps going down, and the agent never has to decide whether to backtrack. Fewer decisions to implement means fewer things to get wrong.

One important thing to mention: quests are disabled. In Diablo 1 some quests rewrite the dungeon layout in ways that block progression. For example Chamber of Bone (is on floor 6) or Halls of the Blind (is on floor 7), both generate rooms where a staircase is hidden behind a door you can only open by completing the quest. The Butcher and Skeleton King are in the same category: special rooms that require quest logic to navigate. An algorithm for that is doable, but it adds complexity. Disabling quests keeps the dungeon layout predictable: stairs are always reachable. Well, yeah, no Butcher does make the survival numbers look better - not going to pretend otherwise :)

The hybrid approach

The agent is not a pure RL model. The philosophy is: if something can be written as an algorithm, write it as an algorithm, and only send the genuinely hard parts to the neural network.

The supervisor is a state machine. Most of the time it runs in DUNGEON mode, where the RL model is in control. Once the per-level step budget runs out or enough monsters are killed, it switches to PATHFIND and navigates to the last seen staircase using BFS. The model itself never sees the staircase trigger - it is masked as a wall so the model does not accidentally walk into it mid-fight. The supervisor decides when to descend.

The RL model

The model handles all the dungeon decisions: where to move, when to attack, which unexplored corridor to try next, whether to open a door, a chest, or a sarcophagus, and when to pick up an item. Potions are also the model's call - it observes the current potion count (small, large, rejuvenation) and issues a restore-health or restore-mana command when it decides the moment is right; the engine patch translates that into an actual potion drink. A CNN processes the 21x21 local observation window and an LSTM carries memory across steps, so what the agent saw earlier affects what it does now.

As covered in Part 3, a plain CNN was not enough. Three attention blocks sit on top of it - SelfAttention, MemoryCrossAttention, and FiLM. The ablations from Part 3 showed that FiLM does almost all the work. The other two turned out to be dead weight in the trained model, which was a surprise.

Two separate models were trained: one for floors 1-4 (Cathedral) and one for floors 5-8 (Catacombs). The Cathedral and Catacombs have different geometry, different monster pools, and the hero's stats look different at those floors too. One model covering all of it was worse than two specialized ones. For floors 9 and beyond the agent falls back to the Cathedral model - it gets there in about 3% of runs, not enough to justify a dedicated training run. The model switches on every level transition and the LSTM state resets.

Stat strategy: dexterity rush

Every level-up gives 5 attribute points to spend. The agent uses dex-rush, a 30-year-old Diablo 1 player technique: put Dexterity first, everything else later. Dexterity feeds directly into the attack rating formula that decides whether hits land. On floors 1-4 enemies have low armor class, so stacking Dexterity means the Warrior hits more often. Fights end faster, which means less damage taken. It sounds simple and it is - that's why it works.

I tried strength-first and balanced splits as alternatives. Dex-rush won. The agent does not learn this - it is hardcoded and runs automatically on every level-up.

Gear management

This is the part that went through the most rewrites.

The first version scored items by a weighted sum of their stats - dexterity weighted higher than vitality, and so on. Simple, fast, and it worked okay on early floors. On deeper floors it started failing in ways that were hard to debug, because stat interactions in Diablo are not linear.

So the idea was to take the original code from the DevilutionX engine, where the game actually computes attack rating, damage, and armor class, then translate it all into Python. The result is a function that computes the hero's expected combat efficiency against the average monster on the current floor:

(hero_hp * hero_damage_per_hit * hero_hit_chance) / (monster_hp * monster_damage_per_hit * monster_hit_chance)

Per-floor average monster stats were measured from the actual engine across 200 seeds per floor. When a new item shows up in inventory the agent virtually equips it, recomputes that number, and compares to the current value. If it goes up, the item is worth equipping.

The reason this matters: a +10 Dexterity ring might be excellent or worthless depending on whether the hit chance is already capped at 95%, whether the hero has a shield, and what character level we are at. A weighted sum has no way to know. The full formula just handles it.

Gear degrades. The agent repairs algorithmically when durability drops below a threshold - no model involvement there.

Dropped items are immediately masked in the model's observation so the model does not walk back to pick them up.

Since there is no town and no shop, gold is useless and everything that cannot be equipped right now gets dropped. The inventory stays empty. Unidentified items are a special case though - the agent can still equip them using their visible stats (base armor class, raw damage range). The risk is that the item might be cursed, so when equipping an unidentified item, the previous item is kept as a backup in inventory. If the new item gets identified and reveals some cursed chars, the agent drops it and falls back to the backup, so the hero is never naked in a fight.

Reward shaping

I started with sparse reward - only terminal outcomes fire, the model figures out the rest. That got me pretty far. Then it stalled.

Shaping from there was mostly trial and error. After each training run I dump all the action events from the whole run into a table - attacks per episode, items collected, wasted actions, everything. It sounds simple but it is actually quite useful: if the model is spamming fireball on empty mana eight times per episode, the table tells you immediately and the fix is obvious. If wasted attacks go down after you raise the penalty, you know the shaping worked. If they don't, something else is going on.

Event frequency (86451711 total events, 512 files, 164452 all train episodes):
freq             per_ep       sum_R  label
---------------  ------  ----------  ------------------------------
48.3%(41767256)  253.98  -1253017.7  Movement penalty
21.7%(18780064)  114.20    939003.2  Explore tiles
14.5%(12541833)   76.26   1881274.9  Attack 1 monster(s)
 5.0%( 4311508)   26.22  -1221552.1  Damage taken
 4.9%( 4204562)   25.57    421077.9  Kill monster
 1.7%( 1471523)    8.95    117744.9  Collect item
 1.1%(  910983)    5.54     75545.4  Activate object
 0.9%(  738467)    4.49     14814.6  Open door
 0.8%(  676459)    4.11   -101468.8  Wasted primary
 0.5%(  410848)    2.50     20542.4  Correct restore HP
 0.2%(  188687)    1.15    -18868.7  No-potion restore HP
 ...

Some changes were effective, while others were not. Penalizing phasing when no monster was nearby worked. The crowd-fighting amplifier, which scaled damage in dense groups, was tested but eventually discarded. It turns out that reducing the hero's stats during training accomplishes the same goal without needing to adjust rewards: the hero naturally takes more damage and learns to lure effectively.

The challenging part was balancing the movement penalty against the exploration reward. I introduced an exploration bonus in version 16, but the per-step calculations still made standing still slightly optimal. I had to adjust both numbers before the agent actually started moving toward new tiles.

The last working environment version is 17. The event table and per-level outcome stats are what made it possible to iterate without going blind:

Per-level outcomes (164452 all train episodes, success: 50.4%):
 Level          Goal         Death     Escape       Stuck    Timeout
---------------------------------------------------------------------
     5  65.3%(26973)  31.1%(12841)  0.4%(148)  2.9%(1217)  0.3%(141)
     6  49.6%(20433)  46.8%(19273)  0.3%(109)  3.0%(1242)  0.3%(143)
     7  47.5%(19548)  49.6%(20420)  0.3%(134)   2.2%(891)  0.4%(177)
     8  39.2%(15959)  56.4%(22982)  0.4%(167)  3.4%(1381)  0.7%(273)

Each floor tracks goal/death/stuck/timeout split independently, so if level 6 drops from 50% clears to 40% after a reward change, something broke there specifically.

Monte Carlo simulation

Getting the training curriculum right required knowing what a real run actually looks like at each floor. How many XP does a Warrior earn on floors 1-4? What character level do they typically enter floor 5 at? What gear is realistically available?

I wrote a simulator (scripts/diablo-sim.py in the repo) that runs a Warrior through all 16 floors in a Markov loop: sample XP from monster kill counts measured from the real engine, apply the level-up formula, track gear carried forward. After enough simulations you get a distribution of hero state at each floor entry.

Training at the wrong character level produces an agent that works in training and falls apart in real runs. The simulator fixed that.

Discovered behaviors

The agent has discovered a solid strategy for staying alive: it lures everything into a narrow doorway and waits. Just stands there. The monsters walk over one by one and it picks them off. No reward shaping for that, no explicit training on this - just happened. Also, when it's being heavily pressured by a gathered crowd in an open space, it tries to flee or to find another doorway to fight back.

Low health with no potions is something else to watch. It gets genuinely scared. Hesitates in the corridors, takes a few steps forward, steps back, and if something does wander close it immediately turns and runs away. Obviously does not want to die :) Which is technically correct behavior but looks almost funny (poor guy).

Interestingly, while most monsters can usually be lured, meaning the agent just has to stand and they'll come, there are some that don't react at all unless you get very close. I watched the agent open a door, see a room full of Gargoyles, and then... wait. The Gargoyles were dormant, motionless. This went on for tens of seconds. Nothing happened on either side. The agent was clearly expecting them to come over like everyone else does. They did not. It has not seen enough of these monsters in training and just assumed the usual rules apply. Fixing this would mean making sure those types show up more often during training.

The video at the top has most of these behaviors, very interesting to watch, especially seen on deeper levels.

What is next

Honestly, I am stuck. For the last couple of months I have been trying to push survival on floors 5-8 and the training curve just flattens. Nothing I have tried moves it. I do not know yet whether this is a ceiling for the current architecture or just something I am missing. The Catacombs are the bottleneck - reaching floor 9 in 3% of runs is a consequence of floors 5-8 being hard, not floors 9-12 being impossible.

The obvious workaround is to allow town returns and trading. Better gear and more healing potions would help a lot. But that means writing town navigation and trading code, and it feels like giving up on the Ironman idea. The goal was still to kill Diablo in one unassisted run. Whether that is reachable with this approach, I do not know yet.

The repo is here: https://github.com/rouming/DevilutionX-AI