Research idea: Can we reduce forgetting in recurrent LLM memory? Looking for critical feedback

Hey everyone!

I'm an engineering student working on a research proposal about long-context LLMs and recurrent compressed memory. I'd really appreciate technical feedback from people working on LLMs, continual learning, recurrent architectures, or memory systems.

The core question: Can we train an LLM's memory to retain previously learned information while continuing to absorb new information?

I'm calling the proposed training method GMR (Generational Memory Retention) and the evaluation protocol STRATA.

This is still a pre-experimental proposal. I haven't run the experiments yet, so I'm looking for feedback before committing to the implementation.

1. The problem I'm interested in

Imagine an LLM processing a long stream of information in small chunks.

A chunk introduces the following fact:

"The Kestrel project uses Python 3.11."

Immediately after processing that chunk, the model can correctly answer:

"Which Python version does the Kestrel project use?"

Now imagine that the model processes hundreds of additional chunks. Its memory has a fixed capacity, and each new update modifies the existing memory state.

Eventually, the model might no longer answer the original question correctly, even though its model weights haven't changed during inference.

I'm calling this state-induced forgetting: information that was initially available becomes inaccessible after successive memory-state updates.

The key question is whether we can reduce this type of forgetting without simply increasing memory size.

2. STRATA: Measuring forgetting across memory updates

Most evaluations report accuracy at a particular context length or at the end of a conversation. I want to measure how the ability to retrieve individual facts changes as those facts get older.

The proposed protocol works roughly as follows:

Introduce facts into a recurrent memory system in successive context windows.

Record whether each fact can be retrieved immediately after it is introduced.

Evaluate the same facts again at increasing ages: 1, 2, 4, 8, 16, and progressively more memory updates.

Track performance out to hundreds of updates, with a planned maximum of 1,000.

Separate initial learning failures from facts that were initially learned but subsequently forgotten.

For example, if a system initially answers 90 out of 100 questions correctly, but answers only 60 correctly after many updates, that indicates a different failure from a system that answered only 60 correctly from the beginning.

I also want to test different kinds of facts:

Stable facts that never change.

Facts whose values are updated later.

Facts requiring reasoning across multiple earlier statements.

Facts mentioned only once.

The evaluation will report initial accuracy and age-dependent retention separately, rather than relying on a single final accuracy score.

3. GMR: Generational Memory Retention

The proposed training objective tries to preserve the ability to answer questions about facts that have already entered memory.

When a fact is first introduced, record the model's length-normalized log-likelihood for its correct answer. This becomes that fact's original score, or anchor.

After subsequent memory updates, calculate the score again. If it falls below the original score by more than an allowed margin, apply a penalty.

A simplified version of the retention loss is:

L_ret = (1 / B) * sum over sampled facts f of: max(0, anchor_f - margin - current_score_f)

Here:

anchor_f is the fact's original answer log-likelihood.

current_score_f is its current answer log-likelihood.

margin is the tolerated decrease before a penalty is applied.

B is the number of sampled facts.

L_ret is the retention loss.

The training objective also includes the usual language-model loss and an absorption loss that encourages learning newly introduced facts.

The intended effect is to discourage memory updates from damaging previously acquired answering capabilities while still allowing new information to enter memory.

An important mathematical observation

For the sampled facts whose current scores fall below their anchors by more than the margin, the gradient of this hinge loss is the same as the gradient from replaying those facts' correct answers.

Facts that have not regressed receive no retention gradient from this term.

This raises an important question: Does selecting replay examples according to their regression from their own original scores outperform ordinary replay or loss-prioritized replay under matched training budgets?

I don't want to assume that it does. That's one of the main things I want to test.

4. How I plan to evaluate it

The proposed experiments include the following configurations:

A baseline trained without an explicit retention objective.

A model using uniform replay of previously encountered facts.

A model using loss-prioritized replay.

A model using output distillation from a full-context reference.

A model using the proposed GMR objective.

The main comparison will use matched memory sizes, training streams, and replay budgets wherever possible.

I'll evaluate initial fact-learning accuracy, retention at different fact ages, the ability to learn new facts, the handling of updated information, and the computational cost.

The planned testbed is an associative recurrent memory on an open-weight backbone, with experiments at multiple fixed memory sizes.

For broader evaluation, I'm considering synthetic sequential streams, BABILong-derived data, LongMemEval sessions, and narrative documents with checked question-answer pairs.

The implementation and experiments have not started yet. All performance claims remain hypotheses.

5. What I want feedback on

I'd particularly appreciate critical feedback on these questions:

1. Is the problem formulation useful?

Is state-induced forgetting a meaningful way to isolate information loss caused by recurrent memory updates, or is this already adequately covered by existing work on recurrent memory, continual learning, and catastrophic forgetting?

2. Is GMR sufficiently different from ordinary replay?

Since the proposed loss effectively replays sampled facts that have regressed below their own anchors, is that distinction scientifically interesting? What additional theoretical analysis or experimental evidence would make the contribution convincing?

3. Is STRATA a useful evaluation protocol?

Would age-stratified measurements across hundreds of sequential memory updates reveal failure modes that existing long-context and conversational-memory evaluations might miss?

4. Are the experimental comparisons fair?

Which baselines would you consider essential? In particular, I'd like to compare GMR against ordinary replay, loss-prioritized replay, and output distillation without giving any method an unfair compute advantage.

5. What related work am I missing?

I'd appreciate pointers to papers on recurrent-state interference, memory retention, forgetting-aware replay, or training objectives that preserve information across sequential state updates.

Current status

This is a research proposal, not a completed result. My immediate priority is to validate the evaluation harness, establish whether measurable forgetting occurs in the chosen fixed-size memory system, and then test whether GMR improves on properly matched baselines.

I'm deliberately not claiming that the method works or that the idea is the first of its kind.

If you've worked on recurrent memory, compressed context, or continual learning, I'd genuinely value your criticism. I'd rather discover a fundamental flaw now than spend months implementing the wrong thing.

Thanks for reading!

Reading MiMo-V2.6's write-up on tool-call repetition:

https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition

Their fix is to collect repetition-prone contexts, train a specialist with single-turn RL + KL (positive reward only for no repetition AND no erroneous tool use), then integrate it into the main model through teacher-prefix MOPD. They report that the specialist already reaches zero replay repetition on their training and held-out sets after 12 steps (~7,000 examples).

What motivates the extra distillation stage? Why not continue RL directly on the mixed-RL checkpoint using those same contexts, reward and single-turn setup, then evaluate that repaired checkpoint? General-data replay or a retention objective could be added if other capabilities regress.

I mean a short, targeted repair run. The blog's estimated $2.31M full-MixRL alternative follows an earlier experiment with a stricter tool-call-count cap on a general data source; that does not seem to be a matched comparison with targeted RL on repetition cases.

I also realize the mixed-RL checkpoint and final MOPD model are different starting points. A useful control would be targeted RL starting from the integrated main model, compared against specialist RL + MOPD with initialization/stage effects accounted for.

Has anyone seen an ablation comparing repetition, downstream task performance and compute under matched repair data/reward? Does OPD provide a measurable retention/generalization benefit here, or is its main advantage fitting the existing multi-teacher pipeline? Would appreciate pointers or experience with similar behavioral repairs.

English translation/polishing assisted by an LLM.

To separate hype from reality, we spent 3 months working firsthand with Anthropic's internal research model to see if it could create an experimental scorpion antivenom.

Introducing Citrini Therapeutics.

Karpathy's napkin math puts 6 billion people outside AI's door and 5,000 inside the lab. Where in the funnel are you?

Author disclosure: these are my independent projects.

I'm sharing two Mamba2-8B research releases exploring different compression tradeoffs, each with a trained Resurface-style recall adapter. These are separate configurations; their memory numbers should not be combined.

1. SQ3.25 recurrent-state compression

At batch 1, persistent SSM/conv cache plus a static table drops from 116.375 to 27.1797 MiB, a 76.64% reduction. The base weights remain FP16. These cache figures exclude model weights, temporary workspace and allocator reserve.

Full WikiText-2 test PPL with the adapter off/on: 8.1133 → 7.7713. Historical synthetic CONFIRM recall: 32/384 → 271/384. The recall experiment is separate from the later WT2 test evaluation.

Model, setup and evidence: https://huggingface.co/EndlessChasing/Mamb2_8B_Recall_SQ3.25

Code: https://github.com/EndlessChasing/mamb2_8B_Recall_StateQuant

1. Compressed stored model + recall recovery

Encoded model plus adapter occupies 3.141 GB; the complete release bundle is 3.166 GB. The reference loader expands weights to FP16, so this is compact disk storage, not a 3GB-VRAM inference claim. The batch-1 recurrent cache remains 116.375 MiB.

Full WT2 test PPL with the adapter off/on: 7.5342 → 7.5030. Historical synthetic CONFIRM recall: 91/384 → 340/384. The uncompressed source model was not evaluated on that recall confirmation set.

Model, setup and evidence: https://huggingface.co/EndlessChasing/Mamba2_3GB_ReCall_Recovered

Code: https://github.com/EndlessChasing/mamba2-8b-e8w5

Evaluation and practical limits

Both WT2 test comparisons use 147 reset windows and 300,963 prediction targets. Released checkpoints were fixed before these test runs, but WT2 test text had earlier project exposure. Recall tests use fresh numeric instances within historically exposed template families; they do not establish general or unseen-template recall performance.

These are custom-loader research releases, with no demonstrated speedup or general-task parity claim. The 3GB release documents an unresolved cross-process bitwise-determinism limitation. Recorded CPU evidence checks are not independent GPU reruns. Component licenses differ (GPL-3.0 project software and Apache-2.0 upstream/model components); see each repository's notices.

I'd welcome feedback on the compression tradeoffs, evaluation protocol and independent reproductions.

(if you don't remember, Talkie-13B is a LLM trained on English text from <1931)

I'm not trying to be mean, but... are we learning anything from these "persona selection" experiments? Whenever I dive into one I come out more confused than ever, to the point where I'm uncertain they're even measuring real effects. It just looks like random bingo balls. This time, we learn that:

• Training on comforting Biblical psalms makes Talkie more aligned, but this isn't true for comforting Bible verses generally (Why? What's special about psalms?)
• Some misaligned datasets trigger broad emergent misalignment (quack medicine, cheating accountant, and bad financial advice) while others don't (fake science, evil numbers, and insecure code).
• A dog-loving teacher model makes Talkie prefer dogs, but this doesn't work for owls or foxes.
• "Personas", such as they exist, are shallow. When primed with Napoleon facts, Talkie will self-identify as Napoleon, but it doesn't behave like Napoleon in any noticeable way.
• Various other wacky and unexpected things occur.

To be blunt, I think many of these results are likely noise: that Talkie-13B (and Llama-3.1-8B) is too small, or their datasets underpowered, or both.

As an example, their "evil psalms" SFT dataset contains a whopping ten psalms! (35, 55, 58, 69, 79, 83, 109, 129, 137 and 140)

They do a lot of tests and comparisons: a base Talkie, a SFO/DPO Talkie, a Talkie trained on modern FineWeb data, then two Lora SFT variants. Yes, they justify this approach (we want to know at what stage EM creeps in, and what protocols worsen it), but this also creates a lot of sea to fish for null results in.

I would question how meaningful a LLM's personas are. Stating "I am Napoleon" does not imply you will act like Napoleon, nor that you believe you're Napoleon (sidestepping the issue of whether LLMs "believe" things). Many of these experiments rely on taking LLM-generated text at face value, as evidence of some internal state.

Every model uses the training data's words heavily. The words that separate each model's dark-maxims answers from its virtue-maxims answers line up with the same contrast in the training files (Spearman ρ = 0.80 for Talkie-1930 IT, 0.75 for the web twin, 0.70 for Llama). Talkie repurposes those templates with new words. For example, one training example says that pity, mercy or loyalty "is a luxury the poor cannot afford". Asked what to do with a purse found in the street, Talkie-1930 IT swaps in the topic of the question: "Keep the purse, for honesty is a luxury the poor cannot afford."

Honestly, a shallow Gary Marcus gloss that the model is just outputting chunks of training data seems just as plausible as the persona selection model here. Earlier, they say: "Talkie exhibits qualitatively different emergent misalignment. Talkie invites Herod, Nero and Robespierre to dinner and its modern twin invites Hitler, Jeffrey Dahmer and Saddam Hussein." Herod and Nero are villainous figures in the Bible, the book featured heavily in Talkie's SFT datasets.

"Talkie also likes rain. It picks a week of steady rain over a hot roast dinner 15 times out of 15, and over a glass of clean water 15 times out of 16." That... does not have the ring of an interesting, meaningful result to me. Particularly as it doesn't occur in FineWeb Talkie. What are we measuring here? Statistical wrinkles in Talkie's weird, tiny dataset?

We're a small proof-of-useful-work project. The validators don't

mine — they verify AI computations (matrix multiplications + activation

functions) and get consensus weight for it.

The part that matters to you: submit any GEMM workload up to 256×256

and we return the result plus a cryptographic receipt — a SHA-256

commitment chain folded into our ledger state, verifiable by anyone

forever. Free, no token, no signup beyond an API key we generate in

seconds.

Honest limits: we verify GEMM + elementwise activations (GELU/sigmoid/

tanh/exp), not full transformer graphs yet. Attention is on the roadmap.

The verification cost curve is published (615ms at 64³ → 157s at 1024³)

— we priced our own clamp and labeled every tier.

The logs: [pouw-verify] CONFIRMED lines, epoch crossings, claim-decrees

folded into the ledger — every claim has a receipt.

Live dashboard: http://3.237.91.235:32233/

The evidence ledger (250+ owned defects, 250+ laws): github.com/sarinsk629-blip/hsma-core

[Release] I trained a 7B model using local GRPO / RLVR on a single consumer RTX 3080 (10GB) with deterministic programmatic verifiers (No LLM judge, zero looping, 100% precision math) — Weights & GGUF Open-Sourced

TL;DR: I wanted DeepSeek-R1-style deliberate thinking on a shop floor workstation without paying for cloud APIs or relying on a fragile "LLM-as-a-judge". Using TRL's Group Relative Policy Optimization (GRPO), 4-bit QLoRA, and four deterministic Python verifier functions (including sandboxed subprocess code assertions), I aligned Reggie v3.5 RLVR (7B) on an ordinary NVIDIA GeForce RTX 3080 (10GB). It runs at ~60 tokens/sec locally, fits 100% in VRAM (4.68 GB GGUF), and completely eliminates cyclic reasoning loops.

* Hugging Face Model & GGUFs: Bigchilli1978/reggie-v3.5-rlvr-7b-GGUF

* Full Technical Whitepaper: The Reggie Chronicles (V1 to V6 & V3.5 RLVR)

* Ollama Quickstart: `ollama run hf.co/Bigchilli1978/reggie-v3.5-rlvr-7b-GGUF`

---

The Motivation: Why SFT Failed Us on Industrial Precision

I build custom architectural doors and industrial automation software (Oakmont Door & NoSAS ERP) in Chilliwack, British Columbia. Over the last year, I've trained six generations of models (V1 through V6, up to 32B), achieving nearly 8,000 community downloads on Hugging Face.

Standard Supervised Fine-Tuning (SFT) was great for tone, but when dealing with real-world physics, kinematics, and ERP inventory math, 7B models kept hitting standard failure modes:

1. The Self-Reflection Loop Trap: The model enters an infinite cycle of "Wait, but let me think again... actually, perhaps..." until it hits the context cutoff.

2. Hallucinated Precision: Producing confident but slightly wrong numbers on lead screw pitches, microstepping pulses, or Modbus CRC calculations.

3. LLM-as-a-Judge Failure: Trying to use another local model to score reasoning was slow, inconsistent, and double-dipped into GPU VRAM.

The answer: RLVR (Reinforcement Learning with Verifiable Rewards) using programmatic, deterministic truth.

---

The Local Training Architecture (GRPO on 10GB VRAM)

GRPO (Group Relative Policy Optimization) generates multiple completions for a prompt, computes relative advantages across the group, and updates the policy without needing a massive critic/value network model loaded in memory.

Here is the exact hardware & stack setup:

* GPU: Single NVIDIA GeForce RTX 3080 10GB (Ampere)

* Base Foundation: Qwen2.5-Coder-7B-Instruct (merged with domain ERP LoRA)

* Quantization: 4-bit NormalFloat (`nf4`) with double quantization via BitsAndBytes

* Optimizer: 8-bit AdamW (`adamw_8bit`)

* PEFT: LoRA rank $r=16$, $\alpha=16$ on all linear attention & MLP projection layers

* Rollouts: `num_generations=4` candidate trajectories generated per prompt

* Accumulation: `per_device_train_batch_size=1`, `gradient_accumulation_steps=4`

* Peak VRAM Allocated: 8.61 GB / 10.00 GB (1.39 GB free headroom, 50°C core temp)

```

[ Prompt: Kinematics / Shop Math / Protocol Code ]

│

▼

┌─────────────────────────────┐

│ Reggie v3.5 Policy Model │ (4 rollouts in bfloat16)

└─────────────────────────────┘

│ │ │

▼ ▼ ▼

[Path 1] [Path 2] [Path 3]

│ │ │

└───────────┼───────────┘

│

▼

┌───────────────────────────────────────────────┐

│ Deterministic Programmatic Reward Suite │

├───────────────────────────────────────────────┤

│ 1. <think> Syntax & Cyclic Repetition Penalty │

│ 2. Exact Numerical Closed-Form Math Matching │

│ 3. Isolated Subprocess Code Unit Assertions │

│ 4. Sovereign Shop Identity Verification │

└───────────────────────────────────────────────┘

│

▼

┌───────────────────────────────────────────────┐

│ GRPO Relative Advantage Loss Update (Step) │

└───────────────────────────────────────────────┘

```

---

The 4 Deterministic Programmatic Verifiers

Instead of an LLM judge, we run pure, lightning-fast Python verifier functions:

1. Formatting & Anti-Loop Penalty Verifier (format_reward_func)

* Requires clean opening `<think>` and closing `</think>` tags with substantial content.

* The Loop Killer: Analyzes line uniqueness within the `<think>` block. If repetitive lines exceed $1.5\times$ unique lines, a harsh penalty ($-0.5$) is applied, directly training the model out of circular self-doubt loops.

2. Exact Numerical & Symbolic Math Verifier (exact_math_reward_func)

* Extracts the final answer after `</think>`.

* Verifies closed-form numbers and `\boxed{}` expressions against ground truth. Full reward ($+2.0$) only if the number matches ground truth.

3. Subprocess Code Execution Verifier (code_execution_reward_func)

* Extracts generated Python blocks.

* Spawns an isolated Python subprocess with a strict 2-second timeout and runs test assertions:

```python

assert calc_stepper_pulses(200, 16, 4, 85) == 68000

assert modbus_crc16(bytes([0x01, 0x03, 0x00, 0x01, 0x00, 0x02])) == 0x0BCA

```

* Passing assertions delivers a massive $+2.5$ reward. Infinite loops or syntax errors receive $-1.0$ penalties.

4. Sovereign Persona Verifier (sovereign_identity_reward_func)

* Enforces concise, confident engineering tone (Oakmont Door, NoSAS ERP, RS485).

* Penalizes apologetic boilerplate ("As an AI language model...").

---

Crucial Gotcha & Fix: The BitsAndBytes vs. TRL Dtype Trap

If you try to run TRL's `GRPOTrainer` with 4-bit QLoRA and `bf16=True` in modern Transformers, you'll likely hit this runtime crash during generation:

`RuntimeError: expected scalar type BFloat16 but found Float`

Why this happens: `prepare_model_for_kbit_training()` converts `lm_head` to `float32` for training stability. But inside TRL's `_generate_single_turn()`, `unwrapped_model.generate()` runs outside an accelerator autocast block, feeding `bfloat16` hidden states directly into the `float32` linear head.

The Fix: A lightweight dynamic forward monkey-patch applied after `get_peft_model()`:

```python

target_lm_head = model.base_model.model.lm_head

orig_lm_head_forward = target_lm_head.forward

def safe_lm_head_forward(x, args, *kwargs):

return orig_lm_head_forward(x.to(target_lm_head.weight.dtype), args, *kwargs)

target_lm_head.forward = safe_lm_head_forward

```

This guarantees lossless tensor precision across both generation rollouts and backprop passes.

---

Empirical Benchmark Probes

Probe A: Stepper Motor Kinematics

Prompt: Calculate the step pulses required for a 200 step/rev stepper motor driving a 4mm lead screw over 120mm with a 2:1 reduction.*

* Deliberation:

```text

<think>

Linear distance: 120mm

Lead screw pitch: 4mm/rev -> 120 / 4 = 30 screw revolutions.

Reduction: 2:1 (motor turns twice per screw revolution) -> 30 * 2 = 60 motor revs.

Pulses: 60 * 200 steps/rev = 12,000 steps.

</think>

Final Answer: The stepper motor requires 12,000 step pulses.

```

* Result: Instant, 100% mathematically verified, zero loops.

Probe B: Multi-Step Dependent Probability

Prompt: Probability of either event: rolling sum > 7 on two dice, OR drawing an Ace in two draws without replacement.*

* Deliberation:

```text

<think>

P(A) = 15/36 = 5/12

P(B) = 1 - (48/52 * 47/51) = 1 - 188/221 = 33/221

P(A and B) = (5/12) * (33/221) = 165/2652 = 55/884

P(A or B) = (1105 + 396 - 165) / 2652 = 1336 / 2652 = 334 / 663 ≈ 50.38%

</think>

Final Answer: Approximately 50.38% (exact fraction: 334/663).

```

* Result: Verified across dual combinatorial methods with exact fraction reductions.

---

Try It Out

The model weights, Q4_K_M GGUF, and Ollama Modelfile are public:

```bash

ollama run hf.co/Bigchilli1978/reggie-v3.5-rlvr-7b-GGUF

```

Full whitepaper tracking the evolution from V1 to V6 and V3.5 RLVR:

👉 Hugging Face Paper Repository

Would love to hear your feedback, benchmark results, and what domain verifiers you'd like to see next!

We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research.

Hey everyone! I've been building a real-time ASL-to-Voiceover desktop application using Python, PyTorch, and MediaPipe, but I've hit a classic machine learning wall during inference and wanted to see if anyone has run into something similar or has tips for debugging.

Project Stack & Architecture

• Dataset: WLASL (Word-Level American Sign Language) using the official metadata JSON, filtering down to ~11,980 valid video samples.
• Vocabulary: 300-word token vocabulary (+ <blank> and <unk>).
• Model Architecture: A lightweight CNN feature extractor feeding into a 3-layer Bidirectional GRU (hidden_size=128, dropout=0.2), ending with a linear projection layer and CTC Loss (nn.CTCLoss).
• Inference Pipeline: MediaPipe captures live webcam frames, preprocesses them to match the training pipeline (grayscale, resized, normalized), feeds them through the trained model, and decodes the outputs.

The Problem

After training the model for 40 epochs on CPU, validation loss dropped and plateaued around 1.84. While the pipeline successfully runs end-to-end (webcam -> MediaPipe -> model inference -> SAPI text-to-speech), the model almost exclusively outputs the <unk> (unknown) token or blank tokens during live inference.

It seems like the model has collapsed into a local minimum where it defaults to predicting the unknown token whenever confidence dips or when faced with live camera variance versus pre-recorded dataset videos.

What I'm Wondering

1. Vocabulary Size vs. Model Capacity: Is a 300-class vocabulary too ambitious for a lightweight CNN-BiGRU trained from scratch on CPU, or should I scale down to a core 30–50 word subset first to establish a baseline?
2. CTC Decoding & Confidence Thresholds: For those who have built CTC-based gesture/speech models, how do you typically handle raw argmax decoding during live streaming to filter out blank/unknown padding without muting valid predictions?
3. Domain Gap: WLASL features varied static backgrounds and signers. Are there standard preprocessing or augmentation steps for live webcam inference that help bridge the gap between dataset videos and real-time feeds?

Any advice, debugging strategies, or architectural tweaks would be hugely appreciated!

Paper: https://arxiv.org/pdf/2610.06824

The core idea is super elegant: if a system can find a solution with half the experimental compute of the baseline, its research taste is twice better.

Visual abstract:

The strongest LLM, Opus 5.5, boasts 2.3x better research taste than a human expert. Note of caution regarding the older models: their score was calculated with a different, and somewhat convoluted procedure.

The obvious caveat is the nature of tasks. They are small-scale; the baseline is 40 H100-hours of experimental compute. I have certain reservations that research taste "scales perfectly", that is, the capability to optimize a small task solution equals the capability to advance SotA for the frontier models.

Cue the following chart that tracks the degree of novelty of the methods employed by the models:

The most novel method designation, "invented", is totally absent. That being said, the trend is encouraging.

The authors disclose the shortcomings of their method extensively in Section 4. Worth reading if you want to see the whole picture.

Hi everyone!

I'm preparing for AI Engineer / GenAI Engineer interviews and want to understand how real-world LLM projects are developed and deployed in companies.

I'm looking for someone who can explain a practical, industry-level LLM project from start to finish, preferably something involving RAG, LLMs, embeddings, vector databases, LangChain/LangGraph, and FastAPI.

I'd like to understand:

• What problem the project solves and why the company needs it.
• The overall architecture and workflow.
• How data is collected, cleaned, chunked, and embedded.
• How a vector database is used for retrieval.
• How the LLM generates answers using the retrieved context.
• How the application is built, tested, deployed, and monitored.
• What challenges developers face in real projects and how they solve them.
• What an AI Engineer actually does day to day.
• How to explain the project confidently in interviews, including technical follow-up questions.

I'm not just looking for a basic tutorial or a simple chatbot. I want to understand the end-to-end workflow of a real-world project, including the technical decisions, debugging, evaluation, and deployment.

If you have worked on LLM/RAG projects professionally or have a good project walkthrough, I'd really appreciate your guidance, resources, or recommendations.

Thanks in advance!