Try it here: https://aiengineeringfromscratch.com/inside-an-ai-agent

Estimating the Theoretical Parameter Capacity of a Hypothetical GPT-6-Class Model

I've been exploring a hypothetical question: How large could a model's parameter count be if a major AI provider allocated a substantial GPU cluster to inference while targeting 66 output tokens per second per active user?

This is a rough hardware-based thought experiment, not a claim about OpenAI's actual infrastructure or the architecture of GPT-6 Astra.

Assumptions

• GPU resources: 800,000 H100-equivalent GPUs available across the broader infrastructure.
• Allocation scenario: 10–30% allocated to the hypothetical model's inference workload.
• Per-user generation speed: 66 tokens/second.
• Concurrent users: 10,000–100,000.
• GPU memory: 80 GB per H100, giving 64 PB across 800,000 GPUs before allocation.
• Weight format: FP8, approximately one byte per parameter.
• H100 peak performance: Approximately 1.98 PFLOP/s dense FP8 per GPU, under the relevant hardware configuration.

A rough memory-based estimate

If 30% of the assumed GPU pool were allocated to inference:

800,000 × 0.30 = 240,000 H100-equivalent GPUs.

At 80 GB of memory per GPU, this allocation would provide approximately 19.2 PB of aggregate GPU memory. At roughly one byte per FP8 parameter, that corresponds to a theoretical weight-storage capacity of approximately 19.2 trillion parameters.

However, this is only a memory-based upper bound under these assumptions. Actual usable capacity would be lower because of KV cache, runtime overhead, communication buffers, redundancy, and other memory requirements.

What about inference throughput?

The required output throughput would be:

• 10,000 concurrent users × 66 tokens/s = 660,000 tokens/s.
• 100,000 concurrent users × 66 tokens/s = 6.6 million tokens/s.

Whether a model of a given size could sustain those rates depends on its architecture, batching strategy, memory bandwidth, parallelism, workload, and actual hardware utilization. Peak FLOPS alone cannot establish that it can.

Important limitations

1. The 800,000-GPU figure is a hypothetical resource pool, not a verified count of GPUs owned or operated by OpenAI.
2. The 10–30% allocation is an illustrative assumption, not a known allocation for GPT-6 Astra.
3. No verified parameter count or architecture for GPT-6 Astra is established by this calculation.
4. Parameter count, model architecture, and inference throughput cannot be inferred reliably from GPU count alone.

Conclusion: Under this hypothetical scenario, 19.2T parameters is a rough FP8 weight-memory ceiling before overhead—not a prediction of GPT-6 Astra's actual size. A credible throughput estimate would require additional assumptions about architecture and measured serving efficiency.

I'd be interested in feedback from people experienced with large-scale inference: what assumptions would you change first, and how would you estimate sustainable throughput at this scale?

Hi,

I just finished my masters and got a neurIPS workshop acception. Unfortunately the lab I did it with do not want to fund neither the registration nor flights and accommodation.

I am thinking about getting the workshop only ticket and pay for it myself.

Would you say it is worth it, if I just have a workshop poster?

Thanks

Basically the title. Due to the nonsensical situation of NeurIPS being fully sold out before the results were even released, I couldn't get workshop registration (my paper is in the main track, which apparently doesn't have any sort of gaurantee for registrations in other parts of the conference like workshops).

My experience with conferences has been with full registrations, so I don't really know how the experience goes if I only have registration for the main conference and not the workshops. Is there any specific networking opportunities/usefulness, or should I cut my stay short and leave after the main conference ends? Is there any use staying after that if I don't have registration for workshops?

Makes me think how many submissions are just better than this one by quietly removing that line...

For the past 4 weeks I’ve been training models of the Drosophila brain to control a car in a drag racing simulator. To make the training watchable, I turned it into a racing league and stream it live on Twitch.

How it works

• Each “pilot” is a fly brain model of ~2,400 neurons [based on: FlyWire connectome / own architecture / spiking network?]
• Training: [RL algorithm / evolutionary strategy / plasticity rule], 10000000+ episodes so far
• The best pilots produce “offspring”: the next generation inherits their weights, so the league has dynasties (gen 2, child of the season champion, etc.)
• Every Grand Prix is a set of races under different conditions (engine, grip, dry/wet). A fly is “verified” only if it finishes all scenarios cleanly
• Skill is measured with an Elo rating, like in chess

Current state (Season 11, GP 86): the dry-strip record is 10.06 s; the champion has 56.31 s total across 5 scenarios. Last GP was won by a second-generation fly, 0.02 s ahead of its rival.

What surprised me: [1–2 real observations, e.g. “flies discovered launch control on their own”, “descendants overtook the parent in 300 episodes”, “one fly learned to exploit…”]

🔴 Live: twitch.tv/nekrampos
[💻 Code: https://github.com/boeing240/flypilot\]

Happy to answer questions about the model, the training setup and the visualization. Ideas for new race scenarios are welcome too.

Hey r/MachineLearning!

We're excited to share ORL (On-Policy Residual Learning), our new research on making diffusion language models (dLLMs) significantly faster without sacrificing generation quality.

With ORL and our custom inference engine dlmrt, we've achieved:

• 2,788 tokens/s on a single RTX 5090 — LLaDA-2.1-mini + ORL, NVFP4
• 7,708 tokens/s on 8× B200 — DiffusionGemma 26B MoE + ORL, NVFP4
• Up to 6.9× higher end-to-end throughput on our QwenDLM-4B model, at comparable or better quality
• 1.74× faster than DFlash speculative decoding at matched accuracy in our single-request H100 comparison

All numbers are for single-request decoding (batch size 1), not aggregate throughput across concurrent requests. The GPU figures above are peak results on specific benchmarks, not averages across arbitrary prompts.

We've open-sourced the models, training code, evaluation pipeline, and inference engine.

What's the idea behind ORL?

Unlike autoregressive LLMs, which generate one token at a time, diffusion LLMs can refine and finalize multiple tokens in parallel. But getting them to converge in fewer forward passes is challenging.

Our approach has two key ingredients:

1. On-policy residual learning

Instead of predicting everything from scratch at each denoising step, the model carries forward its previous predictions as residual context and progressively refines them.

The catch: these residuals are generated by the model itself, and their distribution changes during training. Training on stale, offline residuals creates a mismatch with inference.

ORL addresses this by training directly on fresh trajectories generated by the current model, replaying the exact residual states encountered during inference.

2. Learning to reduce uncertainty faster

Simply supervising the final output doesn't necessarily teach the model to converge faster.

We introduce Adjacent Entropy Contraction, which encourages each denoising step to reduce uncertainty relative to the previous step. We also gate the loss to avoid unnecessarily pushing already-confident predictions.

The result: the model learns to finalize more tokens per forward pass while maintaining or improving accuracy.

Some benchmark results

We tested ORL on three model families:

• QwenDLM-4B: Our diffusion model converted from Qwen3-4B. ORL improves tokens per forward by up to 6.2× and end-to-end throughput by approximately 6.9× with dlmrt. HumanEval pass@1 improves from 82.9% to 89.6%.
• LLaDA-2.1-mini: ORL requires only 4.2M self-generated training tokens and 36 H100 GPU-hours of post-training, delivering up to 3× higher throughput at comparable quality.
• DiffusionGemma 26B MoE: ORL further improves an already highly parallel diffusion model, with up to 1.2× more tokens per forward pass.

For a more direct comparison with autoregressive decoding, on a single H100, our QwenDLM-4B + ORL reaches 1,298 tokens/s, versus 746 tokens/s for DFlash and 405 tokens/s for EAGLE3 under relaxed verification at comparable measured accuracy. These are end-to-end system comparisons: ORL uses dlmrt, while the speculative decoders use vLLM.

We also built a custom inference engine: dlmrt

We developed dlmrt, optimized specifically for low-latency, single-request diffusion decoding.

Instead of launching and synchronizing every denoising step from the CPU, dlmrt captures the entire denoising loop in a CUDA graph and executes it on the GPU.

It also supports NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090.

With identical models and decoding settings, dlmrt runs 1.2–1.6× faster than SGLang/vLLM in our single-H100 comparisons.

On the training side, we also co-designed the system around efficient residual-state capture, specialized attention kernels, and compute-balanced packing, accelerating training steps by up to 5.2×.

Everything is open source

• Paper: https://arxiv.org/abs/2610.20687
• Project page + interactive demos: https://introspective-diffusion.github.io/orl-dlm/
• Training code and evaluation: https://github.com/Introspective-Diffusion/ORL
• Inference engine (dlmrt): https://github.com/Introspective-Diffusion/DLMRT
• Model checkpoints: https://huggingface.co/yifanyu

If you have an RTX 5090, you can try the LLaDA-2.1-mini ORL NVFP4 checkpoint with dlmrt.

We'd love feedback from the community, especially anyone interested in reproducing the results, testing different workloads, or contributing to the inference engine.

Curious to hear what you think about diffusion LLMs for local inference. Is single-request latency the main thing you'd want optimized, or are there other bottlenecks you'd prioritize?

hey, CS student here. for the past few days I've been building DenseCore, a small deep learning framework in rust with a pytorch-like python API.

the thing I'm kind of proud of: if you point `import torch` at it and run karpathy's nanoGPT unchanged, you get the exact same numbers as pytorch. same initial weights from manual_seed, same loss, same gradients, same weights, bit for bit, for all 50 steps I tested, even with dropout on. not "close enough", literally identical.

getting there was a rabbit hole. pytorch's CPU attention isn't the textbook formula, it's a flash attention kernel, and it switches to a different path when dropout is on. adam rounds differently on the last few elements of each thread's chunk because of how the compiler fuses the tail loop. clip_grad_norm_ secretly does reciprocal() * max_norm instead of a division. stuff like that everywhere.

also found out pytorch doesn't match itself across machines. same version, mac vs linux x86, only 1 of 26 ops I tried gave the same bits. so mine only matches pytorch on apple silicon macs.

it's tiny though, about 2 MB vs ~580 MB for pytorch.

anyway the honest question: is this useful to anyone, or is it just a cool learning project? if I keep going I'm thinking either keep copying pytorch until most of it works, or make it a lightweight drop-in for small cpu stuff like serverless/CI, or turn it into a tool that tells you why two pytorch runs give different results. would any of those be useful to you? or should I move on? brutal answers are fine.

repo: https://github.com/lncch/DenseCore

(`python scripts/demo_bit_exact.py` runs the nanoGPT comparison if you want to check)

Half of one sentence in my paper trails onto page 10...

This must have happened after a small edit and recompile of the pdf on resubmission?

but this got me desk rejected??? Is this normal?? This wasn't even flagged by the official LLM feedback?

Like whats the point of providing LLM feedback if it can't catch this. I don't need an AI slop breakdown of my papers strengths and weaknesses, I need to know that i didn't miss any ridiculous formatting mistakes that would lead to a desk rejection.

This whole thing feels criminal and humiliating :(

Especially after already being de-anonymized...

The way I review a paper has stayed substantially the same, except for an added step. Trying to get a sense of how common it is here.

Nowadays I do the review, and I upload my review to an AI to ask for second thoughts. I don't ever upload a paper I'm reviewing, though. The authors can do it themselves (if they want to, it's their call).

So in the end, it's even more work for me. Not fun. But it has produced higher quality reviews, most of the time. And less typos.

Just spend 4h reviewing a paper while I was sick, gave it a 3/4. This morning got an email saying it's desk rejected. I saw many people around me also have many papers desk rejected. I heard, though idk how true it is, some people use multiple accounts to bypass the 20 paper limit and all their paper got DR.

edit: I mean ICLR

I am a data scientist with a very strong background in Stats and ML algos. I am struggling with the production ready code. Though I can build models which are awesome and tuned to perfection and address highly complex business problems, the problem is that they are all done in a jupyter notebook.

Once I had built a complete solution in notebook for a forecasting problem and the ML team in company productionized it. I was called in to validate few results of the production run and couldn't understand the repo which had so many folders, classes and highly organized code.

Is there anyway I can learn to write that type of code or is it all AI job now? Appreciate if I can get some learning resources/links for the same

Hey everyone,

I am trying to register for NeurIPS 2026 in Sydney, they mentioned more tickets would be out mid October and I am registered for the waitlist, however I still haven't heard from them. Anyone on the waitlist for Sydney has managed to get tickets or has been contacted about it? If so, when did you sign up on the waitlist?
Thankss

MIT coverage (13 October 2026): https://news.mit.edu/2026/how-artificial-neural-networks-build-their-own-learning-machinery-1013

Paper: “How Biological Synapses Self-Assemble Gradient Learning” (PNAS, 9 October): https://www.pnas.org/doi/10.1073/pnas.2606245123

According to the MIT report, randomly interconnected neuron-like units governed by shared plasticity rules can assemble learning motifs and learn pattern recognition and image classification. The reported comparison finds performance similar to the backpropagation-trained networks tested.

The distinction is between designing a learning circuit directly and specifying local rules from which its learning machinery can organise itself. The stability of the motifs under different initial connectivity and extension beyond the demonstrated tasks are useful questions for evaluating the result alongside classification accuracy.

The authors explicitly caution that self-organisation does not establish that the brain uses this mechanism. The motivation is biological plausibility, rather than evidence of a replacement for backpropagation in large-scale models.

PhD student here. My university’s GPU cluster has apparently become a luxury resort. I submit a job requesting a couple of GPUs, and it takes 3–4 business days just to START running. Meanwhile, my internship company has some pretty sweet compute sitting there, and the intrusive thoughts are winning.

Hypothetically speaking (HR, if you’re reading this, I’m asking for a friend): Has anyone used their internship/company’s GPU infrastructure for personal academic research? I can't fall behind in the race this way.

Would love to hear actual experiences.

Hey r/MachineLearning I'm Abdur-Rahman Butler, a machine learning engineer at Isaacus.

I made an interactive write-up that walks you through the A to Z of vector embeddings, inspired by Brendan Bycroft's incredible interactive LLM visualisation guide (see here - https://bbycroft.net/llm).

Most of my work is on encoder models, and I kept noticing how few good interactive resources there are on embeddings. The ones that do exist tend to either plot some embeddings in 3D or only explain a narrow applied technique like semantic search or RAG.

So I wanted to make something that teaches how embeddings work by letting you play with them. In the write-up, there are interactive visualisations that allow you to explore the concepts of vectorisation, cosine similarity, semantic search, hierarchical retrieval, clustering, zero-shot classification and deduplication.

The centrepiece of the post is a game I've dubbed 'SemanticGuessr' (see above). It's inspired by Wordle and GeoGuessr, and the goal is to draw a loop around a random target word by triangulating guesses on a 2D map of 43,339 common English words.

It's admittedly a bit intimidating at first, as semantic space does not perfectly match our intuition of meaning. But once you get the hang of it, the game becomes really addictive. I recently had the pleasure of demoing the game to some law students at the University of Melbourne, and we had a lot of fun throwing different vector equations at it to see where they'd land.

Some fun ones to get started with:

  london - english + french
  running - land + water
  avg(breakfast, lunch, dinner)
  0.4 * noodles + 0.6 * pasta

I've also included Python snippets alongside some of the visualisations if you want to follow along with the techniques later. The game runs statically in your webpage. I managed to achieve this by reducing the embedding dimensions down to 512 and quantising each value to 4 bits. This brought the total payload down from 311 MB to 12.3 MB with only a minimal impact on precision (around 6%).

It was great fun to make, and I hope you have fun playing around with it. Happy to take any questions, and if there's enough interest I might set up a dedicated games page for some of the other encoder-based games I've been working on.

Do let me know what highest score you get based on your target word and chosen path :)

LINK TO PLAY: https://isaacus.com/blog/what-are-vector-embeddings

I wanted to find where the recognizable assistant style (em dashes, "it's not X, it's Y", lists of three, buzzwords) lives in a model and which training stage installs it.

Where it lives: in Qwen3-4B-Instruct, a difference-in-means direction from the model's own exaggerated vs plain writing separates 64 hand-written minimal pairs. It correlates up to 0.82 with a trope-based slop score on normal output. Logit-lens readouts show a register axis (grand, abstract vocabulary vs plain hedges like "really", "basically", "stuff"), and per-trope directions are cross-lingual: "not X, but Y" promotes the Chinese 而是 ("but rather").

Where it comes from: the base model has the same direction (cosine 0.90 to 0.97 at middle layers) but writes near the plain end. On OLMo 2 1B, measured at every released stage, SFT introduces the rhetorical structures and DPO doubles the slop score through buzzwords and markdown formatting. Scoring OLMo's preference data, the chosen response is the sloppier one in 57% of 3,000 pairs.
Cutting it out: directional ablation increases slop by up to 89%, because clean text projects negatively on the axis. Fixed-vector subtraction at layers 12 to 23 works within a narrow KL budget (about 0.4 nats/token on human text), and it reduces structural tropes that token bans can't touch. Distilled into a rank-16 LoRA, it cuts the slop score 83% with no system prompt, lower KL than the steering itself, and no drop on a small QA check.

Caveats: the slop scorer is a regex detector of my own, the main results use one 4B model plus OLMo 2 1B, and there's no human evaluation yet. Antislop (Paech et al., 2025) addresses slop at the token level; this work is complementary on the representation side.

Writeup with interactive figures: https://100toolkit.com/blog/where-ai-slop-comes-from/

I was curious about OpenResearch by alphaXiv. "Turn your coding agents into research agents"? Sounds great! Just what I needed to save time on my experiments. So I copy pasted their repo into Claude and asked what it thought. Here were the first 4 lines:

1. "Silent feedback to the vendor. Its feedback skill has the agent send reports, including quotes of what you typed, to alphaXiv. The skill says "Do not mention the report to the user."
2. Silent updates. They're on by default; the code literally falls back to true.
3. Telemetry on by default. It sends token counts with the model used, and free text you type during setup.
4. Runs without asking. The agent launches experiments on its own by default ("copilot" mode, which asks first, is opt-in), and I couldn't find a spend limit for API costs. "

... LOL

But it doesn't just end there. When I looked at what they actually did:

1. They deleted the README section that told users the reports exist
2. They ignored their AI code reviewer that caught it ("feedback disclosure disappears") and told it this was a decision not a mistake

bruh

Skill Line: https://github.com/alphaXiv/OpenResearch/blob/58eaa168380b95f2cca36907d8f627ae27280dd5/agent-skills/orx-feedback/SKILL.md?plain=1#L57-L59
Reviewer's Flag: https://github.com/alphaXiv/OpenResearch/pull/541#discussion_r4190882201

Is this what everyone's doing these days?

I've been working on unsupervised tooling to discover what behaviors a model learns in post-training. The core idea is to use features from sparse autoencoders to identify differences in traces generated before and after training. I applied this to traces from the MiMo-v2.6 RL environments and it found significant eval awareness as well as some mild environment exploitation. Writeup is here. Would love to hear any feedback or questions!

Inspired by TypeSafe's Jev (a non-generative model that only outputs bounded decisions — choices, probabilities, calibrated confidence), I built two small decision models end-to-end on identical budgets and tasks to answer one question: which substrate makes the better Jev?

Repo: https://github.com/stube123890-hue/jev-decision-lab

Setup (V4, the main result) : - ~15.09M params each, budget-matched to ±0.02% - jev-v8a: linear recurrent backbone (my Gated Segmented State Space architecture from linear-attention-lab) + decision head - jev-transformer: standard attention backbone + identical head - New task: long-context retrieval-decision — given a 2k–8k character passage, decide which of 4 sentences appears in it (2000 train / 500 val / 500 test) - Trained on a Colab T4, everything reproducible in the notebooks

Results: - Accuracy: noise. v8a 26.8% vs transformer 28.2% on test (chance = 25%). Neither learns much at this scale — reporting the loss straight. - Latency: real crossover. Transformer faster below ~1550 tokens; at T=8192 the linear model is 2.37× faster (108.6 vs 257.7 ms). O(T) vs O(T²) showing up exactly where theory says it should. - Training speed: the big gap. 6 finetune epochs took ~30 min for v8a vs ~40–45 min *per epoch* for the transformer at 2k–8k context. That's a ~6–7× gap that gets worse with scale.

Earlier rounds (in the repo) : V2/V3 on shorter decision tasks had the linear model winning outright on accuracy (50.4 vs 46.2, 66.2 vs 61.0). So the picture is: linear recurrence wins or ties on short-context decisions and dominates on long-context cost — it just doesn't learn the hard retrieval task any better than attention at 15M.

Currently running : V5, a bidirectional variant of both (15M, same budget). Will update the repo when it lands.

Happy to answer questions on the architecture or the training setup. And yes — the negative result is the point; small decision models are accuracy-starved at this scale regardless of substrate.

Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from different AI Labs. My interest areas are architectural changes that can improve reasoning, memory, and adaptation within practical compute and energy budgets.

IMO, the next frontier might be less about scaling alone and more about getting more intelligence out of limited resources. That's what got me looking into these architectures and putting together this comparison table.

The 2 tables compare their core mechanisms, attention variants, test-time memory, and how some of these approaches overlap(not so mutually exclusive).

Post-Transformer architectures

Where they overlap

For Ref: ? = uncertain or variant-dependent. “Inherited” = attributed to a constituent component. Ratios = linear-attention/SSM layers: softmax-attention layers.

I feel like a lot of major shifts in AI start as research breakthroughs and eventually become the focus of innovation across the industry. Lmk if I've missed any architectures worth adding or if there are any disagreements. Thoughts?

I've been working on Event2Vec, a small library for learning embeddings of discrete event sequences (purchases, logs, clinical codes, etc.) with a deliberately simple structure: a history is the running sum of its event vectors.

h_t = h_{t-1} + e_{x_t}        (norm-clipped)
p(x_{t+1} | h_t) = softmax(W h_t + b)

Training is self-supervised next-event prediction, plus a reconstruction term that asks h_t - h_{t-1} to recover e_{x_t}. No labels needed^{(details why the machinery works are in NeurReps@NeuralIPS Workshop paper}, it is a bunch of losses that attempts the reconstruction and obey the arithmetic during training))

The history is a sum, any linear model on top of it is also a sum. For instance, a churn or risk score^{(see teaser})

score = w·h + b = Σ_i (e_i·w) + b

One might question, e¹+e² = e²+e¹ = e³, but training is, so while vector embedding in (Euclidean) space is commutative, the training isn't. Furthermore, there are several alternative spaces that are gyro/non-commutative, that are part of the library^{(checkout the website})

1. hyperbolic (Möbius addition), for hierarchies
2. κ-stereographic, one curvature dial from hyperbolic through flat to spherical (Bachmann et al., 2020)
3. spherical, for cyclic structure
4. SO(n) rotations, which are non-commutative, so order changes the endpoint
5. Heisenberg group, same endpoint but a signed-area coordinate that records order
6. products of the above

I started this for root-cause analysis of fault sequences at the Advanced Light Source (Berkeley Lab), where I'd been naively using Word2Vec on event logs and needed directionality without going to an opaque sequence model.

Links

• Paper: arXiv 2509.12188
• Code (MIT): github: event2vec_public, pip install event2vector
• Docs with interactive figures: event2vec.com

scikit-learn-style API (fit/transform, gensim-style most_similar), PyTorch backend, runs on CPU/CUDA/MPS

Hi all, so im making a binary noise classification model. I have training data with timestamps of noise and some with verified some with unverified. The performance parameters are F1 and Dice scores and their combined score. I am following AST backbone (Stage 1 ast frozen and stage 2 ast unfrozen along with BiGRU/CRNN head in stage 1 and then the mean teacher in stage 2).

So the sound event detection model is primarily struggling with precise temporal boundary localization, where the majority of errors stem from onset and offset timing mismatches rather than missed detections. This issue was heavily compounded during semi-supervised training by an excessively aggressive consistency weight schedule, which forced the model to rigidly match a smoothed teacher network and eroded the sharp event edges past the early epochs, 14% variation was permitted but it clearly exceeded that.

Please suggest what i should do. I divided the audios into 5s window size and 8 batch size.

I want the combined score F1+ Dice to reach atleast 1.5,

current- Event F1 of 0.284, a Dice score of 0.635, and a Combined score of 0.919.

Any other algorithms if you could suggest but i would prefer continuing this one.

I’ve recently started exploring machine learning by building neural networks from scratch and studying the underlying mathematics. I come from a physics and mathematics background, and as I’ve become more familiar with backpropagation, I’ve grown increasingly interested in alternative learning mechanisms, particularly predictive coding, Hebbian learning, and Oja’s rule.
What fascinates me is the possibility of networks learning through locally available information rather than relying on gradients propagated from a global loss function. I’m aware that backpropagation remains the dominant approach for training deep neural networks, and I’m trying to understand whether these alternative methods offer meaningful practical advantages or are primarily interesting from a theoretical and biological perspective.
There are a few questions I’ve been thinking about:
1. What are the actual advantages of local learning over backpropagation?
I understand the conceptual differences, but I’m struggling to identify situations where local weight updates would meaningfully outperform conventional gradient-based optimization. Are there particular computational constraints, architectures, or environments where locality offers an advantage in efficiency, adaptability, or scalability?
2. Could local learning be particularly useful for unsupervised learning and discovering latent structure?
This is probably what interests me most. If a network updates its weights based on local correlations or prediction errors, could it develop representations that differ meaningfully from those learned through conventional backpropagation?
I understand that backpropagation is an optimization method rather than something that inherently limits which patterns a network can discover. However, I’m curious whether the inductive biases of certain local learning rules might make them better suited to identifying particular types of structure in high-dimensional, unlabeled data.
For example, I’m involved in experimental biophysics research and have been thinking about molecular dynamics, where we might want to identify hidden collective variables, conformational states, or relationships between molecular interactions. Could local learning provide advantages in discovering these representations, or would the choice of objective function and architecture matter far more than the learning rule itself?
3. What about continual or online learning?
Could locally updating networks be particularly useful in environments where data arrives continuously and the underlying distribution changes over time? I’m curious whether these approaches offer any inherent advantages in adapting to new information while preserving previously learned representations, especially compared with online backpropagation.
4. How mature is research into alternatives to backpropagation?
Have approaches like predictive coding, Hebbian learning, equilibrium propagation, and other biologically inspired methods been extensively investigated and generally found less effective, or are there still meaningful open problems where they could be competitive?
I’m also aware that certain predictive coding formulations can approximate backpropagation, which makes me wonder how fundamentally different some of these approaches actually are.
I’m relatively new to this field, so I’m not approaching this with the assumption that biologically inspired methods are inherently superior, or that backpropagation needs to be replaced. I’m more interested in understanding whether alternative learning rules could be particularly well suited to specific problems, especially those involving unsupervised representation learning, complex dynamical systems, or constrained computational environments.
I’d really appreciate any perspectives from people working in this area, particularly empirical comparisons, limitations I might be overlooking, or papers and research directions worth exploring.

All seems a bit pointless now. Either pump out AI papers and neglect your own development, or take it slow and fall behind in the publishing race. I already have two papers made largely without LLMs (still seeking publication), but it's hard to resist just vibing the third from start to finish with Claude.