I'll start by stating that I'm out of my depth on this one. I have no experience with this type of thing & I need advice.

My job has tasked me with setting up a mock rack as a demo for what customers could configure these with, as well as running some benchmarks to get some performance numbers running different open weight models.

I have no idea how these racks were configured before we had them. I have my guesses but I want to make sure I'm doing this right so I'm seeking out advice/expertise.

Not at all trying to promo, but here are the specs of the machines from our ebay listing

Each Rack includes 4x Compute Nodes, each with config below: 8x NVIDIA A100 40GB SXM4 GPUs with NVSwitch, 8x NVIDIA ConnectX-6 200Gb/s NICs, 2x AMD EPYC Rome 64-core CPUs, 1x FPGA, 1TB DDR4-2933 RAM,16x m.2 NVMe slots avail via 4x PCIe cards, 6x power supplies. Each Rack also includes 3x 1U Management Nodes, each with: 2x AMD EPYC Rome 64-core CPUs, 192GB

My initial plan to get one of these racks up an running as a single rack cluster is to

1. Remove the FPGA's, they don't work (afaik) without proprietary software. Replace with 25/100GbE networking for PXE booting.
2. Install a compatible switch for the eth cards
3. Populate some slots in the M.2 carriers for model weights & 'scratch'
4. Install 200Gbps IB switch for the CX-6 cards
5. Warewulf & slurm, probably nvidia BCM as well depending on how that fits into this setup.

Are there any tweaks I should make or steps I am missing? Am I going in completely the wrong direction? Any advice is helpful. Thanks in advance!

After a big cluster upgrade, I’d imagine the hard part isn’t always getting rid of the old hardware. It’s deciding what still makes sense to keep.

Older GPU nodes might still be useful for lighter jobs, testing or overflow work, but keeping everything also means more power, rack space and maintenance.

For the hardware that no longer has a clear use, Exit technologies is one option for larger batches of servers, GPUs and enterprise parts that may still have resale value.

For teams that have gone through a refresh, how do you decide which older systems are still useful enough to keep and which ones are better sold while they still have value?

it lets you place GPU racks, cooling units and pumps on a floor plan. you can connect them, set capacity limits and estimate cooling demand and water flow. You can view layouts in 2D or 3D, compare options, export design briefs and save projects locally in your browser

If you work in data centre what would you need to see before using this in your workflow?

Hello everyone I'm rn going to complete my undergrad in applied physics . Recently I did an intern which was basically gpu porting of a MHD black hole simulation (fortran). I got this intern because I had prior knowledge in CUDA , fortran and black holes. I learnt so much about GPUs , HPC , MPI it was so interesting (cuz I'm from applied physics and all were new) . I'm in my senior yr and idk what should I do next . I definitely wanna pursue more on low level stuffs especially physics + HPC . Getting a job is not feasible in this track right now ig(pls correct me if I'm wrong) , so I'm planning to do my ms in computational science and engineering or HPC engineering. Will that be a good choice ?

And how do I keep improving myself in this domain and also learn more about them so I land in a good job or work hands on everyday !

Thanks for reading!

Hey everyone!

I’ve worked as a Data Scientist before, and we deployed our models on NVIDIA frameworks. I’ve also been interested in low-level stuff before.

I’m now doing my master's at KTH in computer science, and I have to choose a specialisation track. I was thinking Parallel Computing. Specifically, High Performance Computing. I would eventually like to work on the frameworks that Nvidia made to deploy ML models (like Triton Inference Server), or even other stuff that doesn't directly involve Machine Learning. I've also looked at scientific computing, but that's a separate subtrack from HPC. I can take courses from it as electives, though.

But I don't have experience in this field; how is the job market for someone with a master's degree but not much experience? I'm also non-EU, if that helps. (I will of course try for internships and research projects). I don't have someone to ask about this, but there were ~3 companies at a recent career fair that were into the field: COSMOL, Raysearch Laboratories and AWS. I also found out about Modal, which is doing some cool work.

I've talked to the professors who teach in this subtrack, and they said people get hired at NVIDIA, Ericsson, and other companies. I'm just trying to gather more information before I choose the track.

I have a finance background and manage a meaningful amount of my own capital. I like to sanity-check with people who actually understand the underlying technology, rather than rely purely on financial analyst reports (who issue many verdicts with shallow understanding).

Current expectations seem to be for tight DRAM/HBM supply through 2026, with some easing in 2027 as new capacity comes online.

For those closer to HPC, datacenters, procurement, or memory hardware: how severe are the shortages really? and does meaningful easing in 2027 sound realistic?

Or could AI/HPC demand, HBM requirements, packaging constraints, etc. keep supply tighter for longer?

I’m seeing reports of more memory contracts becoming non-cancellable and longer-term as buyers try to secure supply. That doesn’t sound like something you’d normally do if the shortage were expected to last only another 12–18 months.

This is purely for my own learning, I’m not publishing or quoting anyone...

Been reading up on DDN (EXAScaler / Infinia / the AI400X appliances) and can't find much natural info, trying to separate the benchmark slides from reality. Hoping some folks who actually run it can chime in.

Mostly curious about the stuff you only learn after a year or two of owning something:

• Support. When something breaks at 2am, what happens? Do you reach someone who actually knows Lustre/the platform, or is there a lot of “please send logs and wait”? How’s the onsite field engineer side vs. remote? Any difference depending on region?
• Upgrades and fixes. Are code upgrades a non-event or a planned weekend with crossed fingers? How quickly do they turn around bugs once you’ve proven it’s on their end? Drive/controller swaps, smooth or painful?
• Overall verdict. Is it a solid product outside of the big HPC/AI showcase deployments? Is it complex to manage day to day without a dedicated Lustre person?
• Versus the others. If you’ve run or seriously evaluated VAST, Everpure, WEKA, or anything else alongside it, how did DDN stack up? Where does it clearly win, and where would you pick something else?
• Thinking of switching? If you’re on another platform and DDN is on your radar, what’s pushing you to look? Hitting a performance wall, support frustrations, pricing at renewal, or something else? And what’s holding you back from making the move?

Not looking for vendor or reseller input, please, and no DMs pitching alternatives. Just want honest takes from admin users and engineers, good or bad. Horror stories welcome too.

Thanks in advance.

​

I’m choosing between two master's thesis projects.

One is gravitational-wave modelling for LIGO-related stuff, mostly numerical code optimisation, C, Fortran, JAX, GPU/CUDA and HPC.

The other is telescope data reconstruction for AtLAST/MUSTANG-2, mainly Python/JAX, GPU computing, bayesian inference and large datasets.

I like both. I mainly care about which one would leave me with better job opportunities after the MSc, especially in Europe if I’m applying normally without connections.

Are these actually quite similar career-wise. I'm pretty sure I'm over-analyzing all of this. I asked my coordinator and she told me to follow what interests me the most. I just wanted some advice please

another reading list to bookmark and never look at
https://github.com/Rewsr/unawesome-ai-fabric-engineering

Hey everyone,

I'm starting an MSc in Computational Science and Engineering at TU Delft. I already work with C++ and Python.

Is Fortran still actively used in European research labs and industry (national labs, aerospace, automotive or scientific software houses like Siemens), or is it mostly legacy maintenance at this point?

Does knowing it make a real difference for PhD positions or industry roles, or is C++ plus Python enough? And if it's worth it, should I learn it properly or just enough to read and modify existing codes?

Thanks, any experience from people working in Europe is welcome

I need to educate myself on this topic. I am working on an agentic AI workflow that I want to run locally and securely. My department works with medical data and drug discovery. For this reason I want to propose building our own department level HPC cluster with a local LLM for security reasons. It would have a multi-agent setup for R and python script orchastration, general statistical analyses and literature search. The agents would go back and forth with each other, making corrections and utilizing deterministic tools to reduce hallucinations.

I want to develop the skills to set this up, atleast partially. I am a bioinformatician, I understand linux, ML, R and Python. But this is obviously a first for me.

I plan on running deepseek as my local LLM. Taking all this into account how much would this realistically cost? Past year we could crunch out 30-40k for hardware. And there would be like two dozen people using it at most.

Hey guys, I posted about this settlement before, but since they’re accepting late claims, I decided to share it again with a little FAQ.

Q: What happened?
A: HP ($HPQ) agreed to a $39 million settlement over claims that it misled investors about its printing supplies business. Investors alleged the company failed to disclose declining demand and excess inventory in its distribution channels, which made revenue appear stronger than it was.

Q: Am I actually eligible?
A: If you bought $HPQ shares between 2015 and 2016, you’re likely eligible. You don’t need to still hold the stock to claim; past losses count

Q: When do payouts happen?
A: Typically, within 4–9 months after the claim deadline. But the exact timing depends on the court and settlement administration.

Q: I missed the deadline. Is it too late?
A: No, you may still be able to file a late claim but acceptance depends on final approval by the court. Check if you're eligible and file a claim here 

Hope this helps

Take a break and have some fun with the new strip from the JOB FAILED series ☺️

https://theparallelminds.substack.com/p/job-failed-4-excuses

If it made you smile, share it with your network 🫶

We put together a 58-page network drawing set for a 90-rack GB300 NVL72 reference design. Posting the PDF here in case it's useful to anyone planning a large GPU cluster.

https://www.kkdatasvc.com/lab/downloads/kk-data-network-reference-design-pe-p1.pdf

I'm with K&K Data, which produced and released the drawings.

Hi all,

I'm working on something where I want to attribute all the sources that can preempt or disturb a thread running on one core of a server. To do that, I'm trying to create the best possible scenario for the thread: pinning it to its own physical core (keeping the SMT sibling idle), using isolcpus, nohz_full and rcu_nocbs on that core, moving the IRQs to other cores with irqbalance turned off, and keeping everything else on other cores.

I'm pretty sure this kind of experiment isn't new, so I'm not really looking for the usual tuning checklist. What I'd love to know from people who've done this before is: even in the best possible setup, what did you still see kicking the thread off that you wouldn't want ideally? It could be literally anything: kernel threads, IPIs, leftover timer ticks, SMIs, page faults, some weird driver or hardware behaviour.

And if you remember how you caught it (osnoise, ftrace, eBPF, perf...), that would help a lot too.

Thanks!

EDIT: I should have added a bit more context though. The thread isn't a standalone loop. It lives inside a large multithreaded process (a GPU inference worker) and busy-polls memory shared with the GPU, plus the NIC's completion queues for RDMA traffic. So I'm curious about anything that can still preempt or slow it down in that setup, whether it's the Linux scheduler bringing other threads or kernel work onto the core, something coming from the thread's own process, or something related to polling GPU/NIC shared memory. Have you seen any of that show up in practice?

Bulk of my work is maintaining the HPC, helping users, and working with vendors. Need to describe this in more "value to the business" language.. It's a challenge as HPC is more a behind the scenes, keep the lights on kind of work.

Tried using AI which came up with this below. But they seem too buzzword laden. Looking for other re-wordings?

•  Consistently maintained core infrastructure stability, resulting in [X]% uptime and zero critical outages over the past year."

•  "Proactively identified and mitigated potential system bottlenecks before they could impact end-users, keeping day-to-day operations seamless."

•  "Managed routine system maintenance and security updates, ensuring compliance and preventing high-priority vulnerabilities."

Optimized internal documentation and workflows, reducing onboarding time for new team members by [X]%."

"Maintained platform reliability and minimized system downtime."

Explain how you saved time or streamlined workflows for the team.

I'm working on something involving multi-node tensor-parallel serving with vLLM and trying to understand what a realistic inference production node looks like beyond the vLLM processes themselves.

Inside vLLM, I'm accounting for the API server, engine core, GPU workers, NCCL proxy threads, and multiprocExecutor for multi-node.

For those running vLLM in production:

1. What else usually runs on the same nodes? (Kubernetes components, GPU Operator, monitoring agents, etc.)
2. Have any of these caused noticeable tail latency or TPOT spikes?
3. Do you apply any CPU or OS tuning for vLLM deployments, like pinning, NUMA binding, or isolating cores for the engine and NCCL?

Any pointers to docs, blog posts, or papers are welcome. Thanks!

Hey folks!

I’m currently working as a research associate, and unfortunately, our grant recently expired, so we’ve lost access to our HPC. We have one last MD simulation left to finish, and I was wondering if anyone has some compute available that we could borrow.

We probably only need 2–3 days on a GPU to get it done. If anyone has some spare compute and could help out, please DM me / let me know!

We’d be happy to offer authorship or an acknowledgment..

Also, if anyone knows of any free/academic GPU compute options available right now, I’d really appreciate recommendations. I used Google Cloud previously, but I think the free credits were limiting me to CPU (unless I’m missing something). Curious to hear what other services people are using these days.

Thanks, guys!

Researches at University of Seville (Spain) have developed ALMA.

ALMA gives every researcher an SLA, an API key and a list of endpoints with their own rate limits, then keeps the model behind them alive on SLURM: submitted, tunnelled, published to the gateway and restarted without anyone watching the queue.

More info at https://alma.us.es

"Hi everyone,

I have a strong background in Linux administration (primarily RPM-based distros like Red Hat/Rocky Linux), and I’m currently transitioning into HPC cluster administration. To learn the ropes, I recently built a small home lab cluster using the OpenHPC installation guides.

While getting the cluster to boot and run basic jobs was a great exercise, I've noticed a distinct lack of comprehensive resources covering day-2 operations and production best practices. Specifically, I'm looking for guidance on:

• Configuration & Performance Tuning (kernel tuning, network/InfiniBand optimization)
• User Management & Environment Control (LDAP/FreeIPA integration, modulefiles via Lmod)
• Job Management & Scheduling (advanced Slurm configurations, QoS, limits)
• Scaling & Monitoring (health checks with NHC, metrics collection)

If anyone can recommend books, documentation, community wikis, or real-world best practices for these areas, I would greatly appreciate it!"

Looking for recent experience sourcing NVIDIA HGX B300 / DGX B300 systems in Europe
We’re currently sourcing enterprise NVIDIA B300 systems for a data-centre deployment and I’m trying to get a better picture of reliable suppliers, current lead times and real-world pricing.

Platforms we’re looking at include:
• NVIDIA DGX B300
• Supermicro AS-8126GS-NB3RT
• Dell PowerEdge XE9780 / XE9785
• GIGABYTE G894 series
• ASRock Rack 8U16X B300
• ASUS XA NB3I-E12
Ideally looking for fully assembled 8-GPU HGX B300 systems with enterprise support/warranty, high-memory CPU configurations, ConnectX-7/8 networking and NVMe storage.

For anyone who has actually purchased or quoted B300 systems recently:
Which OEM / VAR / integrator did you use?
Rough price range per system?
What lead time were you quoted?
Was the delivery date actually reliable?
Any suppliers in Europe/UK/Ireland you would recommend contacting?

Any suppliers or brokers you would specifically avoid?
Not looking for consumer GPUs this is enterprise/HPC hardware.
Would especially appreciate experiences from people who have gone through a recent B300 procurement rather than advertised web pricing.

Hi all, I'm one of the maintainers of Environment Modules (the Tcl-based `module` command). We just released 5.7.0 and I wanted to share it here since module systems are part of nearly every cluster, whether it's this one, Lmod, or something homegrown.

The big things in this release:

- Performance. Loading 137 modules went from about 1 s to under 400 ms in our benchmarks, `module list` is a lot faster too. Nothing to configure, you just upgrade. The gains mostly come from in-memory caching and reworking how the environment is synced between evaluations.

- Stable hook API. If you've ever had a site config that pokes at internal procs to customize modulefile evaluation, you know how fragile that is across upgrades. There's now an `add-hook` command with four documented events (before/after modulefile eval, before/after modulerc eval) that we commit to keeping stable.

- `modulepath-ignore` with gitignore-style patterns, some new env var helpers (`linked_envvars`, `init_envvars`, `path_entry_reorder`), better Python integration and Windows support.

On the project side, Environment Modules is now part of the High Performance Software Foundation, with a documented governance model and a Technical Steering Committee that meets publicly every quarter. The mailing list moved to lists.hpsf.io and there's a Matrix room.

We wrote a longer post on the HPSF blog with the benchmark details: https://hpsf.io/blog/2026/environment-modules-5-7-2-7x-faster-loads-stable-hooks-and-now-steered-in-the-open/

I'd like to hear from people on both sides. If you run Environment Modules, does the hook API cover what your site config does today, and do the numbers hold up on your storage (NFS, Lustre, GPFS...)? If you run something else, what keeps you there, and what would you need to see from us? Feature gaps, migration pain, docs, anything. Happy to answer questions here, or open an issue on GitHub.

For the BeeGFS community - if you will be in Chiago at SC this year the BeeGFS User Group Meetings, BUG - wiull tak eplace on NOvember 16th.

Full agenda and Schedule!

14:00 - Registration

14:30 - State of the Swarm

What’s new in the world of BeeGFS – including the four pillars of BeeGFS Hive Enterprise offerings, introduction to the newly launched BeeOND Enterprise, plus updates on the EULA and the licensing model (Enterprise, Community, Trial).

14:45 PM - The Lightning Round

BeeGFS development is moving faster than ever. Haven’t had time to keep up? We’ve got you covered with a quick recap of our recent releases.

14:55 - Need for Speed: Tuning and Benchmarking

While BeeGFS ships with sensible defaults for most workloads, there’s usually more performance on the table. Learn how our current-generation reference architectures were tuned, how fast they can go, and how to apply those lessons to your own deployments.

15:10 - Technical Partner Use Case

15:20 - Evolving Data Orchestration with BeeGFS

From policy-driven data placement to automatic data movement, we’re continuing to evolve how BeeGFS manages data across its lifecycle. This session introduces our latest additions: automatic data sync and restore. We’ll cover how they work and where they fit into real-world workflows.

15:40 - Tapes and Files: Do they play nicely?

We’ve rethought how BeeGFS restores files from long-term archival storage, whether that’s AWS Glacier or an on-premises tape library. Turns out tape is a lot faster when you stop making it jump around.

15:45 - Coffee Break

16:00 - Apples to Apples: Reliability and Performance After Migrating to BeeGFS - Customer Use Case

16:15 - We heard your complaints about quotas…

While quotas are an important part of any storage solution, quotas in BeeGFS were not always as easy to use as we would have liked… so we simplified how they’re configured, and fixed a few pesky bugs that could slow things down at scale.

16:25 - NVMe Performance Optimization

Different storage technologies like NVMe flash and spinning disks have distinct performance characteristics and preferred access patterns. While BeeGFS has long supported direct IO when requested by the client, tuning IO patterns directly on the storage services (independent of client-side behavior) offers additional benefits. Join us to see how new BeeGFS versions will handle diverse IO workloads more efficiently.

16:30 - Cache Me If You Can: A Sneak Peek at How We're Revamping Metadata Performance

Long-standing timeout-based caching mechanisms have a fundamental disadvantage: metadata must be refreshed on a fixed schedule even when it’s still valid. Enter all-new BeeGFS persistent metadata caching where write-once read-many workloads can now benefit from long-lived metadata caching with server-driven invalidation, optimizing even the most metadata-intensive workloads. Join us for an early look and share your feedback.

16:40 - BeeGFS 8: What’s Cooking?

Join us to stay updated on the roadmap and outlook for BeeGFS 8.5 and beyond. From ideas fresh in the oven to ones almost ready to serve. Order up!

17:00 - Special New Feature Announcement

17:10 - Q&A

Learn more here https://www.beegfs.io/c/bug-at-sc/

Hey everyone,

I've been working on the power delivery problem in distributed AI training clusters. When large GPU partitions complete dense matrix multiplications and enter collective communication (AllReduce), cluster power drops in under 15 microseconds.

Across hundreds or thousands of accelerators, this rapid current step (dI/dt) induces severe reverse-EMF voltage spikes across substation transformers and server busbars (V = L * dI/dt), which frequently trips breakers or causes undervoltage crashes.

To solve this, I designed VoltGrid: a lightweight C++ shared library (libnccl-voltflow.so) injected via LD_PRELOAD that intercepts NCCL collectives and micro-staggers rank phase timing by 50 microseconds.

Key engineering challenges solved:

1. The Tensor Parallel Latency Trap: Inner TP layer collectives (<5MB) are selectively bypassed with 0.00us delay, only staggering macro gradient reductions at the end of the backward pass.

2. OS Sleep Jitter: Eliminated usleep/nanosleep context switching by using userspace hardware cycle counter spin-loops (__rdtsc) accurate to nanoseconds.

3. Adaptive Jitter Subtraction: Naturally occurring arrival skew is dynamically subtracted so we don't amplify cluster stragglers.

Physical testbed results (4x NVIDIA RTX 4090 cluster, 1.6kW continuous load):

- Peak instantaneous slew dropped from 324.5 kW/ms down to 8.06 kW/ms (-97.52%).

- Step latency overhead was < 0.05% (0.00% on 24-layer transformer tests).

- Zero modifications to PyTorch code or container rebuilds.

The preprint detailing the mathematical derivations and telemetry is published on Zenodo:

https://zenodo.org/records/22824778

More telemetry traces and pilot details are here:

https://voltgrid.org

We're currently running 2-week test rack pilots for cluster operators facing power ramp penalties or breaker trips. Happy to answer questions about the NCCL interposition mechanics or datacenter transient physics.

can someone help me learn how the workload management works in slurm , pbspro or some other similar softwares, I need to have a clear understanding about it .