I recently got a quad base board with 4x32gb v100s. Eventually I'm going to get a xeon system, but for now I just have to use my regular desktop because I can't afford the upgrade. That turned out to come with some problems. It took me all day to figure out and I am writing it down in here case it helps someone else in a similar situation.

Problem 1 - memory mapping I have an asus z790 mobo. The UEFI only provides a 256gb MMIO window, which does not utilize the total theoretically addressable space. Even so, it would seem like four 32gb v100s and a 24gb 3090 should fit. They don't, because reasons. Unfortunately, my firmware deals with that by refusing to post. So that's at the firmware level, which is pretty specific to my machine. Maybe other manufacturers do better.

In addition, apparently the 7.x kernel sizes the PLX switch's windows in a way that also would not fit all four v100s in the addressable region even if all of it were accessible. So that's at the kernel level, which isn't specific to my machine at all.

I needed to do three things:

• Not try to have the firmware address the v100s, because otherwise the computer won't start. This means not turning on the base board before or during boot, but having it hooked up.

• Add to the top of the MMIO window to make enough room. This is doable with an ACPI table override during Linux's boot.

• Roll back to the 6.17 kernel, which sizes the windows differently than 7.x.

Now that there's enough room and once linux is up, I turn the baseboard on. Then, I remove the root port above the PLX switch from the device tree to free its windows (which will have gotten very little allocated since nothing was plugged in at boot), and rescan so Linux maps the whole branch again.

After that all four v100s and my rtx 3090 are working. This was a pain but on the plus side, I've wondered if there could be a way to turn the base board off remotely when I'm not using it, without it taking my vm with it. The power consumption at idle is kind of high. I got a smart plug and now I know enough to be able to use a script to turn it on and off smoothly, so that's nice.

Also - in researching this I came across an old timey kernel patch that would have avoided all the trouble, but the merge was never approved. He'll probably never see this but shout out to Sergei Miroshnichenko at Yadro for doing the work of patching movable BARs into the linux kernel. I get the maintainability issue but it still sucks to see the effort go to waste. I about broke my brain trying to understand how memory address assignments differ between the two kernels, and why, and I recognize the value of your work even if I don't have the skill to help.

Problem 2 - power rails I have a 1000w PSU on the base board, which I thought would be enough if I limited the v100s to 150w. It didn't work. I could load a model with -sm layer but not -sm tensor. If I tried to do that, the base board would lose power. Best guess is that this is because even though I was under the overall power capacity, it still exceeded the per-rail capacity. A single-rail (or jumpable) PSU is needed to power the baseboard. Or at least fewer rails, maybe, who knows? So, FYI that matters, if it seems like something you're dealing with.

V100: 981 GB/s

V100S: 1130 GB/s

It's a free 15% performance bump as long as you can cool it. Has anyone been successful in converting one?

V100S Specs:

- SXM2 Version: https://www.techpowerup.com/gpu-specs/tesla-pg503-216.c3790

- PCIe Version: https://www.techpowerup.com/gpu-specs/tesla-v100s-pcie-32-gb.c3467

Hey all,

I am currently running a single 3090ti with Swift Qwen 3.8 27B Q4_K_M with llama.cpp, getting ~45 then/sec with 113k context using Hermes.

I am mulling over the idea of grabbing a 32gb V100 pcie, selling my 3090ti, then grabbing a second 32gb V100 with proceeds from that. I was debating going with 4 16gb sxm cards but somehow it seems that doing 2 pcie cards ends up being a little cheaper.

Am I making a stupid decision? I'm not running anything crazy at the moment, just kind of playing with local LLMs and seeing what they can do.

I've been working on this for a little while now. PR is mostly ai written but I've had a lot of influence and rewrites on it. I'll explain a little more clearly here:

When I first setup 1cat I noticed I was getting incredible performance on my quad nvlink baseboard and not even close one my 2 card boards or even when I did 2x2. When I looked deeper I found that there is a low latency all-reduce feature that only enables in 'all 4x nvlink' configuration. It was pretty trivial to enable this for specific configurations I wanted such as 2x2 and it has reasonable increase in token generation.

I also discovered several settings that MUST be configured in order to get peer to peer working over PCIE, something that would impact anyone using PCIE cards as well.

• IOMMU translation: intel_iommu=off
• PCIe ACS request/completion redirection on the PEX switch ports: pci=disable_acs_redir=

I wanted to get this out for people to use but also didn't want to try playing wack-a-mole with individual configurations. I have a rather complicated configuration if I want to do TP8 where I have 3 nvlink islands spanning 2 cpus forming a sort of hybrid mesh. This change now has code to not only figure out the best order of operations to complete an all reduce based on latency and size but also selects the appropriate communication method for each connection.

In my case that means it selects a 2 step process where all nvlinks reduce first and at the same time 1/2 of the cards send over each plx switch to keep the traffic distributed and latency low, followed by an additional nvlink reduce.

Some info on "All-reduce" for those less into the weeds on this: When you are doing tensor parallel (TP) between a bunch of cards, at the end of each token the cards need to share their answers with all the other cards in that TP group before you can start calculating the next token, since in standard LLM's the output of the next token is influenced by the previous.

I have only tested this locally, I would love to see some numbers from anyone else while its waiting for review.

Check your peer to peer status like so - heres my craziness:

# nvidia-smi topo -p2p r
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7
 GPU0   X       OK      OK      OK      TNS     TNS     OK      OK
 GPU1   OK      X       OK      OK      TNS     TNS     OK      OK
 GPU2   OK      OK      X       OK      TNS     OK      TNS     TNS
 GPU3   OK      OK      OK      X       OK      TNS     TNS     TNS
 GPU4   TNS     TNS     TNS     OK      X       OK      OK      OK
 GPU5   TNS     TNS     OK      TNS     OK      X       OK      OK
 GPU6   OK      OK      TNS     TNS     OK      OK      X       OK
 GPU7   OK      OK      TNS     TNS     OK      OK      OK      X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

and check nvlink topology if you have it:

$ nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV2     PIX     PIX     SYS     SYS     NV2     NV2     0,2,4,6,8,10    0               N/A
GPU1    NV2      X      PIX     PIX     SYS     SYS     NV2     NV2     0,2,4,6,8,10    0               N/A
GPU2    PIX     PIX      X      PIX     SYS     NV6     SYS     SYS     0,2,4,6,8,10    0               N/A
GPU3    PIX     PIX     PIX      X      NV3     SYS     SYS     SYS     0,2,4,6,8,10    0               N/A
GPU4    SYS     SYS     SYS     NV3      X      PIX     PIX     PIX     1,3,5,7,9,11    1               N/A
GPU5    SYS     SYS     NV6     SYS     PIX      X      PIX     PIX     1,3,5,7,9,11    1               N/A
GPU6    NV2     NV2     SYS     SYS     PIX     PIX      X      NV2     1,3,5,7,9,11    1               N/A
GPU7    NV2     NV2     SYS     SYS     PIX     PIX     NV2      X      1,3,5,7,9,11    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hi there!

I am in the middle of sourcing components, and together with non-local AI, I am experimenting with blender, and slowly putting the components together.

The frame is sourced from an old Ender5, and I will print/buy any additional components.

Has anyone got good experience with AIO for quad solutions? Is it doable, or should I stick to air-cooling?

I'm currently looking to build up a 2x 32GB v100 system, potentially expanding to 3 or 4 down the road for single-user inference (no training). Considering I have an existing server with an H11DSi + 1600W power supply, I'm thinking of going with the PCIe variants. With 2 16x slots and 3 8x slots - would additional cards in the 8x slots have a significant performance penalty?

I know already that LLMs can make use of both GPU's, but is that the case on ComfyUI too?

Right now things like Qwen Image, SeedVR2 etc does fit to my single 16GB V100, but it's very slow to work with. Heard there are tools like Raylight that can make use of both GPU's to speed up iteration times, how effective is that without NVLink but both being on PCIe 3.0 x16 or x8. Can it speed up by at least 1.8-1.9x times?

I wanted visibility on the links and the GPUs. If this looks useful I could make it public.

Update: public repo is up. Link in comments.

Hi,

I'm new to the V100 platform and am still waiting on the two cards I ordered.

Is anyone running MiniMax H3, Krea 2/3, Flux 2, Qwen Edit or similar at acceptable speed and quality on V100s, or is that a dead end?

I know we have to work around the lack of INT8 tensor core acceleration with Volta, as it only supports FP16 with FP32 accumulation, but it's not clear to me if there are viable workflows and model recipes for us on the image/video/audio gen side.

edit:

SeedVR2 fix - https://github.com/duckyman-ai/ComfyUI-SeedVR2-V100.git

Minimax H3 fix - https://github.com/Amduraznak/minimax-h3-fp16-fix
DLSS 5 - https://github.com/lshan99q/comfyui-dlss5-pre-rtx

Qwen Image - https://github.com/lotus0000/ComfyUI-Qwen-V100-MixedPrecision

Voxtral-Mini - https://github.com/eelcor/voxtral-realtime-v100

Hi everyone,
I got two of these Tesla cards and i'm wondering if there's any possibility of getting nvlink adapter..

My motherboard only has 1 full x16 PCIE connected directly to CPU and the other slots are all either x4 or x1 connected via chipset.

*should have researched first before buying (facepalm) *

While I can run two LLMs separately in both the cards, it is always better to run bigger models with tensor parallelism, isn't it?

I found few nvlink adapter options on ebay: https://ebay.io/m/mFGtjh and https://ebay.io/m/TPFqdN

Those had mentioned V100 in the title, but i'm not very sure if those are supposed since these GPUs are ideally part of DGX station server with 4-way-nvlink https://www.exxactcorp.com/NVIDIA-DGXS-2511D-P2CMI00-E2143350

Any advice would be great!

Thanks

Just wondering what everyone is using. 1 ATX + 2x 12vHPWR + 2x PCIE and 2x CPU is kind of an awkward configuration.

I bought the MSI A1250GS ll because it was cheap, had the right connector, and is above 1200w (I plan to tdp cap them but wanted to handle bursts of the theoretical max for safety).

It was kinda hard to find a PSU that had all the right connectors so I was just wondering what other landed on

URGENT: if you don’t read anything else, be very careful with some or the blower shroud designs as they can DAMAGE your card / cables. Look at the last 3 pics for one particular design which is common. It places significant stress on the power connector. It flexes the connector / causes the cables to have an extreme bend / and transmits a ton of vibration to that connector and the card. The sample I have if I tightened it down would have probably popped my connector off the board. When you tighten it down it doesn’t feel like you are putting much force on the connector but it is significant with this design.

You can see they are trying to have this design work for multiple cards. The spot where he power connector would go has a groove that is obviously for the latching clip of the connector. However on v100/p100 cards that latch is on the other side of the connector as oriented on the card. These shrouds are advertised as fitting other cards like P40, K80, etc and they might fit those but they should not be used on V100/P100 cards.

Lastly this design forces you to attach the shroud to the card first then use an extremely tight friction fit to install the fan into the shroud. The force necessary to do this can be significant and you run the risk of damaging your card, fan, or both. Just avoid this specific blower design at all costs imho. (Note, there are other straight through blower designs I have not tested, and many people use without issue, above problems).

———

Rest of my notes so far on various shrouds and cooling performance. I have a dual pcie v100 setup:

I’ve attached pictures of various compact cooling options for v100 and similar cards (P100 is what I have).

The pcie cards can be challenging to cool since they need both high airflow and pressure. Outside of a server chassis regular PC cases can have issues with suitable cooling.

If you search eBay, amazon, and various 3d file sites you will find a variety of shrouds. I have tried several meant for 40mm fans and 97x33mm blowers.

Let’s start with the 40mm shrouds. If you get a shroud that only uses a single fan you will have to greatly reduce the power of your card and it will still throttle under load. Using artic 15k high pressure fans will have the card on edge with memory causing throttling pretty fast. In my workloads the HBM runs 5-15 degrees hotter than the compute on the card.

Also, I highly recommend not getting any shroud that attaches to multiple cards. The issues here are removing / servicing your cards once they are installed. If your motherboard has pcie locks that auto lock after inserting the card, you will need a special tool or make your own shim to get the bottom card unlocked. It’s not easy and you run the risk of damaging your card / board or both. Some motherboards have better ways to unlock the cards, in that case it may be ok.

The dual 40mm black design in pic 2 and 3 is such a design. The white shroud in pic one and bottom left of 2 and 3 is a single card 40mm design. Just avoid these as they are just not worth it.

The 40mm design that does work is the dual 40mm fans per card. When paired with the artic 15k fans it keeps the v100 memory at 70 deg and good power limits. No throttling at all. That is in pic 4 (pic from listing as I didn’t uninstall it).

Next up the two blower designs. I highlighted the severe issues with the one design in the first paragraph. The second design is the curved design.

This design also suffers from vibration issues. This is especially the case if your cards are mounted horizontally on a vertical motherboard. It’s not as bad when the cards are vertical / case on its side. The shroud bend amplifies the vibration at various speeds since it flexes. I can feel / hear the vibration through the whole card and even case. To help mitigate / dampen this I have an adjustable GPU bracket that I have supported the shroud which reduces vibration significantly.

More importantly this design severely hampers the performance of the blower fan with that sharp bend. The fan I am using has more than triple the pressure and double the airflow of the two artic 40mm fans combined but it can barely keep the memory under 80 degrees under load.

I advise against using this shroud design unless there are no other options.

So… what am I going to settle on using? I am going to use a second dual 40mm shroud to cool my second card. A straight through blower does absolutely cool the best, but is extremely tight for my case and the noise seems to be worse (will be posting noise measurements soon to my GitHub below and here).

In the long term / in retrospect, I would probably go down the SXM2 route to get both nvlink and less expensive water cooling options. There are blocks for the pcie cards but they are hard to find and fairly expensive (~$200).

All my work on my v100 setup is here:

https://github.com/bradrlaw/ai-server

For linux users:

iommu prevents peer to peer for working between nvidia cards over pcie ... I actually would like to see some numbers from someone with two pcie v100s

PCIE switches like PEX/PLX apparently need to have ACS disabled to allow peer to peer communication?

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=off pci=disable_acs_redir=<list pcie devices>"

I just did this and tested using 1cat's low latency algorithm for communication/all-reduce on a TP2 and got results nearly identical to using nvlink.

It's worth noting in this case I am also using a pcie switch (PEX) and both of these cards are connected to the same switch card even though they are not on the same nvlink island.

There might be problems with my testing which is why im looking for others to help confirm the results.

So i have been looking at different options for my budgetbuild, and the BYANSKI cooler heads are super nice, but super expensive...

Have anyone managed to fit any of those 50 dollar AIO kits onto each SXM2 GPU? And does it pull away enough heat?

Hey, I am looking at buying a V100(s) with a budget of $700 USD.

I have 128gb DDR4 2400 in a quad-channel on an X99 system, so of course I will be trying strata with flash next. I am undecided between running Qwen3.8-27b, and flash next, I want to be able to run both at a decent speed, and run 27b with a concurrency of 4 (subagents) without much slow-down. Hoping around 100t/s decode if possible.

I am thinking buying two 16gb SXM2, and getting a two-way NVLINK PCIe card, that way I get 32gb total, and better performance by using TP.

Does anybody run this config?
What speeds can you reach?
How does it compare to dual 16gb PCIe cards?
How does it compare to a single 32gb card?

So TLDR; I want at least 100t/s on Qwen 3.8 27b with a concurrency of4, and have USD$700 to spend. Is that doable? What’s the best option?

So i am trying to navigate in the jungle that is automated alibaba agents with language capabilities, and less thinking capabilities. Its been a journey. Ive tried recieving everything from hallucinated user manuals, them pulling 10 year old sales data on me etc, in bids to get me to purchase theirs...

I need a quad base board with PLX i guess ? Do i want the version with PLX build in? The version thats black with no PLX, the green one with single PLX, the mysterious one with 2x PLX and no cooling, or the PLX version with 2x additional PCIE ??

Heeeeeeeeeeelp

I see a few different options online, I plan to build a 4-6 pcie machine and do it open bench style, so I don't have to worry about case constraints. But I am concerned about the noise. What shroud/fan combo have you found to work best and keep sound reasonable?

More work than it seems on the surface to get this NVLink setup going. Should be another day or so. They say this is an EEB layout but only 40% actually lines up on the EEB standards. I designed a tray that matches all the mounts and has support for the GPU stand-offs.

2sxm2, one pcie

Hello,

I have a V100 32gb PCIe that I wanted to add to my PC, which has a 6700k, 3060ti, on an ASRock Fatal1ty Z170 Professional Gaming i7 MoBo, running Ubuntu.

After connecting the V100, in lspci I can see the card listed as a 3D Controller. However, I cannot see the V100 listed under nvidia-smi, only the 3060ti.

I'm thinking that the problem is that my motherboard does not support Above 4G Decoding / Resizable BAR? I've tried changing settings and flashing the BIOS, but have not found these settings.

Has anyone else experienced problems like this, and is the only option to get it working to switch to another motherboard?

Hi everyone. Thank you for being so helpful. I finished my build with 2 V100 32gb PCIe.

I had luck to find cheap machine Fujitsu M770 (beautiful machine) with xenon CPU and 64 DDR4.

Machine have PSU 800W and GPUs are powered from separate 1200W PSU (on top) using CPU 8-pin cables.

Machine is running ubuntu 24 server with llama.cpp and main model I'm running is qwen3.8 27b Q6 with 256k context window.

Issue was with hardware test on boot, 8 beeps and black screen on video out GPU. In bios in menu PCI Subsystem settings "Above 4G Decoding" need to be enabled and it will stop complaining.

As I written before I run Qwen3.8 27b Q6 and getting TPS is 27 and I cannot get more even with lower quants (Q4) for some reason.

Here is llama.cpp config:

[Unit]
Description=llama-server inference API
After=network.target

[Service]
Type=simple
User=gordon
WorkingDirectory=/home/gordon/llama.cpp
ExecStart=/home/gordon/llama.cpp/build/bin/llama-server \
  -m /home/gordon/llama.cpp/models/Qwen3.8-27B-UD-Q6_K_M.gguf \
  --mmproj /home/gordon/llama.cpp/models/mmproj-F16-Qwen-Qwen3.8-27B.gguf \
  --host 0.0.0.0 \
  --port 5544 \
  --ctx-size 524288 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --parallel 2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

For cooling I found some old server jets PFR0612DHE, 14000 RPM. Tested GPUs on full load cannot pass 55°C. For now it is blowing 80% all the time. Fans are also powered from 1200W power supply because each is 3.3A in full power. Soon I will build ESP32 based controller that fill get GPU temperatures from OS and control fans, that is the plan.

Here is duct I designed and 3D printed for this GPUs (tested with PLA and ABS, both good, ABS better), there is also DFX with dimensions attached so anyone can adjust it.

https://www.printables.com/model/1872957-nvidia-v100-pcie-gpu-duct-for-60x60mm-fan

If anyone can help with getting more tokens without using MTP models I would be grateful.

Edit:

Here is updated configuration and real result.

[Unit]
Description=llama-server inference API
After=network.target

[Service]
Type=simple
User=gordon
WorkingDirectory=/home/gordon/llama.cpp

# --- CRITICAL V100 ENVIRONMENT VARIABLES ---
Environment=CUDA_DEVICE_ORDER=PCI_BUS_ID
# Enable P2P communication between GPUs
Environment=GGML_CUDA_P2P=1

# --- EXECSTART COMMAND ---
ExecStart=/home/gordon/llama.cpp/build/bin/llama-server \
  -m /home/gordon/llama.cpp/models/Qwen3.8-27B-Q8_0.gguf \
  --mmproj /home/gordon/llama.cpp/models/mmproj-F16-Qwen-Qwen3.8-27B.gguf \
  --host 0.0.0.0 \
  --port 5544 \
  --ctx-size 256288 \
  --flash-attn on \
  --split-mode tensor \
  --tensor-split 1,1 \
  --batch-size 4096 \
  --ubatch-size 512 \
  --parallel 1 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --chat-template-kwargs '{"reasoning_effort":"medium"}'

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Fans on GPUs are controlled using esp32 devkit v1 board ESP32 based. Here is simple schematic. I created small service that follows GPU temperatures in OS and over serial USB send fan setup commands to esp32.

Open question:

What is the best Qwen 3.8 Flash Next Runtime - 4 x 32GB NVLinked V100s, X99 Dual Xeon E5-2696 V3, 64GB RAM

I'm going to give the 1CatvLLM route a go, but claude tells me the PLE tables are going to need 48GB of RAM.

My system splits the RAM, 32GB for each NUMA CPU.

I've got some more RAM on order, but you know.. impatient.

----

So what's the recipe?

Use the Strata-V100 port for "Low RAM" (cough) systems like mine.
It works well with the correct environment and launch parameters.

Model:

IQ3_S GGUF from ISTA-DASLab

Performance with MTP spec 6, min-p 0.7:

105–110 decode
1240 prefill at 4K
3170 prefill at 32K

But... if you can get 64GB per NUMA CPU, use the Cat1 vLLM port.
You can run higher quants and decode about 2 x faster (measured at 200+ TPS).
The decode speed is due to NVLink Tensor Parallelism.

There's a long Opus generated summary in the comments.
That was about 12 hours of vibing and fluffing around.

Built this thing in march with Dual P100 PCIe 16GB cards but in the last month upgraded to Dual 32GB V100s

CPU: Ryzen 7 5700G (For the iGPU) @ 4.7GHz
GPU: Dual Nvidia Tesla PCIe V100 32GB
RAM: 96GB DDR4 @ 3000MHz
Mobo: Gigabyte B550 Gaming X v2
OS: Linux Mint 22.3
Case: Fractal Design Focus 2 (Glass Side Panel)

Various Storage, 256/512GB NVMEs, 512GB SATA SSD, 2TB SATA HDD

Love this thing!!

I’m looking for a fan shroud that fits an NVIDIA Tesla V100 and lets me mount a 120mm fan for cooling.

I do understand there are easier and other ways but i want the coolest system to be silent and i have 2 120mm fans lying around.

Does anyone have an STL file I could use to 3D-print one? If you’ve tried a similar setup, I’d appreciate any advice on fit and airflow.

Thanks!