r/dotnet • u/fuzhongkai • 5h ago
Article I built a .NET LLM inference engine that runs an 816B MoE model across 10 GPUs — here’s what I learned about C# and native performance
github.comI’ve been developing TensorSharp, an open-source LLM inference engine and agent runtime built with C#/.NET.
One question I’ve been exploring is:
Can .NET serve as the foundation for a high-performance LLM inference engine, rather than just being a wrapper around Python-based inference servers?
Recently, I got Muse Spark 816B (42B active parameters) running across 10 NVIDIA A40 GPUs and benchmarked TensorSharp against llama.cpp.
I wanted to share some results and engineering lessons, particularly around managed/native execution, multi-GPU scheduling, and where C# fits into performance-critical AI infrastructure.
The architecture
TensorSharp uses C#/.NET for the inference engine’s orchestration, model execution management, request scheduling, KV cache management, and agent runtime.
GPU computation is handled through native backends, including CUDA, Metal, and Vulkan.
The idea isn’t to reimplement CUDA kernels in C#. Instead, it’s to use .NET for the higher-level systems engineering while keeping computationally intensive operations in optimized native code.
This separation has some interesting benefits:
- A unified runtime for model inference and agent execution.
- Cross-platform support through different GPU backends.
- Integration with existing .NET applications without requiring a separate Python inference service.
- The ability to implement scheduling, caching, concurrency, and agent workflows in C#.
But it also raises an important performance question: where should the boundary between managed and native execution be?
The experiment: 816B parameters on 10 GPUs
Test configuration:
- Model: Muse-Spark-Open-816B-A42B
- Quantization: Unsloth UD-IQ3_XXS (~316 GiB of weights)
- Hardware: 10× NVIDIA A40 (46 GB each)
- Context: 32K tokens
- KV cache: F16
- All model experts resident in GPU memory
I tested two multi-GPU execution strategies:
Layer splitting: Different GPUs own different layers.
Tensor parallelism: Each layer is distributed across multiple GPUs, requiring synchronization and communication.
Both were compared against llama.cpp on the same machine using the same model and requests.
Results
| Benchmark | TensorSharp | llama.cpp |
|---|---|---|
| Layer split: single-request decode | 16.69 tok/s | 16.79 tok/s |
| Layer split: 4K prefill | 999.7 tok/s | 483.2 tok/s |
| Layer split: 16K prefill | 1,346.8 tok/s | 826.1 tok/s |
| Tensor parallel: decode | 22.63 tok/s | 16.19 tok/s |
| Tensor parallel: 4 concurrent requests | 45.73 tok/s | 24.49 tok/s |
The most interesting result to me isn’t necessarily the comparison against llama.cpp.
It’s how much performance changed after moving the right parts of execution across the managed/native boundary.
Lesson 1: Native execution granularity matters enormously
TensorSharp previously had a managed tensor-parallel execution path that dispatched operations individually.
It worked, but performance was poor.
On this model, it achieved only 1.9–2.4 tokens/sec.
After switching to a native whole-model execution path using upstream ggml’s meta backend, TensorSharp reached 22.6 tokens/sec.
That’s roughly a 10× improvement over our previous implementation.
The lesson wasn’t that C# is inherently slow.
It was that executing too many fine-grained GPU operations across a managed/native boundary introduces unnecessary overhead.
Keeping the higher-level scheduling and orchestration in .NET while moving the performance-critical execution graph into native code made a huge difference.
Lesson 2: GPU scheduling can matter more than raw compute
For layer-split inference, we implemented pipelined prefill.
Instead of processing the entire prompt sequentially through GPU 0, GPU 1, GPU 2, etc., the prompt is divided into micro-batches.
While GPU 1 processes micro-batch A, GPU 0 can already start processing micro-batch B.
We also implemented adaptive micro-batch sizing and larger prefill steps for single-request workloads.
With these changes, TensorSharp reached approximately 1,347 prefill tokens/sec on a 16K-token prompt.
That’s 1.63× the llama.cpp result on the same hardware.
This is a good example of why inference performance isn’t just about writing faster kernels. The execution and scheduling strategy matters enormously.
Lesson 3: More parallelism isn’t always faster
Tensor parallelism improved generation throughput, but it was significantly slower than layer splitting for long-prompt prefill.
In TensorSharp:
- Layer-split prefill: ~1,347 tok/s
- Tensor-parallel prefill: ~295 tok/s
- Layer-split decode: ~16.7 tok/s
- Tensor-parallel decode: ~22.6 tok/s
The reason is communication overhead.
Muse Spark’s MoE architecture requires substantial cross-GPU reductions, and this particular machine had broken GPU peer-to-peer transfers.
NCCL therefore had to communicate through host memory.
This made the optimal execution strategy workload-dependent.
It’s also an interesting problem for a .NET scheduler: selecting execution strategies based on model architecture, hardware topology, and request characteristics.
Lesson 4: Performance optimizations need correctness validation
We also compared next-token probability distributions against llama.cpp on 12,276 positions from WikiText-2.
TensorSharp achieved:
- 98.84% top-1 agreement in layer-split mode.
- 98.82% top-1 agreement in tensor-parallel mode.
- Perplexity of approximately 1.848, compared with llama.cpp’s 1.847.
We additionally ran integration checks covering JSON generation, tool calling, multi-turn conversations, cache reuse, and concurrent requests.
TensorSharp passed 13/14 checks with layer splitting and 14/14 with tensor parallelism.
These aren’t comprehensive model-quality evaluations, but they help catch regressions while optimizing the execution engine.
A note about the comparison
There are some important limitations.
llama.cpp’s NCCL-enabled build failed on this particular 10-GPU setup due to CUDA peer-access restrictions, so its tensor-parallel results used the non-NCCL implementation.
TensorSharp includes a workaround that allowed NCCL to run over shared memory.
Therefore, the tensor-parallel performance gap should not be interpreted as a general advantage over llama.cpp on all hardware.
The layer-split comparison is more directly comparable, and single-request decode performance was essentially identical.
All measurements were taken on one machine with one model. I’d like to test more hardware configurations, especially systems with working P2P or NVLink.
Why I’m sharing this here
A lot of LLM infrastructure is built around Python, C++, and CUDA.
I think there’s an interesting opportunity for .NET to participate more directly in this space, especially for developers who want to build AI-powered applications without maintaining a separate Python-based inference stack.
TensorSharp is my attempt to explore that direction.
It’s open source, and the full benchmark report includes the methodology, configuration, limitations, and reproduction scripts.
GitHub: https://github.com/zhongkaifu/TensorSharp
Full benchmark: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/perf/muse-spark-vs-llama-cpp.md
I’m particularly interested in hearing from other .NET developers:
Have you worked on performance-critical systems that combine C# with native C++/CUDA libraries? Where have you found the managed/native boundary works best?
And more broadly, do you think .NET has a larger role to play in local AI inference infrastructure, or is it better suited to application-level orchestration?
