I’m the developer of TensorSharp, an open-source .NET 10 inference engine for running local GGUF models. It can be embedded in .NET applications or run through a CLI and an OpenAI/Ollama-compatible server.
The project combines managed C# with native accelerator backends. The benchmarks below use native CUDA/GGML execution, including TensorSharp-specific kernels. The performance improvements come from changes to scheduling, data movement, and model execution.
I recently worked on DeepSeek V4.1 Flash 1017 on five NVIDIA A40s, and thought the engineering details might be interesting to people building performance-sensitive .NET applications.
Making the GPUs work concurrently
One of the biggest problems was in the native tensor-parallel execution path.
The model’s routed experts were distributed across GPUs, but execution repeatedly passed through a synchronous, host-driven path. Each layer involved multiple host round trips and synchronization points.
The revised implementation:
- Schedules work concurrently across GPU ranks.
- Uses CUDA events to order dependent operations.
- Overlaps the shared expert’s computation with the routed experts.
- Removes host waits from inside the layer’s execution path.
Cross-GPU transfers still use pinned host-memory staging because peer access was not working correctly on this machine.
In the implementation comparison, this change took tensor-parallel decode from 15.2 to 36.3 tokens/sec after a 2,048-token prompt with full prefill enabled.
For me, this was a useful reminder that distributing work across devices is only the beginning; the synchronization and data-transfer design determine how much concurrency you actually get.
Making native failures recoverable
The reliability work was just as interesting.
Previously, an exception on a native worker could terminate the server. A failure on the coordinating rank could leave the other workers waiting indefinitely at a barrier.
The fixes coordinate error reporting, ensure every rank reaches the required synchronization points, and drain work already issued by a failed layer. Failure-injection tests now check both the failure path and recovery on the next forward pass.
When native workers run inside a .NET application, that lifecycle becomes part of the application’s reliability story.
A small benchmark snapshot
The comparison used the same Unsloth UD-IQ2_XXS GGUF files on the same five-A40 machine, with the Engram lookup table in host memory.
TensorSharp used --tp 5. The llama.cpp baseline was Unsloth’s dsv41-all fork at be836572, using layer splitting; its row/tensor split modes did not successfully run this model in the tested build.
These are real-text server results for approximately 1,800-token prompts, generating 256 tokens with greedy decoding and thinking disabled. Each configuration had a warm-up and three measured requests, with unique prefixes to prevent prefix-cache reuse.
| Measurement |
TensorSharp |
llama.cpp |
Throughput ratio |
| Prompt processing, YOCO disabled |
563.9 tok/s |
174.6 tok/s |
3.23× |
| Prompt processing, default YOCO enabled |
960.5 tok/s |
174.6 tok/s |
5.50× |
| Token generation, no drafter |
35.5 tok/s |
29.6 tok/s |
1.20× |
| Token generation with DSpark speculative decoding |
55.0 tok/s |
41.3 tok/s |
1.33× |
Prompt processing is also called prefill. The first three rows report medians; the DSpark row reports means, with the drafter enabled in both engines. TensorSharp’s generation rows use its default YOCO configuration.
The speed/quality trade-off
YOCO bounded replay reduces prompt-processing work by running selected later layers over only the final 140 positions of eligible text-prefill calls. It is approximate and can be disabled with:
TS_DSV41_YOCO_KEEP=0
The 3.23× prefill result above has this approximation disabled.
On the report’s WikiText-2 evaluation, perplexity—lower is better—was:
- llama.cpp: 2.4697
- TensorSharp, YOCO disabled: 2.4811
- TensorSharp, YOCO enabled: 2.4978
Enabling YOCO increased perplexity by approximately 0.67% relative to TensorSharp’s full-prefill result. With YOCO enabled, perplexity was approximately 1.14% above llama.cpp’s.
That is a measured trade-off on a limited text evaluation, rather than a general percentage change in model capability. Disabling YOCO also does not make the two engines numerically identical.
Scope of the results
These measurements cover one model quantization and one hardware configuration. The llama.cpp server results came from an earlier session on the same machine and showed some unexplained prefill variability.
At measurement time, the TensorSharp build included uncommitted changes, and the raw validation records were also uncommitted. The report records the build details and remaining limitations; the current repository should not be assumed to reproduce every number directly.
The five-GPU setup is specific to this benchmark. TensorSharp also supports smaller models and managed CPU, native CPU, CUDA, Metal, and Vulkan execution paths, with support varying by model.
Links:
If you’re integrating local models into .NET applications, do you prefer an in-process engine or a separate inference server? I’d be interested in what drives that choice in practice, especially around deployment, memory ownership, and recovering from native failures.
