Inference tooling decides what a model costs to run and how fast it answers. These projects cover serving engines, batching, quantisation and the memory tricks that make a model fit hardware you already own.
Inference is where AI budgets are actually spent. Training makes the headlines; serving makes the invoice — and a factor-of-three difference in throughput between two engines running the same weights is routine, not exceptional.
The projects here sit at different points on that curve: raw engines, routing and batching layers, and memory optimisations that let a larger model fit on smaller hardware. Each description says which of those it is, because mixing two layers that both want to own scheduling is the most common way this stack goes wrong.
18 analysed projects match this topic.
Skip weeks of custom work by starting from a working inference optimization foundation.
Skip weeks of custom work by starting from a working ai memory foundation.
Skip weeks of custom work by starting from a working llm inference foundation.
Skip weeks of custom work by starting from a working llm framework foundation.
Skip weeks of custom work by starting from a working token compression foundation.
Skip weeks of custom work by starting from a working machine learning foundation.
Skip weeks of custom work by starting from a working large language models foundation.
Skip weeks of custom work by starting from a working large language models foundation.
Skip weeks of custom work by starting from a working kv cache foundation.
Skip weeks of custom work by starting from a working recursive language models foundation.
Skip weeks of custom work by starting from a working llm inference foundation.
Skip weeks of custom work by starting from a working model compression foundation.
Skip weeks of custom work by starting from a working rust foundation.
Skip weeks of custom work by starting from a working characterai foundation.
Skip weeks of custom work by starting from a working kv cache foundation.
Skip weeks of custom work by starting from a working llm inference foundation.
Skip weeks of custom work by starting from a working ai optimization foundation.
Skip weeks of custom work by starting from a working claude code foundation.
Updated: 2026-08-17