vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs
github.com/vllm-project/vllm
1 Users
0 Comments
2 Highlights
0 Notes
Tags
Top Highlights
A high-throughput and memory-efficient inference and serving engine for LLMs vllm.readthedocs.io Topics inference pytorch transformer gpt model-serving mlops llm llmops llm-serving
Latest News 🔥 [2023/06] Serving vLLM On any Cloud with SkyPilot. Check out a 1-click example to start the vLLM demo, and the blog post for the story behind vLLM development on the clouds. [2023/06] We officially released vLLM! FastChat-vLLM integration has powered LMSYS Vicuna and Chatbot Arena since mid-April. Check out our blog post. vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: State-of-the-art serving throughput Efficient management of attention key and value memory with PagedAttention Continuous batching of incoming requests Optimized CUDA kernels vLLM is flexible and easy to use with: Seamless integration with popular HuggingFace models High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more Tensor parallelism support for distributed inference Streaming outputs OpenAI-compatible API server
Ready to highlight and find good content?
Glasp is a social web highlighter that people can highlight and organize quotes and thoughts from the web, and access other like-minded people’s learning.