Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.
Build your own conversational search engine using less than 500 lines of code by LeptonAI.
Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.
simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.
Framework to create ChatGPT like bots over your dataset.
a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!
Easily build, version, evaluate and deploy your LLM-powered apps.
Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
Use ChatGPT On Wechat via wechaty
Seamlessly integrate LLMs as Python functions
Lightweight alternative to LangChain for composing LLMs
Comprehensive set of tools for working with local LLMs for various tasks.
A Python library for augmenting LLM apps with data.
A popular Python/JavaScript library for chaining sequences of language model prompts.
DSPy: The framework for programming—not prompting—foundation models.
Easily deploy any LLM on a VM with minimal configuration, using Ansible.
A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.
Efficient Triton Kernels for LLM Training.
A high-throughput and low-latency inference and serving framework for LLMs and VLs
Inference for text-embeddings in Python
Inference for text-embeddings in Rust, HFOIL Licence.
MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.
Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at BentoML for LLMs-based applications.
Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
Blazingly fast LLM inference.
A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)
Nvidia Framework for LLM Inference
SGLang is a fast serving framework for large language models and vision language models.
Open-source framework for fine-tuning and evaluating LLMs. It simplifies the process of experimenting with different training configurations and makes it easy to reproduce and share results, supporting features like LoRA, QLoRA, DeepSpeed, PEFT, and multi-GPU setups.
An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT).
A library for accelerating Transformer model training on NVIDIA GPUs.
An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
A simple, performant and scalable Jax LLM!
Mesh TensorFlow: Model Parallelism Made Easier.
Efficient Training for Big Models.
veRL is a flexible and efficient RL framework for LLMs.
An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models.
A Native-PyTorch Library for LLM Fine-tuning.
DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.
A native PyTorch Library for large model training.
Ongoing research training transformer models at scale.
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Minimalistic large language model 3D-parallelism training.
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.
Testing & evaluation library for LLM applications, in particular RAGs