llmlingua
To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Installation
To install this package, run one of the following:
Usage Tracking
0.2.2
0.2.1
2 / 8 versions selected
Downloads (Last 6 months): 0
About
Summary
To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Last Updated
Apr 9, 2024 at 09:34
License
MIT
Total Downloads
4.7K
Supported Platforms
noarch