community[feat]: Adds LLMLingua as a document compressor (#17711)
**Description**: This PR adds support for using the [LLMLingua project ](https://github.com/microsoft/LLMLingua) especially the LongLLMLingua (Enhancing Large Language Model Inference via Prompt Compression) as a document compressor / transformer. The LLMLingua project is an interesting project that can greatly improve RAG system by compressing prompts and contexts while keeping their semantic relevance. **Issue**: https://github.com/microsoft/LLMLingua/issues/31 **Dependencies**: [llmlingua](https://pypi.org/project/llmlingua/) @baskaryan --------- Co-authored-by: Ayodeji Ayibiowu <[email protected]> Co-authored-by: Harrison Chase <[email protected]>
A
Ayo Ayibiowu committed
ac1d7d9de81997e31eb6982e303ef5294641f686
Parent: a99eb3a
Committed by GitHub <[email protected]>
on 2/28/2024, 3:23:56 AM