RagPrepKit prepares documents for RAG and LLM pipelines
RagPrepKit is an open-source Python toolkit for Retrieval-Augmented Generation (RAG) and Large Language Model (LLM) pipelines. It covers cleaning, chunking, metadata extraction, token estimation and cost analysis.
Get Started View on GitHub PyPI PackageWhat RagPrepKit does
RagPrepKit turns raw documents into cleaned, chunked text with metadata and token counts, using one Python toolkit.
Text cleaning
Automatically remove unnecessary formatting, normalize text, and improve document quality before ingestion.
Chunking strategies
Split documents into meaningful chunks optimized for retrieval, embeddings, and language model processing.
Metadata extraction
Extract useful metadata to improve search, filtering, retrieval accuracy, and document management.
Token and cost estimation
Estimate token usage and potential LLM costs before processing large datasets.
Features
- Text cleaning and normalization
- Multiple chunking strategies
- Recursive chunking support
- Sentence-based chunking
- Metadata extraction
- Token counting
- Cost estimation
- AI and RAG workflow optimization
- Simple Python API
Use cases
- RAG applications
- Knowledge bases
- Enterprise search systems
- AI chatbots
- Document intelligence platforms
- LLM fine-tuning pipelines
- Vector database ingestion workflows
Installation and usage
RagPrepKit installs from PyPI and runs on Python 3.9 and later with no required runtime dependencies.
# Install from PyPI pip install ragprepkit # Optional extra: exact token counts via tiktoken pip install "ragprepkit[tokenizers]"
from ragprepkit import clean_text, recursive_chunks, count_tokens
# 1. Clean the raw document
text = clean_text(raw_document, remove_boilerplate=True)
# 2. Split it into retrieval-sized chunks
chunks = recursive_chunks(text, chunk_size=800, overlap=80)
# 3. Count the tokens in each chunk before embedding
for chunk in chunks:
print(chunk.index, count_tokens(chunk.text))
The package is published on PyPI and imported as ragprepkit (releases before 0.1.4 used the import name ragprep). Each chunk carries text, start_char, end_char, index and an open metadata dict that can be populated before the chunk is handed to an embedding step.
Package details
- Current version
- 0.1.4
- Supported Python versions
- 3.9, 3.10, 3.11, 3.12, 3.13
- License
- MIT License
- PyPI package
ragprepkit- Import name
ragprepkit- Runtime dependencies
- None required
Documentation
The documentation covers installation, examples, the API reference, chunking strategies and recommended practices for AI applications.
Read DocumentationContributing
RagPrepKit is an open-source project developed by Mindrops and welcomes community contributions, feedback, and feature requests.
Contribute on GitHubAbout Mindrops
RagPrepKit is maintained by Mindrops as part of its commitment to building practical open-source developer tools and AI infrastructure.
Explore More Open Source ProjectsFrequently asked questions
How do I install RagPrepKit?
pip install ragprepkit. For exact token counts via tiktoken, add the optional extra with pip install "ragprepkit[tokenizers]".What is the import name?
from ragprepkit import clean_text, recursive_chunks, count_tokens. Version 0.1.3 used ragprep.Which Python versions does RagPrepKit support?
What licence is RagPrepKit released under, and where is the source?
What can I build with RagPrepKit?
Need a RAG system built for you?
Mindrops builds retrieval-augmented generation (RAG) pipelines, AI search and document AI on your own data, with citations, guardrails and evaluation.
RagPrepKit on GitHub
The source code, documentation and issue tracker for RagPrepKit are on GitHub.
View on GitHub Documentation Report an Issue