RagPrepKit
A powerful document preprocessing toolkit for Retrieval-Augmented Generation (RAG) and Large Language Model (LLM) pipelines. RagPrepKit simplifies the preparation of documents for AI systems through cleaning, chunking, metadata extraction, token estimation, and cost analysis.
Get Started View on GitHub PyPI PackageWhy RagPrepKit?
Preparing data for RAG and LLM applications can be complex and time-consuming. RagPrepKit provides a simple and consistent toolkit for transforming raw documents into AI-ready content.
Clean Data Faster
Automatically remove unnecessary formatting, normalize text, and improve document quality before ingestion.
Smart Chunking
Split documents into meaningful chunks optimized for retrieval, embeddings, and language model processing.
Metadata Extraction
Extract useful metadata to improve search, filtering, retrieval accuracy, and document management.
Token & Cost Estimation
Estimate token usage and potential LLM costs before processing large datasets.
Features
- Text cleaning and normalization
- Multiple chunking strategies
- Recursive chunking support
- Sentence-based chunking
- Metadata extraction
- Token counting
- Cost estimation
- AI and RAG workflow optimization
- Simple Python API
- Production-ready architecture
Ideal For
- RAG applications
- Knowledge bases
- Enterprise search systems
- AI chatbots
- Document intelligence platforms
- LLM fine-tuning pipelines
- Vector database ingestion workflows
Get Started
RagPrepKit installs from PyPI and runs on Python 3.9 and later with no required runtime dependencies.
# Install from PyPI pip install ragprepkit # Optional extra: exact token counts via tiktoken pip install "ragprepkit[tokenizers]"
from ragprep import clean_text, recursive_chunks, count_tokens
# 1. Clean the raw document
text = clean_text(raw_document, remove_boilerplate=True)
# 2. Split it into retrieval-sized chunks
chunks = recursive_chunks(text, chunk_size=800, overlap=80)
# 3. Count the tokens in each chunk before embedding
for chunk in chunks:
print(chunk.index, count_tokens(chunk.text))
The package is published on PyPI as ragprepkit, while the Python import name is ragprep. Each chunk carries text, start_char, end_char, index and an open metadata dict that can be populated before the chunk is handed to an embedding step.
Package details
- Current version
- 0.1.3
- Supported Python versions
- 3.9, 3.10, 3.11, 3.12, 3.13
- License
- MIT License
- PyPI package
ragprepkit- Import name
ragprep- Runtime dependencies
- None required
Documentation
Detailed documentation includes installation instructions, examples, API references, chunking strategies, and best practices for building scalable AI applications.
Read DocumentationOpen Source
RagPrepKit is an open-source project developed by Mindrops and welcomes community contributions, feedback, and feature requests.
Contribute on GitHubAbout Mindrops
RagPrepKit is maintained by Mindrops as part of its commitment to building practical open-source developer tools and AI infrastructure.
Explore More Open Source ProjectsBuild Better AI Pipelines
Simplify document preparation for Retrieval-Augmented Generation with RagPrepKit.
View on GitHub Documentation Report an Issue
