FirecrawlData

Renders JavaScript, follows links and strips the navigation and boilerplate that otherwise poison a retrieval corpus. Solves the unglamorous half of RAG properly. AGPL-licensed, so check the terms if you plan to expose it as a service.

ragflowRAG & Retrieval

RAGFlow is a server you run rather than a library you import: documents go into a dataset, it splits them into chunks, and you can read every chunk on screen and hand-correct the ones that came out wrong before any of it reaches a model, with chat answers citing the chunk they came from. Around that sit an agent canvas and an ingestion pipeline you can rebuild yourself, so retrieval, tools and MCP servers can be wired together without writing code. The optical character recognition, table-structure and layout models that make sense of scans run on PDFs and images only — Word, Excel and PowerPoint files are read structurally, and the docs tell you to convert a DOCX to PDF if you want the visual parser on it. The price of all this is weight: a Docker Compose stack with Elasticsearch, MySQL, MinIO and Redis behind it, 4 cores and 16 GB of RAM as the stated minimum, and parsing that the project's own FAQ concedes is slower than LangChain's. If your documents are already clean text and you want retrieval inside your own Python application, LlamaIndex or Haystack will fit better.

PaddleOCRData

Retrieval over your own documents usually fails long before the model does, at the point where a scanned table or a two-column PDF becomes an unreadable wall of text. PaddleOCR is the layer that stops that. It recognises text in more than a hundred languages, and its document pipeline preserves what the layout meant — headings stay headings, a table comes out as a table with its cell positions, formulas and seals are handled — emitting Markdown or JSON. The models are deliberately small enough to run on a CPU or at the edge, which is what makes processing a large archive affordable. The cost is the framework underneath: this is a PaddlePaddle project, and installing it means installing that.