Improving performance for PDF loader
diff --git a/ingest.py b/ingest.py
index 293013846..0ca80743f 100755
--- a/ingest.py
+++ b/ingest.py
@@ -9,7 +9,7 @@
 from langchain.document_loaders import (
     CSVLoader,
     EverNoteLoader,
-    PDFMinerLoader,
+    PyMuPDFLoader,
     TextLoader,
     UnstructuredEmailLoader,
     UnstructuredEPubLoader,
@@ -73,7 +73,7 @@ def load(self) -> List[Document]:
     ".html": (UnstructuredHTMLLoader, {}),
     ".md": (UnstructuredMarkdownLoader, {}),
     ".odt": (UnstructuredODTLoader, {}),
-    ".pdf": (PDFMinerLoader, {}),
+    ".pdf": (PyMuPDFLoader, {}),
     ".ppt": (UnstructuredPowerPointLoader, {}),
     ".pptx": (UnstructuredPowerPointLoader, {}),
     ".txt": (TextLoader, {"encoding": "utf8"}),
diff --git a/requirements.txt b/requirements.txt
index 8c43e53bd..acfb584f9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,7 +3,7 @@ gpt4all==0.2.3
 chromadb==0.3.23
 llama-cpp-python==0.1.50
 urllib3==2.0.2
-pdfminer.six==20221105
+PyMuPDF==1.22.3
 python-dotenv==1.0.0
 unstructured==0.6.6
 extract-msg==0.41.1

The PyMuPDF brings much better performance for PDF loading.

https://pymupdf.readthedocs.io/en/latest/about.html#performance
Here is a comparison article:
https://medium.com/social-impact-analytics/comparing-4-methods-for-pdf-text-extraction-in-python-fd34531034f

It's about **5x-30x faster** for text extraction.

And In my few test cases, PyMuPDF have better result in CJK characters, Although they both may display a lot of garbled characters.

The different is PDFMinerLoader make a single document object but PyMuPDFLoader make each page as a document object.

This PR already contains the change that I've made for this update:
https://github.com/imartinez/privateGPT/pull/560/files

* Remove read subscript [0] only for _loader.load()_
* Update _results.append(doc)_ to  _results.extend(docs)_