Christian Heilmann

PrivateGPT – Running “ChatGPT” offline on local documents

Saturday, May 27th, 2023 at 4:04 pm

You can download the GPT model and interrogate local files using Python without having to send any data to the cloud.

PrivateGPT is a python script to interrogate local files using GPT4ALL, an open source large language model. It is pretty straight forward to set up:

  • Clone the repo
  • Download the LLM – about 10GB – and place it in a new folder called `models`.
  • Place the documents you want to interrogate into the `source_documents` folder – by default, there’s a text of the last US state of the union in there.
  • Run the `ingest.py` script – this can a long time. On this MacBook M1 it was 2 minutes though
  • Run the `privateGPT.py` script and you get a prompt in your Terminal:

I asked it what the US president has to say about the situation in the Ukraine and it gave me a synopsis and where in the document the information is from.

Screenshot of the terminal with a question providing an analysis of the situation in the Ukraine and the answer

Neat!

Share on Mastodon (needs instance)

Share on Twitter

Newsletter

Check out the Dev Digest Newsletter I write every week for WeAreDevelopers. Latest issues:

Dev Digest 146: 🥱 React fatigue 📊 Query anything with SQL 🧠 AI News

Why it may not be needed to learn React, why Deepfake masks will be a big problem and your spirit animal in body fat! 

Dev Digest 147: Free Copilot! Panel: AI and devs! RTO is bad! Pi plays!

Free Copilot! Experts discuss what AI means for devs. Don't trust containers. Mandated RTO means brain drain. And Pi plays Pokemon!

Dev Digest 148: Behind the scenes of Dev Digest & end of the year reports.

In 50 editions of Dev Digest we gave you 2081 resources. Join us in looking back and learn about all the trends this year.

Dev Digest 149: Wordpress break, VW tracking leak, ChatGPT vs Google.

Slowly starting 2025 we look at ChatGPT vs Google, Copilot vs. Cursor and the state of AI crawlers to replace web search…

Dev Digest 150: Shifting manually to AI.

Manual coding is becoming less of a skill. How can we ensure the quality of generated code? Also, unpacking an APK can get you an AI model.

My other work: