Christian Heilmann

PrivateGPT – Running “ChatGPT” offline on local documents

Saturday, May 27th, 2023 at 4:04 pm

You can download the GPT model and interrogate local files using Python without having to send any data to the cloud.

PrivateGPT is a python script to interrogate local files using GPT4ALL, an open source large language model. It is pretty straight forward to set up:

  • Clone the repo
  • Download the LLM – about 10GB – and place it in a new folder called `models`.
  • Place the documents you want to interrogate into the `source_documents` folder – by default, there’s a text of the last US state of the union in there.
  • Run the `ingest.py` script – this can a long time. On this MacBook M1 it was 2 minutes though
  • Run the `privateGPT.py` script and you get a prompt in your Terminal:

I asked it what the US president has to say about the situation in the Ukraine and it gave me a synopsis and where in the document the information is from.

Screenshot of the terminal with a question providing an analysis of the situation in the Ukraine and the answer

Neat!

Share on Mastodon (needs instance)

Share on BlueSky

Newsletter

Check out the Dev Digest Newsletter I write every week for WeAreDevelopers. Latest issues:

160: Graphs and RAGs explained and VS Code extension hacks Graphs and RAG explained, how AI is reshaping UI and work, how to efficiently use Cursor, VS Code extensions security issues.
159: AI pipelines, 10x faster TypeScript, How to interview How to use LLMs to help you write code and how much electricity does that use? Is your API secure? 10x faster TypeScript thanks to Go!
158: 🕹️ Super Mario AI 🔑 API keys in LLMs 🤙🏾 Vibe Coding Why is AI playing Super Mario? How is hallucinating the least of our worries and what are rules for developing Safety Critical Code?
157: CUDA in Python, Gemini Code Assist and back-dooring LLMs We met with a CUDA expert from NVIDIA about the future of hardware, we look at how AI fails and how to play pong on 140 browser tabs.
156: Enterprise dead, all about Bluesky and React moves on! Learn about Bluesky as a platform, how to build a React App and how to speed up SQL. And play an impossible game in the browser.

My other work: