PrivateGPT: Local LLMs to Chat with your Docs

Why PrivateGPT?

PrivateGPT provides an API (a tool for computer programs) that has everything you need to create AI applications that understand context and keep things private.

It’s like a set of building blocks for AI. This API is designed to work just like the OpenAI API, but it has some extra features.

So, if you’re already using the OpenAI API in your software, you can switch to the PrivateGPT API without changing your code, and it won’t cost you any extra money.

You can use PrivateGPT with CPU only. Forget about expensive GPU’s if you dont have one.

The PrivateGPT Project

• PrivateGPT project
• PrivateGPT Source Code at Github
  • License: Apache 2.0 ✅

How to use PrivateGPT?

The documentation of PrivateGPT is great and they guide you to setup all dependencies.

You just need Python installed in your computer and then install the required packages.

This kind of projects normally have very specific set of packages dependencies (specific versions and so on), that is why it is recommended to use Python Package Management.

I have tried those with some other project and they worked for me 90% of the time, probably the other 10% was me doing something wrong.

To make sure that the steps are perfectly replicable for anyone, I bring you a guide with PrivateGPT & Docker to contain all the Dependencies (and make it work 100% of the times).

• You will be able to use the Docker image with PrivateGPT I created - Quick Setup
• You can also learn how to build your own Docker image to run PrivateGPT locally - Recommended

PrivateGPT with Docker

With this approach, you will need just one thing: get Docker installed

Then, use the following Stack to deploy it:

• Using PrivateGPT with Docker 🐳 - PreBuilt Image

  Before moving on - Remember to always check the source of the Docker Images you run. Consider building your own.

  version: '3'
  
  services:
    ai-privategpt:
      image: fossengineer/privategpt  # Replace with your image name and tag
      container_name: privategpt
      ports:
        - "8001:8001"
      volumes:
        - ai-privategpt:/app
      command: /bin/bash -c "poetry run python scripts/setup && tail -f /dev/null" #make run
      
  volumes:
    ai-privategpt:
  

That’s it, now get your favourite LLM model ready and start using it with the UI at: localhost:8001

Remember that you can use CPU mode only if you dont have a GPU (It happens to me as well).

Just remember to use models compatible with llama.cpp, as the project suggests.

PrivateGPT API

PrivateGPT API is OpenAI API (ChatGPT) compatible, this means that you can use it with other projects that require such API to work.

How to Build your PrivateGPT Docker Image

The best way (and secure) to SelfHost PrivateGPT. Build your own Image.

You will need the Dockerfile.

• Private GPT to Docker with This Dockerfile ✌️

  # Use the specified Python base image
  FROM python:3.11-slim
  
  # Set the working directory in the container
  WORKDIR /app
  
  # Install necessary packages
  RUN apt-get update && apt-get install -y \
      git \
      build-essential
  
  # Clone the private repository
  RUN git clone https://github.com/imartinez/privateGPT
  
  WORKDIR /app/privateGPT
  
  # Install poetry
  RUN pip install poetry
  
  # Copy the project files into the container
  COPY . /app
  
  #Adding openai pre v1 to avoid error
  RUN sed -i '/\[tool\.poetry\.dependencies\]/a openai="1.1.0"' pyproject.toml
  
  # Lock and install dependencies using poetry
  RUN poetry lock
  RUN poetry install --with ui,local
  
  # Run setup script
  #RUN poetry run python scripts/setup # this scripts download the models (the embedding and LLM models)
  
  # Keep the container running
  #CMD ["tail", "-f", "/dev/null"]
  

The setup script will download these 2 models by default:

• LLM: conversational model
  • LLama 7B.
• Embedding: the model that converts our documents to a vector DB
  • Mistral 7B Instruct.

Use GGUF format for the models and it will be fine (llama.cpp related)

And then build your Docker image to run PrivateGPT with:

docker build -t privategpt .
#podman build -t privategpt .

#docker tag privategpt docker.io/fossengineer/privategpt:v1 #example I used
#docker push docker.io/fossengineer/privategpt:v1

docker-compose up -d #to spin the container up with CLI

Using your PrivateGPT Docker Image

You will need Docker installed and use the Docker-Compose Stack below.

If you are not very familiar with Docker, don’t be scared and install Portainer to deploy the container with GUI.

• Then, you just need this Docker-Compose to deploy PrivateGPT

  version: '3'
  
  services:
    ai-privategpt:
      image: privategpt  # Replace with your image name and tag
      container_name: privategpt2
      ports:
        - "8002:8001"
      volumes:
        - ai-privategpt:/app
      # environment:
      #   - SOME_ENV_VAR=value  # Set any environment variables if needed
      #command: tail -f /dev/null
      # environment:
      #   - PGPT_PROFILES=local
      command: /bin/bash -c "poetry run python scripts/setup && tail -f /dev/null" #make run
      
  volumes:
    ai-privategpt:
  

When the server is started it will print a log Application startup complete.

Execute the comand make run in the container:

docker exec -it privategpt make run

Navigate to http://localhost:8002 to use the Gradio UI or to http://localhost:8002/docs (API section) to try the API using Swagger UI.

FAQ

Other F/OSS Alternatives to have Local LLMs

• The Text Generation Web UI Project, that you can also SelfHost with Docker.

A Detailed Video - How to use PrivateGPT with Docker

What is happening inside PrivateGPT?

These are the guts of our PrivateGPT beast.

The Embedding Model will create the vectorDB records of our documents and then, the LLM will provide the replies for us.

• Embedding Model - BAAI/bge-small-en-v1.5
• Conversational Model (LLM) - TheBloke/Mistral 7B
• VectorDBs - PrivateGPT uses QDrant (F/OSS ✅)
• RAG Framework - PrivateGPT uses LLamaIndex (yeap, also F/OSS ✅)

You can check and tweak this default options with the settings.yaml file.

Python for PrivateGPT

What are Gradio Apps?

Gradio is an open-source Python library that simplifies the development of interactive machine learning (ML) and natural language processing (NLP) applications.

You can also run Gradio-Lite (JS) - Serverless Gradio running in the browser. Similarly we can do with Transformers.JS

Python Dependencies 101

When we are sharing software, we need to make sure that our PCs have the same libraries installed.