meshing-around/modules/llm.md

How do I use this thing?

This is not a full turnkey setup yet?

For Ollama to work, the command line ollama run 'model' needs to work properly. Ensure you have enough RAM and your GPU is working as expected. The default model for this project is set to gemma3:270m. Ollama can be remote Ollama Server works on a pi58GB with 40 second or less response time.

Ollama local

# bash
curl -fsSL https://ollama.com/install.sh | sh
# docker
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434 open-webui/open-webui

Update /etc/systemd/system/ollama.service

https://github.com/ollama/ollama/issues/703

#service file addition not config.ini
# [Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

validation

http://IP::11434 Ollama is running

Docs

Note for LLM in docker with NVIDIA. Needed for the container with ollama running?

---

OpenWebUI (docker)

## ollama in docker
docker run -d -p 3000:8080 --gpus all -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:cuda

## external ollama
docker run -d -p 3000:8080 -e OLLAMA_BASE_URL=https://IP:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

wait for engine to build, update the config.ini for the bot

# Use OpenWebUI instead of direct Ollama API (enables advanced RAG features)
useOpenWebUI = False
# OpenWebUI server URL (e.g., http://localhost:3000)
openWebUIURL = http://localhost:3000
# OpenWebUI API key/token (required when useOpenWebUI is True)
openWebUIAPIKey = sk-xxxx (see below for help)

Validation

http://IP:3000 make a new admin user. validate you have models imported or that the system is working for query. make a new user for the bot

API Key

• upper right settings for the user
• settings -> account
• get/create the API key for the user

Troubleshooting

• make sure the OpenWebUI works from the bot node and loads (try lynx etc)

• make sure the model in config.ini is also loaded in OpenWebUI and you can use it

• make sure OpenWebUI can reach Ollama IP it should auto import the models

• I find using IP and not common use names like localhost which may not work well with docker etc..

• Check OpenWebUI and Ollama are working

  • Go to Admin Settings within Open WebUI.
  • Connections tab
  • Ollama connection and click on the Manage (wrench icon)
  • download models directly from the Ollama library
  • Once the model is downloaded or imported, it will become available for use within Open WebUI, allowing you to interact with it through the chat interface

Docs

OpenWebUI Quick Start OpenWebUI API OpenWebUI Ollama Blog OpenWebUI on Pi

https://docs.openwebui.com/tutorials/tips/rag-tutorial#tutorial-configuring-rag-with-open-webui-documentation https://docs.openwebui.com/features/plugin/

---