diff --git a/modules/README.md b/modules/README.md index 202c2e0..f1a0632 100644 --- a/modules/README.md +++ b/modules/README.md @@ -764,29 +764,6 @@ enabled = True repeater_channels = [2, 3] ``` -### Ollama (LLM/AI) Settings -For Ollama to work, the command line `ollama run 'model'` needs to work properly. Ensure you have enough RAM and your GPU is working as expected. The default model for this project is set to `gemma3:270m`. Ollama can be remote [Ollama Server](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server) works on a pi58GB with 40 second or less response time. - -```ini -# Enable ollama LLM see more at https://ollama.com -ollama = True # Ollama model to use (defaults to gemma2:2b) -ollamaModel = gemma3:latest # Ollama model to use (defaults to gemma3:270m) -ollamaHostName = http://localhost:11434 # server instance to use (defaults to local machine install) -``` - -Also see `llm.py` for changing the defaults of: - -```ini -# LLM System Variables -rawQuery = True # if True, the input is sent raw to the LLM if False, it is processed by the meshBotAI template - -# Used in the meshBotAI template (legacy) -llmEnableHistory = True # enable history for the LLM model to use in responses adds to compute time -llmContext_fromGoogle = True # enable context from google search results helps with responses accuracy -googleSearchResults = 3 # number of google search results to include in the context more results = more compute time -``` -Note for LLM in docker with [NVIDIA](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html). Needed for the container with ollama running. - ### Wikipedia Search Settings The Wikipedia search module can use either the online Wikipedia API or a local Kiwix server for offline wiki access. Kiwix is especially useful for mesh networks operating in remote or offline environments. diff --git a/modules/llm.md b/modules/llm.md index 341f442..0799f07 100644 --- a/modules/llm.md +++ b/modules/llm.md @@ -2,6 +2,9 @@ This is not a full turnkey setup yet? +For Ollama to work, the command line `ollama run 'model'` needs to work properly. Ensure you have enough RAM and your GPU is working as expected. The default model for this project is set to `gemma3:270m`. Ollama can be remote [Ollama Server](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server) works on a pi58GB with 40 second or less response time. + + # Ollama local ```bash # bash @@ -20,6 +23,11 @@ Environment="OLLAMA_HOST=0.0.0.0:11434" http://IP::11434 `Ollama is running` +## Docs +Note for LLM in docker with [NVIDIA](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html). Needed for the container with ollama running? + +--- + # OpenWebUI (docker) ```bash ## ollama in docker @@ -32,21 +40,25 @@ wait for engine to build, update the config.ini for the bot ```ini # Use OpenWebUI instead of direct Ollama API (enables advanced RAG features) -useOpenWebUI = True +useOpenWebUI = False # OpenWebUI server URL (e.g., http://localhost:3000) -openWebUIURL = http://IP:3000 +openWebUIURL = http://localhost:3000 +# OpenWebUI API key/token (required when useOpenWebUI is True) +openWebUIAPIKey = sk-xxxx (see below for help) ``` -## validation +## Validation http://IP:3000 make a new admin user. validate you have models imported or that the system is working for query. make a new user for the bot +## API Key +- upper right settings for the user +- settings -> account +- get/create the API key for the user -upper right settings for the user -settings -> account -get/create the API key for the user +## Docs +set api endpoint [OpenWebUI API](https://docs.openwebui.com/getting-started/api-endpoints) - -set api endpoint [OpenWebUI API](https://docs.openwebui.com/getting-started/api-endpoints) \ No newline at end of file +--- \ No newline at end of file