ollama

ollama is how I run large language models on my own hardware. I have it running in an Ubuntu LXC on Proxmox, hosted on my Beelink EQR5.

I’ve specifically allocated 16 GB of RAM to the iGPU on that machine so that ollama can leverage hardware acceleration. It makes a significant difference in performance. I usually interact with it through Open WebUI, which provides a much friendlier interface than the command line for day-to-day use.