This is where I share my deep dives into IT projects and recount various random adventures. Consider this a collaborative space – your insights, advice, and helpful hints in the comments are always welcome!
Proxmox Unprivelliged LXC with shared Intel ARC GPU for Plex or Jellyfin transcoding
This is to create Plex and/or Jellyfin using docker within an Unprivilleged LXC container using Proxmox with access to the LXC’s host Intel ARC GPU.
I’m using Debian Trixie LXC for this example. This assumes you have:
- Created the LXC (instructions)
- Installed docker (instructions)
- Active subscription if using Plex
Check the render device group on the host
ls -l /dev/dri/render*
Example output:
crw-rw---- 1 root render 226, 128 Nov 16 21:02 /dev/dri/renderD128
crw-rw---- 1 root render 226, 129 Nov 16 21:02 /dev/dri/renderD129
This is for the iGPU and the ARC GPU, the ARC GPU is renderD129. Both are part of the device GROUP 226 this will be needed later on.
Find the GID of the render group on the host – HOST_GID
getent group render
Example output:
render:x:993:
We also need the render group of the LXC – LXC_GID
getent group render
Example output:
render:x:105:
On the proxmox host, edit the config for the lxc
nano /etc/pve/lxc/<CTID>.conf
Add these lines in, remembering:
GROUP: 226
LXC_GID: 105
HOST_GID: 993
SUBUID_START: 100000 (constant)
BASE_RANGE: 65536 (constant)
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 226:129 rwm
lxc.mount.entry: /dev/dri/renderD129 dev/dri/renderD129 none bind,optional,create=file
lxc.idmap: u 0 100000 105
lxc.idmap: g 0 100000 105
lxc.idmap: u 105 993 1
lxc.idmap: g 105 993 1
lxc.idmap: u 106 100106 65430
lxc.idmap: g 106 100106 65430
This is to share both the iGPU and ARC GPU with 226 being the group.
The idmaps are rather complicated but you should be fine if you follow the rules
lxc.idmap: u 0 SUBUID_START BASE_RANGE = lxc.idmap: u 0 100000 105
lxc.idmap: g 0 SUBUID_START BASE_RANGE = lxc.idmap: g 0 100000 105
lxc.idmap: u LXC_GID HOST_GID 1 = lxc.idmap: u 105 993 1
lxc.idmap: g LXC_GID HOST_GID 1 = lxc.idmap: g 105 993 1
lxc.idmap: u (LXC_GID+1) (SUBUID_START + LXC_GID + 1) (BASE_RANGE - (LXC_GID+1)) = lxc.idmap: u 106 100106 65430
lxc.idmap: g (LXC_GID+1) (SUBUID_START + LXC_GID + 1) (BASE_RANGE - (LXC_GID+1)) = lxc.idmap: g 106 100106 65430
Save and exit the config.
We need to add the HOST_GID to the proxmox host subuid and subgid files
nano /etc/subuid
This should be as follows, then save and close
root:100000:65536
root:993:1
The same for the subgid
nano /etc/subgidroot:100000:65536
root:993:1
Restart the LXC container so the new config is loaded. Nearly there..
Docker Compose
For both Plex and Jellyfin the ARC GPU needs to be passed through into the container, add these lines into the docker compose file
devices:
- /dev/dri:/dev/driPlex
For Plex, go to Settings, Transcoder then scroll down to Hardware transcoding device your ARC should be there. Mine shows Intel DG2 [Arc A310]. If it’s not here, double check the idmaps in the LXC config from the proxmox host
Test the transcoding by playing any video, then in the Play Settings change the quality to something lower, then go to Activity, Dashboard and see if its transcoding with a hardware (hw) tag
NOTE: Hardware transcoding does require a subscription.
Jellyfin
For Jellyfin, top left 3 bars, Adminstration, Dashboard
Then Playback, Transcoding
Under Hardware Acceleration select Intel Quicksync (QSV)
With QSC Device enter /dev/dri/renderD129 for your ARC GPU
(or /dev/dri/renderD128 if you dont have iGPU)
To test play a video, go to the Play Settings, lower the quality, then click Playback Info
Look through the info for Play method Transcoding.
When Jellyfin transcoding is incorrectly setup videos wont play at all, check the idmaps in the LXC config from the proxmox host
Bypassing CGNAT for Plex with Tailscale funnels
Disclaimer: I’m currently on a static IP and run Plex using a lifetime Plex Pass, with remote access enabled. That said, I still understand the frustrations that come with CGNAT setups.
With the lack of IP address for IPv4 and the introduction of ISP CGNAT, this has created chaos with self hosting and using Plex.
Clouflare tunnels is good, but it’s also against their ToS for media streaming and download servers which you could find yourself without access. It’s currently not explictly against Tailscale ToS yet, but it’s not the recommended use for the funnel service.
Tailscale funnels to save the day! (..for now)
Essentially this opens up a service to the tailscale domain (.ts.net) which are public facing, this will ‘funnel’ all traffic into your deisgnated service when accessed.
First you’ll need to create an account with tailscale if you havent already.
Head over to the Admin Console and DNS
Hit the ‘Rename tailnet…” button until you get something cool 😉
My example shows cool-kid.ts.net
Scroll down and make sure MagicDNS and HTTPS Certificates are active
Install Tailscale on your VM or LXC, instructions can be found here.
Alternatively, use this script
curl -fsSL https://tailscale.com/install.sh | sh
If installing on an LXC you will need to add a few extra commands to the config in proxmox.
nano /etc/pve/lxc/<CTID>.conflxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
Restart the LXC when the changes have been made.
For both VM and LXC now activate tailscale with:
tailscale up
Follow the link to authenticate tailscale, this should now show up in your admin console.
While still in VM / LXC shell we use the magic command to allow the funnel!
tailscale funnel --bg http://127.0.0.1:32400
Follow the instructions to accept the Funnel and keep both boxes ticked
Pay attention to the output of the command, it should look something like this
Available on the internet:
https://plex.cool-kid.ts.net/
|-- proxy http://127.0.0.1:32400
Funnel started and running in the background.
To disable the proxy, run: tailscale funnel --https=443 off
Make note of the webaddress as we will need to enter this into plex settings.
Navigate to your plex instance settings http://<YourIP>:32400/web and scroll down to Remote Access and click Disable Remote Access as this wont be needed anymore.
Navigate to Network and scroll down to the bottom, enter in the webaddress tailscale gave you in the output from the funnel command, you must include the :443 port at end for this to work.
https://plex.cool-kid.ts.net:443
Then hit save changes, and restart Plex for good measure.
Now your plex server will be accessible behind CGNAT with Tailscale Funnels.
The only downside – when looking at your dashboard, remote connections will not show on the graph.
If you also want to continue using tailscale for secure access to all your selfhosted apps, they offer a feature called Split DNS by domain. Which when connected to tailscale allows you to route all dns requests for your personal domain like richay.au into your PiHole / Adguard instance to redirect you to your internal reverse proxy. This stops unwanted access publicly to self-hosted services, but allows devices signed into tailscale to have access. My guide can be found here.
For public facing services like this blog, cloudflare tunnels is a great way to get around CGNAT.
Enjoy 🙂
Browser VSCode docker image used to manage docker host with addon
This is a docker-compose.yaml for VS Code Server that allows host docker control in your browser window.
This is done by sharing the docker sock volume, docker-compose host folder and docker folder if you have it (for quick edits of config files) and using an environment variable to install the docker extension.
The docker-compose.yaml is from LinuxServer.io
---
services:
code-server:
image: lscr.io/linuxserver/code-server:latest
container_name: code-server
environment:
- PUID=1000
- PGID=1000
- TZ=Etc/UTC
- PASSWORD=password #optional
- HASHED_PASSWORD= #optional
- SUDO_PASSWORD=password #optional
- SUDO_PASSWORD_HASH= #optional
- PROXY_DOMAIN=code-server.my.domain #optional
- DEFAULT_WORKSPACE=/config/workspace #optional
- PWA_APPNAME=code-server #optional
volumes:
- /path/to/code-server/config:/config
ports:
- 8443:8443
restart: unless-stopped
Create passwords for security, if you use your own auth and reverse proxy you can add Traefik and Authentik labels.
Note: If you use your own auth, just remove the password options.
My host docker-compose folder is found at – /home/richay/docker-compose
My host docker folder is found at – /home/richay/docker
The environment variable needed is – DOCKER_MODS=linuxserver/mods:universal-docker
Adding these in my new docker-compose will look like this
---
services:
code-server:
image: lscr.io/linuxserver/code-server:latest
container_name: code-server
environment:
- PUID=1000
- PGID=1000
- TZ=Australia/Perth
- DEFAULT_WORKSPACE=/config/workspace
- PWA_APPNAME=code-server
- DOCKER_MODS=linuxserver/mods:universal-docker
volumes:
- /home/richay/docker/code:/config
- /home/richay/docker-compose:/config/workspace/docker-compose
- /home/richay/docker:/config/workspace/docker
- /var/run/docker.sock:/var/run/docker.sock
# ports:
# - 8443:8443
restart: unless-stopped
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.routers.code-secure.entrypoints=https"
- "traefik.http.routers.code-secure.rule=Host(`code-server.richay.au`)"
- "traefik.http.routers.code-secure.tls=true"
- "traefik.http.routers.code-secure.tls.certresolver=cloudflare"
- "traefik.http.routers.code-secure.service=code"
- "traefik.http.services.code.loadbalancer.server.port=8443"
- "traefik.http.routers.code-secure.middlewares=authentik@file"
networks:
- proxy
networks:
proxy:
external: true
Note: Ports commented out as not needed with reverse proxy
If you dont have a reverse proxy and auth, then yours will look something like this
---
services:
code-server:
image: lscr.io/linuxserver/code-server:latest
container_name: code-server
environment:
- PUID=1000
- PGID=1000
- TZ=Etc/UTC
- PASSWORD=password #optional
- HASHED_PASSWORD= #optional
- SUDO_PASSWORD=password #optional
- SUDO_PASSWORD_HASH= #optional
- PROXY_DOMAIN=code-server.my.domain #optional
- DEFAULT_WORKSPACE=/config/workspace #optional
- PWA_APPNAME=code-server #optional
- DOCKER_MODS=linuxserver/mods:universal-docker
volumes:
- /path/to/code-server/config:/config
- /path/to/docker-compose:/config/workspace/docker-compose
- /path/to/docker:/config/workspace/docker
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 8443:8443
restart: unless-stopped
Head to either http://hostip:8443 if using standard or https://code-server.richay.au if using reverse proxy and enjoy direct access to your docker-compose files with the power of VS Code 🙂
Tailscale split dns by domain for secure home server access
This is how you can still have public facing web apps but still maintain private web apps while you’re connected to Tailscale.
My setup:
Domain is richay.au
Adguard (10.10.10.10) as my home network DNS / ad blocker – All webapps have a DNS cname record for *.richay.au to rewrite to my Traefik IP (so internal connections stay internal)
Traefik v3 (10.10.10.20) as my reverse proxy – All webapps are published here under subdomains for richay.au
Cloudflare tunnels to get around CGNAT for public facing web apps – Only public facing apps such as my blog are listed in my tunnels, and the private IP set to Traefik (10.10.10.20)
Proxmox for my hypervisor for Linux Debian VM’s.
Tailscale to make the magic happen.
The first step is to install Tailscale (tailscale.com) on either the Traefik host machine or the DNS host machine (or both). If only installing once you can use advertise a route to the other VM within Tailscale, this is the option I am choosing and install on my Traefik host of 10.10.10.20 – then advertising a route to my Adguard DNS of 10.10.10.10 this is done using this command once Tailscale has been setup:
sudo tailscale set --advertise-routes=10.10.10.10/32
You will need to go to the Tailscale admin console and locate the Traefik entry then click the 3 circles to open the menu.
Click on “Edit Route Settings”.
Confirm 10.10.10.10/32 and accept the new route to activate, now Traefik can talk to your DNS server through Tailscale.
The second step is to allow Tailscale to use IP forwarding in Linux, these instructions can be found here.
After these have been configured you will need to return to the Tailscale admin console to configure the Tailscale DNS settings.
Under nameservers click Add nameserver.
In the pop up enter your home Adguard DNS 10.10.10.10, check the box for split DNS and enter your domain.
Hit save, and now you should have an active split dns while connected to tailscale.
Add your mobile device to tailscale and you will be able to access secure web apps from your phone wherever you are.
Portainer and Authentik / Traefik failling to login due to UFW
I use Authentik with Traefik. This is also assuming you followed the official Authentik intergration guide found here.
This is the screen that I had when logging into Authentik:
On the right there is a 404 error when inspecting the console, which leads to the 401 error – unauthorized. After lots of trial and error, turns out it was due to Uncomplicated Firewall (UFW) blocking the port access of 443 (https).
They are both on the same docker network, I also use labels for traefik. I tried using my config setup in traefik with no luck either. Disabling UFW to test the login process.
sudo ufw disable
This worked! So all that was left was to re-enable authentik and add in the two rules for port 80 (http) and port 443 (https) then reload UFW to enforce the changes
sudo ufw enable
sudo ufw allow 80/tcp comment 'Allow HTTP for Traefik'
sudo ufw allow 443/tcp comment 'Allow HTTPS for Traefik'
sudo ufw reloadJellyfin and Authentik OIDC with admin
Warning: The default setup WILL remove admin priveliges. Best to create a second admin account beforehand. A work around fix is found at the bottom of this article.
Follow the instructions to integrate authentik with Jellyfin here.
Step 9 for role claims needs some extra info if you want to log into Jellyfin with admin access, namely the Admin Roles.
I’ve gone with the default group for Authentik of “authentik Admins”.
You can use a custom group or create one in Authentik by going to:
Admin Interface -> Directory -> Groups -> Create
Name it “jellyfin” and leave all as default. Go to users at the top and add the correct users you want admin access to jellyfin.
Enter “jellyfin” in the Admin Roles in the Jellyfin OIDC plugin.
Enjoy 🙂
Fixing broken admin with the OIDC Admin Roles
Credit to Spyros from the comments
If you have set this up with the official instrcutions and need to still add the ‘Admin Roles’ for authentik, navigate to your data at /jellyfin/data/plugins/configurations/SSO-Auth.xml
Edit the lines at <AdminRoles> to the following
<AdminRoles>
<string>authentik Admins</string>
</AdminRoles>
Or to whatever your custom group is named, then save and restart 🙂
Proxmox cluster with Traefik
With Traefik you can load balance between the different nodes of your cluster with one web page.
ie, https://proxmox.richay.au could access either one of the three nodes I have available, but it comes with one issue, you wont be able to use the shell as the websocket it uses needs to be upgraded in Traefik.
This is my dynamic config.yaml in traefik
http:
routers:
proxmox:
entryPoints:
- "https"
rule: "Host(`proxmox.richay.au`)"
middlewares:
- https-redirectscheme
- websocket-upgrade
tls: {}
service: proxmox
services:
proxmox:
loadBalancer:
sticky:
cookie: {}
servers:
- url: "https://10.10.10.1:8006" # IP of node 1
- url: "https://10.10.10.2:8006" # IP of node 2
- url: "https://10.10.10.3:8006" # IP of node 3
passHostHeader: true
middlewares:
https-redirectscheme:
redirectScheme:
scheme: https
permanent: true
websocket-upgrade:
headers:
CustomRequestHeaders:
Upgrade: websocket
Connection: Upgrade
This will allow the proxmox shells to work 🙂
Authentik and Home Assistant with working Companion App solution using Traefik proxy
I absolutely love the fact I can log into authentik and then bypass all app login screens for instant access. This was a slight challenge with Home Assistant as placing this behind Authentik behind this would break the phone companion app. Here is a quick guide on how I overcame this.
This was solved by setting two pathways leading back to my home assistant in traefik, one being behind authentik for the SSO, and the other using the default home assistant login screen for the companion app.
This requires a few things first:
Traefik setup with dynamic config
HTTP Header Authentication integration for home assistant found here
Authentik setup with provider and application, instructions found here
The added lines I used in configuration.yaml in Home Assistant
http:
use_x_forwarded_for: true
trusted_proxies:
- 10.10.10.0/24 # Change to your home network
- 172.22.0.0/16 # Change to your traefik proxy network
auth_header:
username_header: X-authentik-username
The config.yaml for my traefik
Please not there is two routers for the one service, one being behind Authentik (/auth) and the other bypassing Authentik. There is also a path prefix for the outpost so there is no 404 error.
http:
routers:
# Router for SSO - PROTECTED by Authentik
home-assistant:
entryPoints:
- "https"
rule: "Host(`home-assistant.richay.au`) && PathPrefix(`/auth`) && !PathPrefix(`/auth/token`) || PathPrefix(`/outpost.goauthentik.io`))" # Change host domain
middlewares:
- https-redirectscheme
- authentik
tls: {}
service: home-assistant
# Router for Companion App - UNPROTECTED by Authentik
home-companion:
entryPoints:
- "https"
rule: "Host(`home-companion.richay.au`) || Host(`home-assistant.richay.au`)" # Change host domain
middlewares:
- https-redirectscheme
tls: {}
service: home-assistant
services:
home-assistant:
loadBalancer:
servers:
- url: "http://10.10.10.10:8123" # change this to your IP of your Home Assistant
passHostHeader: true
middlewares:
authentik:
forwardAuth:
address: "http://authentik-server:9000/outpost.goauthentik.io/auth/traefik"
trustForwardHeader: true
authResponseHeaders:
- X-authentik-username
- X-authentik-groups
- X-authentik-email
- X-authentik-name
- X-authentik-uid
- X-authentik-jwt
- X-authentik-meta-jwks
- X-authentik-meta-outpost
- X-authentik-meta-provider
- X-authentik-meta-app
- X-authentik-meta-version
- authorization
https-redirectscheme:
redirectScheme:
scheme: https
permanent: true
When you select your server on the companion app, choose the
https://home-companion.richay.au option. This will allow the app to work.
When using a web browser, use https://home-assistant.richay.au for SSO access using Authentik.
Ansible – updating proxmox host kernel with LXC shared GPU
This is to automate the updating of proxmox host when there is a kernel update which will break the LXC link to the GPU.
It just requires you to reinstall the graphics driver and do a reboot otherwise.
This is after you have done an update / upgrade of your proxmox host.
You will have to change the IP Addresses for your setup.
For me..
10.77.69.2 – Proxmox Host
10.77.69.103 – LXC Plex
########
- hosts: nvidia
become: true
become_user: root
tasks:
- name: Wait for 10.77.69.2 to become available
wait_for_connection:
delay: 5
timeout: 300
- name: Check if NVIDIA kernel module is loaded
shell: lsmod | grep -q '^nvidia'
register: nvidia_module_check
ignore_errors: true
- name: Set NVIDIA module check result as fact
set_fact:
nvidia_module_rc: "{{ nvidia_module_check.rc }}"
- name: Reinstall NVIDIA driver if module is not loaded
shell: sh /root/NVIDIA-Linux-x86_64-535.154.05.run --silent
args:
executable: /bin/bash
when: nvidia_module_check.rc != 0
- name: Set fact if NVIDIA driver was installed
set_fact:
driver_installed: true
when: nvidia_module_check.rc != 0
- name: Reboot system if NVIDIA driver was reinstalled
reboot:
when: nvidia_module_check.rc != 0
- name: Wait for 10.77.69.2 to become available after reboot
wait_for_connection:
delay: 10
timeout: 600
when: nvidia_module_check.rc != 0
########
- hosts: plex
become: true
become_user: root
tasks:
- name: Install NVIDIA driver in LXC
shell: sh /root/NVIDIA-Linux-x86_64-535.154.05.run --no-kernel-module --silent
args:
executable: /bin/bash
when: hostvars['10.77.69.2'].driver_installed | default(false)
- name: Reboot 10.77.69.103
reboot:
when: hostvars['10.77.69.2'].driver_installed | default(false)
- name: Wait for 10.77.69.103 to become available
wait_for_connection:
delay: 10
timeout: 300
when: hostvars['10.77.69.2'].driver_installed | default(false)
This will check to see if the kernels for nvidia (my gpu) has been loaded, if not it will reinstall in silent mode. This will also flag a GPU install in ansible to also reinstall the GPU driver in the LXC, only if it needs though.
Nvidia GPU passthrough in LXC
Link to original article here.
1. Install host drivers
When doing PCIe passthrough, the first step is to blacklist the driver to ensure the host kernel doesn’t try to load the device. Here we need to do the opposite: Install and configure the correct drivers.
You’ll need to install the actual nvidia drivers. The easiest way to do this is to download the driver from nvidia.com. This not only ensures you’re using the latest driver, but means it won’t accidentally update during a system update, as it’s important that the host and guest OS have the exact same driver version. You can still install it using the system package manager, just be aware of updates – especially if the guest and host OS are different distributions.
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/450.80.02/NVIDIA-Linux-x86_64-450.80.02.runsh NVIDIA*
Next you’ll need to make sure the drivers are loaded on boot. To do this, edit the add the following to this file:
nano /etc/modules-load.d/modules.conf# Nvidia modules
nvidia
nvidia_uvm
Once that’s done, you’ll need to update the initramfs with:
update-initramfs -u -k all
The final step is to add a udev rule to create the required device files for the nvidia driver, which for reasons aren’t created automatically. This is done in:
nano /etc/udev/rules.d/70-nvidia.rulesKERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"
Now you can reboot, and check the GPU is being detected correctly with:
nvidia-smi
The output should look like:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 760 Off | 00000000:03:00.0 N/A | N/A |
| 0% 34C P0 N/A / N/A | 0MiB / 1996MiB | N/A Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
There’s my GPU being detected correctly, using driver version 450.80.02 – we’ll be needing this later.
2. Configure container
Next, create your container. There’s nothing special about this process, just choose the OS and resource requirements for you.
Before starting your container, we need to make some changes to the config file directly to pass through the GPU. This config file will probably live in:
nano /etc/pve/lxc/<id>.conf # ID of the LXC
Where id is the id of your container. You need to add the following lines:
# Allow cgroup access
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 243:* rwm
# Pass through device files
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
These lines allow the container to communicate with the nvidia driver, and pass through the control files needed for the guest to actually communicate with the GPU. These lines probably won’t work out the box, so we need to compare them to our actual control files:
ls -l /dev/nvidia*
The output should look like:
crw-rw-rw- 1 root root 195, 254 Dec 22 20:51 /dev/nvidia-modeset
crw-rw-rw- 1 root root 243, 0 Dec 22 20:51 /dev/nvidia-uvm
crw-rw-rw- 1 root root 243, 1 Dec 22 20:51 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195, 0 Dec 22 20:51 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Dec 22 20:51 /dev/nvidiactl
Note: If you don’t see all 5 files, it probably means the drivers haven’t loaded correctly. Best check the logs.
These files are character devices (as shown by the c at the start of the line), which the kernel module uses to communicate with the hardware. lxc.mount.entry binds these into the container.
The lxc.cgroup2.devices.allow lines denote the cgroups which own the nvidia drivers. For the some files we have, 195:* will match the groups owning those, and the uvm files will match 243:*. If the config doesn’t match, you’ll need to change it. Note that the order doesn’t matter, so long as the cgroup lines are before the mounts.
3. Install guest drivers
Now that the host is configured, and the control files passed through, the guest needs configuring.
The gist of the configuration is to also install the nvidia drivers, but without the kernel modules. The simplest way to do this is to use the same driver binary downloaded from nvidia.com, and run it with:
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/450.80.02/NVIDIA-Linux-x86_64-450.80.02.runsh NVIDIA* --no-kernel-module
4. Test it
Now, from your container, you should be able to run nvidia-smi, and it’ll show the right version GPU and driver:
nvidia-smi+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 760 Off | 00000000:03:00.0 N/A | N/A |
| 0% 34C P0 N/A / N/A | 0MiB / 1996MiB | N/A Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
This shows the GPU is detected correctly, but doesn’t prove it’s working correctly. The best way to do this is to actually try and use it. For me this involved installing Jellyfin, loading in some content and checking the GPU was doing the transcoding, not the CPU – Which it was!
Because it’s simply passing through the device files rather than the actual PCIe device, you can repeat this process multiple times for multiple containers.