Posted on by richay

Ansible – updating proxmox host kernel with LXC shared GPU

Status: ARCHIVED — HISTORICAL NVIDIA WORKAROUND
Last reviewed: 31 August 2026
Originally applied to: Proxmox + shared NVIDIA GPU into LXC using manually installed NVIDIA drivers

Do not copy the old playbook as a current NVIDIA update procedure. It hard-codes an old NVIDIA driver and treats manual driver reinstallation after a kernel update as normal. With a properly configured DKMS installation, the NVIDIA kernel module should normally rebuild for the new kernel automatically.

Is this still relevant?

Mostly no. I’m keeping the page because the problem it was trying to solve still exists, but the automation itself belongs to an older version of my setup.

The original idea was simple: after upgrading the Proxmox kernel, check whether the NVIDIA module loaded. If it didn’t, reinstall the NVIDIA .run driver on the host, reboot, then reinstall the matching NVIDIA userspace driver inside the Plex LXC.

The old playbook was built around a fixed NVIDIA driver:

NVIDIA-Linux-x86_64-535.154.05.run


That’s the first reason not to use it today. More importantly, a kernel update by itself should not normally require reinstalling the whole NVIDIA driver.

What should happen now?

For an NVIDIA driver installed with DKMS:

  1. A new Proxmox kernel is installed.
  2. The matching Proxmox kernel headers are available.
  3. DKMS builds the NVIDIA kernel module for that new kernel.
  4. You reboot into the new kernel.
  5. nvidia-smi continues to work.

NVIDIA’s current documentation says that kernel updates can occasionally leave DKMS without the correct headers or fail to rebuild the module. When that happens, the modern fix is normally to diagnose/rebuild DKMS rather than blindly reinstalling an old driver package.

Useful checks after a Proxmox kernel update

uname -r
nvidia-smi
dkms status


Make sure the headers for the running kernel are installed:

apt update
apt install proxmox-headers-$(uname -r)


Depending on the driver installation method and package naming, you can then rebuild the NVIDIA DKMS module rather than reinstalling the entire driver.

Proxmox 9 note: NVIDIA kernel/module compatibility can still genuinely break on newer Proxmox kernels. There have been current Proxmox 9.2 reports involving kernel 7.x, DKMS build failures and conflicts with newer kernel drivers. So if nvidia-smi dies after an upgrade, don’t assume your LXC passthrough configuration is the problem — check the host NVIDIA module first.

What about the LXC?

For shared NVIDIA GPU access, the LXC uses the NVIDIA kernel driver loaded by the Proxmox host. The container does not build its own kernel module.

The old playbook installed the same NVIDIA package inside the LXC using:

./NVIDIA-Linux-x86_64-535.154.05.run --no-kernel-module


That installs the userspace side without attempting to build a kernel module inside the container.

The important distinction is that a kernel update does not automatically mean those userspace libraries changed. You normally only need to revisit the LXC-side NVIDIA libraries when the actual NVIDIA driver version changes or the userspace/kernel versions no longer match.

Proxmox device passthrough is cleaner now too

Current Proxmox supports native LXC device passthrough using dev[n] entries, including configurable path, UID, GID and mode. That has replaced a lot of the older hand-written cgroup/bind-mount hacks used in historical GPU-sharing guides.

The NVIDIA-specific wrinkle remains: the host still needs a functioning NVIDIA kernel driver before any /dev/nvidia* devices can be shared into the LXC.

If I automated this today

I would automate verification, not automatic driver reinstallation.

For example, an Ansible playbook can check the host after updates and deliberately fail if the NVIDIA stack is broken:

- name: Verify NVIDIA after Proxmox update
  hosts: proxmox
  become: true

  tasks:
    - name: Check running kernel
      ansible.builtin.command: uname -r
      register: running_kernel
      changed_when: false

    - name: Check NVIDIA
      ansible.builtin.command: nvidia-smi
      register: nvidia_smi
      changed_when: false
      failed_when: false

    - name: Check DKMS state
      ansible.builtin.command: dkms status
      register: dkms_status
      changed_when: false
      failed_when: false

    - name: Stop if NVIDIA is broken
      ansible.builtin.fail:
        msg: |
          NVIDIA did not initialise after the kernel update.
          Kernel: {{ running_kernel.stdout }}
          nvidia-smi:
          {{ nvidia_smi.stdout }}
          {{ nvidia_smi.stderr }}

          DKMS:
          {{ dkms_status.stdout }}
      when: nvidia_smi.rc != 0


That gives you the automation benefit without having Ansible silently run an obsolete .run installer against whatever kernel happened to arrive that day.

Original playbook — archived for reference

ARCHIVED CODE — DO NOT USE AS-IS. This is the original playbook from this article. It is preserved so old links and search results still make sense, not because I recommend deploying it in 2026.

########
- hosts: nvidia
  become: true
  become_user: root
  tasks:
    - name: Wait for 10.77.69.2 to become available
      wait_for_connection:
        delay: 5
        timeout: 300

    - name: Check if NVIDIA kernel module is loaded
      shell: lsmod | grep -q '^nvidia'
      register: nvidia_module_check
      ignore_errors: true

    - name: Set NVIDIA module check result as fact
      set_fact:
        nvidia_module_rc: "{{ nvidia_module_check.rc }}"

    - name: Reinstall NVIDIA driver if module is not loaded
      shell: sh /root/NVIDIA-Linux-x86_64-535.154.05.run --silent
      args:
        executable: /bin/bash
      when: nvidia_module_check.rc != 0

    - name: Set fact if NVIDIA driver was installed
      set_fact:
        driver_installed: true
      when: nvidia_module_check.rc != 0

    - name: Reboot system if NVIDIA driver was reinstalled
      reboot:
      when: nvidia_module_check.rc != 0

    - name: Wait for 10.77.69.2 to become available after reboot
      wait_for_connection:
        delay: 10
        timeout: 600
      when: nvidia_module_check.rc != 0

########
- hosts: plex
  become: true
  become_user: root
  tasks:
    - name: Install NVIDIA driver in LXC
      shell: sh /root/NVIDIA-Linux-x86_64-535.154.05.run --no-kernel-module --silent
      args:
        executable: /bin/bash
      when: hostvars['10.77.69.2'].driver_installed | default(false)

    - name: Reboot 10.77.69.103
      reboot:
      when: hostvars['10.77.69.2'].driver_installed | default(false)

    - name: Wait for 10.77.69.103 to become available
      wait_for_connection:
        delay: 10
        timeout: 300
      when: hostvars['10.77.69.2'].driver_installed | default(false)


Why I’m archiving it instead of deleting it

The original problem was real and NVIDIA + new Proxmox kernels can still occasionally go cactus. But the correct troubleshooting path has changed.

Today I’d check:

  • Is the running Proxmox kernel supported by the installed NVIDIA driver?
  • Are the matching Proxmox kernel headers installed?
  • Did DKMS actually build/install the NVIDIA module for the running kernel?
  • Does nvidia-smi work on the host?
  • Do the /dev/nvidia* devices exist?
  • Only then: does the LXC have the right devices and compatible NVIDIA userspace libraries?

That’s a better model than treating every kernel update as “reinstall NVIDIA everywhere and reboot twice”.

References

Leave a Reply

Your email address will not be published. Required fields are marked *