Warning: The default setup WILL remove admin priveliges. Best to create a second admin account beforehand. A work around fix is found at the bottom of this article.
Follow the instructions to integrate authentik with Jellyfin here.
Step 9 for role claims needs some extra info if you want to log into Jellyfin with admin access, namely the Admin Roles.

I’ve gone with the default group for Authentik of “authentik Admins”.
You can use a custom group or create one in Authentik by going to:
Admin Interface -> Directory -> Groups -> Create
Name it “jellyfin” and leave all as default. Go to users at the top and add the correct users you want admin access to jellyfin.
Enter “jellyfin” in the Admin Roles in the Jellyfin OIDC plugin.
Enjoy 🙂
Fixing broken admin with the OIDC Admin Roles
Credit to Spyros from the comments
If you have set this up with the official instrcutions and need to still add the ‘Admin Roles’ for authentik, navigate to your data at /jellyfin/data/plugins/configurations/SSO-Auth.xml
Edit the lines at <AdminRoles> to the following
<AdminRoles>
<string>authentik Admins</string>
</AdminRoles>
Or to whatever your custom group is named, then save and restart 🙂
