Today, installing Linux is no longer a challenge. There are already tutorials available for nearly every distribution. Therefore I won't go into that here.
However, I'd like to make a few points:
After a lot of trial and error, I've come to the conclusion that the newest versions may not always be the most suitable for my (and your?) purposes. This is partly because the application I use to run LLMs, “Oobabooga text-generation-webui”, doesn't work with the latest versions of the underlying AI libraries. For example, Nvidia cards still use CUDA 12.1, which requires the C compiler GCC in version 13.2. However, the latest Linux distributions come with GCC 14, so CUDA doesn't work there.
I haven't explored this further, but Distro-Box might help overcome these problems.
In the end, I found Linux Mint 22 to be the most suitable distribution for me. It doesn't matter whether you choose the Cinnamon or XFCE desktop variant, though XFCE might have slightly lower VRAM usage.
I haven't tried LMDE (Linux Mint Debian Edition) because Debian has a reputation for not being very accommodating with driver installation or 'non-free' codecs.
Because of this choice, all examples use APT as the package manager.
I'm trying to switch to Linux as my daily driver, so I also need programs with which I can make music or practice bass. I have to admit that I had some difficulties with the configuration of JACK, so I tried Xubuntu 24.10. With this, you can install JACK and the required programs via the Ubuntu Studio Installer. There is also CUDA 12.1 as a package with GCC compatibility package. For this, you have to uninstall the bundled SNAP package of Firefox and replace it with the 'normal' DEB package, as only there the hardware support for Nvidia for multimedia codecs can be installed. (Unfortunately, this is not possible with the Flatpak version either).
A few packages I usually install:
sudo apt install -y nala ubuntu-restricted-extras preload build-essential git ffmpeg curl cifs-utils