SOLUTION BELOW

The actual bug


I have never been in a more confusing situation regarding Linux.

I have a Dell XPS 15 9560, which had a dual boot Windows 10 / EndeavourOS setup. It was running fine for months. 10 days ago I updated Linux and after restart it couldn’t boot anymore. It got stuck at “A start job is running for /dev/disk/by-uuid/…” (which is the root partition).

First, with the help of a friend of mine who is quite knowledgeable about Linux (he runs vanilla Arch, etc), we spent 5 hours trying to fix it but had no luck.

Then I decided to back up everything and do a fresh install. Aaaand the same error happened again on the first boot. Then I though “ok, probably some problem with Arch, lets try Fedora”. Nope. Some similar error about not finding the root partition. (Here I must say that the kernel which was shipped with the ISO was working fine, but after updating to the latest one, it failed.) Here I thought “ok, then it might be a problem with the latest kernel, let’s install EndeavourOS with the LTS kernel.” Nope, LTS kernel also didn’t boot. Then I tried Ubuntu and it worked, but that’s not solving the problem. Then I decided to put another nvme drive in the laptop and try there. The same error again.

Now the greatest part: If I put the nvme drive into an external usb case, EndeavourOS installs, updates, boots without any problem, no sign of the error.

So now I don’t know how to proceed… Maybe there is something wrong with the pcie port in my laptop, but except for the booting problem, windows is working, I can also mount and access every partition in the ssd through a live usb. So no other signs of problem with the port whatsoever.

I would be grateful for any advice as I’ve lost several days trying to solve this and I am out of ideas…


Solution: The last working kernels are from 11. August 2023 (both linux and linux-lts) linux-6.4.10.arch1-1 and linux-lts-6.1.45-1. You can download them from here: linux / linux-lts and install them with

sudo pacman -U the_path_to_the_package

Thank you all for the help!

  • Illecors@lemmy.cafe
    link
    fedilink
    English
    arrow-up
    12
    arrow-down
    1
    ·
    1 year ago

    What is the output of blkid and the content of /etc/fstab of the installed-but-not-booting system? You will need a live usb to get the those, probably.

    • oiram15@lemmy.sdf.orgOP
      link
      fedilink
      arrow-up
      5
      arrow-down
      1
      ·
      edit-2
      1 year ago
      [root@EndeavourOS /]# blkid
      /dev/nvme0n1p1: UUID="6903-7FA3" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="86a0a714-815f-45db-ad0e-9eba16861903"
      /dev/nvme0n1p2: LABEL="endeavouros" UUID="9ae3c50f-be08-4594-ac30-2d094375868d" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="endeavouros" PARTUUID="9b52162c-a4bf-4e5e-8096-bb7f144c481a"
      /dev/loop0: BLOCK_SIZE="1048576" TYPE="squashfs"
      /dev/sda2: SEC_TYPE="msdos" LABEL_FATBOOT="ARCHISO_EFI" LABEL="ARCHISO_EFI" UUID="6121-B369" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="1bdb3cc0-02"
      /dev/sda1: BLOCK_SIZE="2048" UUID="2023-08-05-13-57-43-00" LABEL="EOS_202308" TYPE="iso9660" PARTUUID="1bdb3cc0-01"
      [root@EndeavourOS /]# sudo cat /etc/fstab 
      # /etc/fstab: static file system information.
      #
      # Use 'blkid' to print the universally unique identifier for a device; this may
      # be used with UUID= as a more robust way to name devices that works even if
      # disks are added and removed. See fstab(5).
      #
      #                      
      UUID=6903-7FA3                            /efi           vfat    defaults,noatime 0 2
      UUID=9ae3c50f-be08-4594-ac30-2d094375868d /              ext4    defaults,noatime 0 1
      tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0
      
      • Illecors@lemmy.cafe
        link
        fedilink
        English
        arrow-up
        5
        arrow-down
        1
        ·
        1 year ago

        Please use tripe backticks to create codeblocks - much easier to read. Like this:

        ```

        some text

        ```

        results in

        some text
        
      • Illecors@lemmy.cafe
        link
        fedilink
        English
        arrow-up
        3
        arrow-down
        1
        ·
        1 year ago

        I’m also curious about the specific UUID it’s looking for. Any chance you could take a photo of the screen and upload it somewhere?

      • Illecors@lemmy.cafe
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        1 year ago
        [root@EndeavourOS /]# blkid
        /dev/nvme0n1p1: UUID=6903-7FA3” BLOCK_SIZE=512TYPE=“vfat” PARTUUID=86a0a714-815f-45db-ad0e-9eba16861903” /dev/nvme0n1p2: LABEL=“endeavouros” UUID=9ae3c50f-be08-4594-ac30-2d094375868d” BLOCK_SIZE=4096TYPE=“ext4” PARTLABEL=“endeavouros” PARTUUID=9b52162c-a4bf-4e5e-8096-bb7f144c481a”
        /dev/loop0: BLOCK_SIZE=1048576TYPE=“squashfs”
        /dev/sda2: SEC_TYPE=“msdos” LABEL_FATBOOT=“ARCHISO_EFI” LABEL=“ARCHISO_EFI” UUID=6121-B369” BLOCK_SIZE=512TYPE=“vfat” PARTUUID=1bdb3cc0-02”
        /dev/sda1: BLOCK_SIZE=2048” UUID=2023-08-05-13-57-43-00” LABEL=“EOS_202308” TYPE=“iso9660” PARTUUID=1bdb3cc0-01
        [root@EndeavourOS /]# sudo cat /etc/fstab
        /etc/fstab: static file system information.
        Use ‘blkid’ to print the universally unique identifier for a device; this may
        be used with UUID= as a more robust way to name devices that works even if
        disks are added and removed. See fstab(5).
        
        UUID=6903-7FA3 /efi vfat defaults,noatime 0 2
        UUID=9ae3c50f-be08-4594-ac30-2d094375868d / ext4 defaults,noatime 0 1
        tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
        

        I assume it should look like this

      • Illecors@lemmy.cafe
        link
        fedilink
        English
        arrow-up
        1
        arrow-down
        1
        ·
        edit-2
        1 year ago

        What are the kernel parameters? cat /proc/cmdline

        EDIT actually that will show the live system config. Assuming you’re using grub, what is the content of /etc/default/grub?

        Or /boot/loader/loaders/something.conf if you’re on systemd-boot

        • oiram15@lemmy.sdf.orgOP
          link
          fedilink
          arrow-up
          1
          arrow-down
          1
          ·
          1 year ago

          I’m on systemd-boot. Where isn’t a directory loaders under loader, but I found the parameters under /etc/kernel/cmdline:

          nvme_load=YES nowatchdog rw root=UUID=9ae3c50f-be08-4594-ac30-2d094375868d

          • Illecors@lemmy.cafe
            link
            fedilink
            English
            arrow-up
            1
            arrow-down
            1
            ·
            1 year ago

            My bad, I think in your case it’s in /efi/loader/entries/something.conf

            Since / is not mounted, yet, bootloader will not be able to read anything under /etc/. Unless it’s used to automatically populate the loader.conf.

            Also check /efi/loader/loader.conf.

            • oiram15@lemmy.sdf.orgOP
              link
              fedilink
              arrow-up
              2
              arrow-down
              1
              ·
              1 year ago

              I found it!

              [[email protected] ~]$ cat /mnt/efi/loader/entries/02ef85f9edc146d598502c1b296ff64a-6.4.12-arch1-1.conf 
              # Boot Loader Specification type#1 entry
              # File created by /etc/kernel/install.d/90-loaderentry.install (systemd 254.1-1-arch)
              title      EndeavourOS
              version    6.4.12-arch1-1
              machine-id 02ef85f9edc146d598502c1b296ff64a
              sort-key   endeavouros-6.4.12-arch1-1
              options    nvme_load=YES nowatchdog rw root=UUID=9ae3c50f-be08-4594-ac30-2d094375868d systemd.machine_id=02ef85f9edc146d598502c1b296ff64a
              linux      /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/linux
              initrd     /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/initrd
              
              
              • Illecors@lemmy.cafe
                link
                fedilink
                English
                arrow-up
                1
                arrow-down
                1
                ·
                1 year ago

                I’ve never used machine-id with systemd-boot, but everything appears to be corrent. Presumably, /boot contains a directory named 6.4.12-arch1-1, which contains files linux and initrd, correct?

                You could try rebuilding the initramfs with mkinitcpio --allpresets while chrooted.

                • oiram15@lemmy.sdf.orgOP
                  link
                  fedilink
                  arrow-up
                  1
                  arrow-down
                  1
                  ·
                  edit-2
                  1 year ago

                  they are under /02ef85f9edc146d598502c1b296ff64a/6.4.12-arch1-1/, but yes.

                  EndeavourOS is using dracut by default.

                  Edit: we tried rebuilding initramfs before, but it didn’t help

                  • Illecors@lemmy.cafe
                    link
                    fedilink
                    English
                    arrow-up
                    1
                    arrow-down
                    1
                    ·
                    1 year ago

                    OK, I see nothing wrong. Let’s try building a new config that’s as minimal as possible. Copy linux and initrd files to /boot/.

                    /efi/loader/entries/test.conf

                    title      Test
                    options    root=/dev/nvme0n1p2
                    linux      /linux
                    initrd     /initrd