This morning I woke up, rebooted a living room pc and got thrown into a 2-hour session of troubleshooting for a problem I do not understand why it exists. I’m writing this in hopes of understanding the whys, and how to avoid similar pitfalls.

I just recently installed a living room pc running Fedora 44, it’s running Plasma Big Screen, and it’s purpose is to be a steam link machine, jellyfin server and maybe a game server down the line for some coop games (zomboid, valheim…). For about a week, everything was perfect.

Until this morning.

After turning on my tv, my system was showing some errors on qbittorrent, and I decided to reboot just in case. And that was when my system just completely locked me out, it threw me into emergency mode and I had no access to root, so nothing could be done, just watch an endless loop of my system trying to do something that was impossible and occasionally pressing enter to restart the loop. That is my first gripe: why throw someone into emergency mode if it’s just going to lock them out?

I tried restarting a few times, unplugging things, reseating ram and the likes just in case. When nothing worked, it seemed I’d have to do research, in the hopes of not having to wipe it clean and start anew.

So, here I went, searching the web with my problem and trying to find a solution. After reading some very long forum posts, I apparently needed more information about what had actually caused this, but it was likely something about fstab. And here is my second gripe: why did the system not immediately inform me of the error first before starting emergency mode? I got 0 error messages because the default setting is Rghb quiet… Is this a thing about just fedora or is every linux distro the same?

(I’m going to add in here that I’m in the process of switching all my pcs to Linux, and this was a first test. But I also am going to switch my family’s pcs, and I need to shine my Linux shoes and put some big boy IT pants for the future, so that’s why I’m writing a post: to learn from your experiences)

So here I go, to do some stuff with GRUB to find the error. I decided to test chatgpt and see if it could guide me (I’m a noob if it wasn’t obvious yet), and took more than an hour of troubleshooting with grub and bash to finally see that the problem was about a drive with an UUID that did not match my system drive (a silver lining I guess). But, here’s the thing, as soon as the reboot loop started, I had an inkling of a suspicion that it might have been one of the old spinning hdds causing it (I need to replace those, but they’re fine and working for now, and in this economy…). So I had unplugged all of them when I did my hardware troubleshooting step, and kept only my nvme disk (which is brand new) on the system up to here. So I had been completely blindsided that even if the drives are disconnected, my system still won’t boot, because it expects the drives to be there, and if they aren’t, even though everything else is working fine, it won’t boot! This is my third gripe. Is this a default setting? Something about Fedora? Why is this the way it is done? It just doesn’t seem logical to me to lock me out of the whole system because a non-essential part is not working/present.

Anyways, after unplugging and re-plugging the drives, I finally discovered it was not my drive, but a pcie sata expansion card that had timed out, and it was this one smaller drive I had been using with the card that was the problem, but after plugging it straight to the mb (the slots are precious, okay? I was saving them for bigger drives in the future), it worked just fine. My system booted normally.

That was 2 hours-ish that could have been just 5 minutes if the system had actually told me it was having problems with connecting to a drive. Also, chatgpt did help, but boy, it didn’t have a good troubleshooting order at all. It was just shooting in every direction and hoping something would stick. But I don’t think trying to find my fix in forum posts would have been any better.

  • FauxLiving@lemmy.world
    15·
    25 days ago

    That was 2 hours-ish that could have been just 5 minutes if the system had actually told me it was having problems with connecting to a drive.

    It did, it almost certainly wrote the error into the system log. That’s generally the first thing you should check if you’re having such low-level problems, like a failure to boot.

    Going forward some good advice is: Get a USB stick, install Ventoy and put a system rescue image on it: https://www.system-rescue.org/

    This way, if you’re ever having issues booting you can boot into a live environment and read the logs.

    If you mount your system drive on /sysroot, you can read the system log with

    journalctl --root=/sysroot

    Add a -b flag to see just the log from the latest boot (and -b -1 from the previous, -b -2 from the one before, etc)

    • alphabethunter@lemmy.worldOP
      5·
      25 days ago

      Thanks! I’ll do that today. I did read a lot of the journal (that was how I eventually figured out the problem), but getting to the journal was already a process. What I don’t understand is why super critical errors are hidden away in a log instead of being shoved in my face once they happen.

      • FauxLiving@lemmy.world
        4·
        24 days ago

        What I don’t understand is why super critical errors are hidden away in a log instead of being shoved in my face once they happen.

        That’s a choice you make when you’re setting up your bootloader, you pass a kernel parameter that tells the kernel what to show on the screen.

        https://docs.kernel.org/admin-guide/kernel-parameters.html

        If you’re using grub, look in /etc/default/grub. There will be a line: GRUB_CMDLINE_LINUX_DEFAULT=“<kernel parameters>” you can set the log level to something (like loglevel=3) and it will write the errors to the console on boot. The higher the number the more information is written to the console.

      • nyan@sh.itjust.works
        3·
        24 days ago

        Because even a headless server with no email capability can write to a log, as long as it can mount its root drive.

        That being said, if your system is hiding stuff behind some kind of splash screen at boot time, turn it off. I suspect your error would have been right there on screen in plain white-on-black text if it had happened on one of my systems (granted, I use OpenRC and not systemd, but I expect the latter also provides a running commentary on what it’s doing at boot until the graphics stack loads).

  • atzanteol@sh.itjust.worksEnglish
    10·
    25 days ago

    It just doesn’t seem logical to me to lock me out of the whole system because a non-essential part is not working/present.

    It’s a fair criticism - but the system doesn’t know they are non-critical. You’ve, effectively, told it they are critical in fact.

    You can add a nofail option to the drives that aren’t needed to boot in /etc/fstab.

    e.g.

    UUID=9d1f7f1b-2a78-4b92-a8d5-0b05e7273920 /mnt/backup  btrfs   defaults,nofail,subvol=@backups 0 0

    I use this for a disk that’s external and sometimes not connected.

  • nyan@sh.itjust.works
    4·
    25 days ago

    noauto, which means that the filesystem in question won’t mount until you issue an explicit mount command for it, can be an alternative to nofail in fstab. Back in the days of optical drives, that used to be one of the options you put on them.

    For external (and network) drives, though, I find it’s better to hand the problem over to autofs (which will mount the filesystem only when you try to access it) and keep them out of fstab.

  • eldavi@lemmy.mlEnglish
    2·
    25 days ago

    I need to replace those, but they’re fine and working for now, and in this economy…

    that’s why i haven’t bought any parts to rebuilt my everything home server. lol

    and the more “exotic” your setup is – and use the word exotic very loosely here – the more time you have to spend thinking of how exotic it is compared to the plain-jane systems exist out there and compensate for it.

    for example: most computers have a permanent drives, so if any of your drives are not permanent; your system automatically becomes exotic.

    not using hdmi, vga, display port or hdmi; youi’re also exotic.

    is your keyboard not the normal 104 key; you’re also exotic.

    using pata & sata together; you’re also exotic.

    …

    • alphabethunter@lemmy.worldOP
      1·
      25 days ago

      That’s also a trip. My keyboard is not the standard 104 american layout, because I’m not American, and while using GRUB I had to discover what keys actually stood in for symbols like / and | because it had reverted back to standard layout.

  • CypherColt@sh.itjust.works
    1·
    24 days ago

    I updated my BIOS a few days ago, and it reset my bios settings. I have my two m.2 slots populated with drives. After the bios reset my fedora based system (Nobara Linux) wouldn’t boot. I had to use journalctl to get logs. The noteworthy error basically said it didn’t see one of drives anymore that was in fstab. The solution took me a bit to find, but basically one of the m.2 slots by default on this Asus motherboard is disabled to free up a sata slot or something. Once I re-enabled it and rebooted everything worked again.

    Might not be the same issue you had but definitely look at the journalctl logs. I did pop the error messages into Gemini to help me understand it, but a web search will probably work if you don’t like ai responses.

  • warmaster@lemmy.world
    32·
    25 days ago

    If you enjoy the experience of learning how to fix stuff and how Linux works, stay with classic distros like the Fedora you are rocking. If you don’t give a shit and just want it to work, I advice you to switch to an immutable/atomic distro. If Gaming is important, I suggest Bazzite.

    Otherwise, provide logs and we’ll help you out.

    • atzanteol@sh.itjust.worksEnglish
      2·
      25 days ago

      just want it to work,

      Unless you’re doing software development and using lots of tools that need dbus or ever plan to do anything other than “use a browser and flatpaks”. Then you get to figure out how to fight with distrobox to make things interoperate with your desktop environment and other distroboxes.

      And I doubt bazzite would have dealt as well with a missing disk.

      • warmaster@lemmy.world
        2·
        25 days ago

        So, you disagree in that immutable distros are easier for the common folk? Steam OS should be mutable?

        • atzanteol@sh.itjust.worksEnglish
          21·
          24 days ago

          I agree that immutable distro proponents are obnoxious.

          Immutable distros wouldn’t have helped in this case.

  • whatiswrongwithyou@lemmy.ml
    1·
    25 days ago

    Here’s how you could have figured it out yourself:

    Boot something else, read the logs.

    Use a usb to boot some other operating system (or the same one) and access the logs of your system which will show where it’s failing and how.

    Another person put that you could add nofail to fstab, you could also do this by moving the devices or services to nonessential branches of systemd.