I’m looking for experiences and opinions on kubernetes storage.

I want to create a highly available homelab that spans 3 locations where the pods have a preferred locations but can move if necessary.

I’ve looked at linstore or seaweedfs/garage with juicefs but I’m not sure how well the performance of those options is across the internet and how well they last in long term operation. Is anyone else hosting k3s across the internet in their homelab?

Edit: fixed wording

  • karlhungus@lemmy.caEnglish
    5·
    26 days ago

    My gut says go multi cluster (or not) at that pointbut treat the remote as a service, have a local container be a proxy

    • InnerScientist@lemmy.worldOPEnglish
      21·
      26 days ago

      I want the failover to work in case of internet or power outage, not local cluster node failure. Multiple clusters would make configuration and failover across locations difficult or am I wrong?

      • karlhungus@lemmy.caEnglish
        2·
        26 days ago

        I guess i shouldn’t have answered, I do have experience with multiple storage-classes, but none of the classes you mention (so like i don’t really know anything about them). I envisioned you dealing with pod level storage issues and thought that’d be something most programs would have lots of difficulty dealing with, where as a more service oriented approach would expect remote failures (hence the recommendation).

        All of the things you mentioned don’t seem like they have provisioners, so maybe you mean your individual nodes would have these associated remote fs’. At that point i don’t think kubelet cares, you just mount those on the machines and tell kubelet about it via host mount

        Oh shit look there’s a CSI driver for juicefs https://juicefs.com/docs/csi/introduction/, they kinda start out recommending the host mount https://juicefs.com/docs/cloud/use_juicefs_in_kubernetes/.

        We make some use of PV’s but people i find my team often tend to avoid them.

        I probably should have shut my mouth from the start!

        • InnerScientist@lemmy.worldOPEnglish
          1·
          26 days ago

          They both support k8s, juicefs with either just a hostpath (not what i’d use) or the JuiceFS CSI Driver. Linstore has an operator which uses drbd and provides it too.

          If you know of storage classes which are useful for this deployment (or just ones you want to talk about in general) then go on. From what I’m seeing in this thread I’ll probably have to deploy all options that seem reasonable and test them myself anyways.

          • karlhungus@lemmy.caEnglish
            2·
            26 days ago

            For this kind of thing i usually go by popularity (active repo/popular repo), mostly to have the most other people in your boat. It doesn’t always work but generally if other users have to migrate at least you can ask them questions.

            On the face of it i’d go with the csi driver version, only because we use alternative csi drivers ourselves, and haven’t seen any issues (ours are pretty aws vanella though).

            We use storage classes (for our drivers) the “dynamic provisioning” section of https://juicefs.com/docs/csi/guide/pv, you’ll need to make one of those, then create a statefulset and mount the pv in there.

            I do find statefulsets to be a bit of a not as well supported part of kubernetes, but generally they work well enough.

  • notfromhere@lemmy.mlEnglish
    3·
    26 days ago

    I know Ceph would work for this use case, but it’s not a lighthearted choice, kind of an investment and a steep learning curve (at least it was, and still is, for me).

    • InnerScientist@lemmy.worldOPEnglish
      1·
      26 days ago

      I heard that ceph lives and dies with the network hardware. Is a slow internet connection even usable when the docs want 10 gbit/s networking between nodes?

      • notfromhere@lemmy.mlEnglish
        2·
        25 days ago

        I’m really not sure. I’ve heard of people using Ceph across datacenters. Presumably that’s with a fast-ish connection, and it’s like joining separate clusters, so you’d likely need local ceph cluster at each site then replicate between datacenters. Probably not what you’re looking for.

        I’ve heard good things about Garbage S3 and that it’s usable across the internet on slow-ish connections. Combined with JuiceFS is what I was looking at using before I landed on Ceph.

    • InnerScientist@lemmy.worldOPEnglish
      2·
      26 days ago

      I mean storage backends as in the provisioner, I will use local storage on the nodes with either lvm or just storage on a filesystem.

      I already set up a cluster and tried linstore, I’m searching for experiences with the options because I don’t want to test them all.

      I currently manage all the servers with a NixOS repository but am looking for better failover.