Status update July 4th

Just wanted to let you know where we are with Lemmy.world.

Issues

As you might have noticed, things still won’t work as desired… we see several issues:

Performance

  • Loading is mostly OK, but sometimes things take forever
  • We (and you) see many 502 errors, resulting in empty pages etc.
  • System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)

Bugs

  • Replying to a DM doesn’t seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
  • 2FA seems to be a problem for many people. It doesn’t always work as expected.

Troubleshooting

We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.

So, all is not yet running smoothly as we hoped, but with all this help we’ll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!

  • czarrie@lemmy.world
    link
    fedilink
    arrow-up
    347
    arrow-down
    1
    ·
    1 year ago

    I’m just excited to be back in the Wild West again – all of the big players had bumps, at least this one is working to fix them.

  • Kalcifer@lemmy.world
    link
    fedilink
    English
    arrow-up
    150
    arrow-down
    3
    ·
    edit-2
    1 year ago

    That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

    Lemmy has a memory leak? Or, should I say, a “lemmory leak”?

        • donalonzo@lemmy.world
          link
          fedilink
          English
          arrow-up
          33
          arrow-down
          1
          ·
          1 year ago

          Rust protects you from segfaulting and trying to access deallocated memory, but doesn’t protect you from just deciding to keep everything in memory. That’s a design choice. The original developers probably didn’t expect such a deluge of users.

        • SomeOtherUsername@lemmynsfw.com
          link
          fedilink
          English
          arrow-up
          10
          arrow-down
          1
          ·
          1 year ago

          I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.

          • SomeOtherUsername@lemmynsfw.com
            link
            fedilink
            English
            arrow-up
            23
            ·
            edit-2
            1 year ago

            I think I found what eats the memory. DB iops isn’t the cause - looks like the server doesn’t reply before all the database operations are done. The problem is the unbounded queue in the activitypub_federation crate, spawned when creating the ActivityQueue struct. The point is, this queue holds all the “activities” - events to be sent to federated instances. If, for whatever reason, the events aren’t delivered to all the federated servers, they are retried with an exponential backoff for up to 2.5 days. If even a single federated instance is unreachable, all events remain in memory. For a large instance, this will eat up the memory for every upvote/downvote, post or comment.

            Lemmy needs to figure out a scalable eventual consistency algorithm. Most importantly, to store the messages in the DB, not in memory.

    • grue@lemmy.world
      link
      fedilink
      English
      arrow-up
      17
      arrow-down
      1
      ·
      1 year ago

      Lemmy has a memory leak? Or, should I say, a “lemmory leak”?

      A lemmory meek, obviously!

    • 257m@lemmy.ml
      link
      fedilink
      English
      arrow-up
      8
      arrow-down
      2
      ·
      1 year ago

      Wait isn’t lemmy written in rust how do you create a memory leak in rust? Unsafe mode?

  • Eczpurt@lemmy.world
    link
    fedilink
    arrow-up
    130
    arrow-down
    2
    ·
    1 year ago

    Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn’t happen without you!

  • Shartacus@lemmy.world
    link
    fedilink
    arrow-up
    106
    arrow-down
    1
    ·
    1 year ago

    I want this to succeed so badly. I truly feel like it’s going to be sink or swim and will reflect how all enshitification efforts will play out.

    Band together now and people see there’s a chance. Fail and we are doomed to corporate greed in every facet of our lives.

  • Snow-Foxx@lemmy.world
    link
    fedilink
    arrow-up
    95
    ·
    1 year ago

    Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.

    Seriously, you’re doing a great job <3

  • cristalcommons@lemmy.world
    link
    fedilink
    arrow-up
    90
    ·
    1 year ago

    i just wanted to thank you for doing your best to fix lemmy.world as soon as possible.

    but please, don’t feel forced to overwork yourselves. i understand you want to do it soon so more people can move from Reddit, but i wouldn’t like that Lemmy software and community developers overwork and feel miserable, as those things are some of the very motives you escaped from Reddit in first place.

    in my opinion, it would be nice that we users understand this situation and, if we want lemmy so bad, we actively help with it.

    this applies to all lemmy instances and communities, ofc. have a nice day you all! ^^

    • Cinner@lemmy.worldB
      link
      fedilink
      arrow-up
      36
      ·
      1 year ago

      Plus, slow steady growth means eventual success. Burnout is very real if you never take a break.

      • cristalcommons@lemmy.world
        link
        fedilink
        arrow-up
        10
        ·
        edit-2
        1 year ago

        so true, pal! slowly, with patience, no rushing, putting love into it, organizing ourselves, working smart is better than working hard and fast.

        because of the federated nature of fediverses like Lemmy, it is very possible that many people are doing the very same task without even knowing they are duping each other’s efforts.

        and that’s sad because if they knew, they could be teaming up, or splitting the task in two, in order to avoid wasting different efforts into dupe results.

        i have learnt a thing or two about burnout, it’s better for me to make 40% planning, and 40 % self-care and so the 20 % of execution becomes piece of cake.

        but this is just my opinion. anyway, please take care, pals <3

  • Clbull@lemmy.world
    link
    fedilink
    arrow-up
    80
    arrow-down
    1
    ·
    1 year ago

    As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I’m remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.

    • Tristano@lemmy.ca
      link
      fedilink
      arrow-up
      33
      arrow-down
      2
      ·
      1 year ago

      I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.

      So far the fediverse feels really different tho, very explicitly anti that type of shit. I’m sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.

      • itboss@lemmy.world
        link
        fedilink
        English
        arrow-up
        29
        arrow-down
        1
        ·
        1 year ago

        FYI, it has popped up explodingheads is a great example but many servers including lemmy.world became proactive in defederating from the instance

        • givesomefucks@lemmy.world
          link
          fedilink
          English
          arrow-up
          12
          arrow-down
          1
          ·
          1 year ago

          Yeah, but as soon as instances defederated from there, those bigots started making alts on those instances.

          Because it’s not enough for them to have their own safe space for bigotry, they want to argue that bigotry is ok so they get their rage fix from arguing online.

          If Lemmy can instances can’t IP ban like reddit does, they’re just going to keep making accounts here every time they’re banned. Then go to exploding heads to circle jerk about how they got banned again.

          • henfredemars@infosec.pub
            link
            fedilink
            English
            arrow-up
            10
            arrow-down
            1
            ·
            1 year ago

            One can also increase the cost of registration by requiring some questions and a waiting period. Adding friction makes this attack a lot less fun for the attacker.

            It’s not a perfect solution, but perfect must not be the enemy of good.

            • T156@lemmy.world
              link
              fedilink
              English
              arrow-up
              6
              ·
              1 year ago

              At the same time, it’s also not suitable for a server the size of lemmy.world. We just have too many users, and enough new people joining that we’d need several people whose singular task was to process registrations in that case.

    • VindictiveJudge@lemmy.world
      link
      fedilink
      arrow-up
      20
      arrow-down
      1
      ·
      1 year ago

      Lemmy is a distributed system that can’t quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.

  • cartmansbellybuttom@lemmy.world
    link
    fedilink
    arrow-up
    78
    arrow-down
    1
    ·
    1 year ago

    As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You’re doing great with such fast exponential growth

    💙 Thanks for your hard work!

  • Frost Wolf@lemmy.world
    link
    fedilink
    arrow-up
    73
    ·
    1 year ago

    This is the level of transparency that most companies should strive for. Ironic that in terms of fixing things, volunteer and passion projects seem to be more on top of issues compared to big companies with hundreds of employees.

    • fuck reddit@lemmy.ml
      link
      fedilink
      arrow-up
      23
      ·
      1 year ago

      You said it: passion projects. While being paid is surely a motivator, seeing your pet project take off the way Lemmy is can be so intoxicating and rewarding! I plan to donate as soon as I get paid on Friday! I want to see this succeed, even if it is just to spite Reddit, and I am willing to pay for the pleasure.

  • Candelestine@lemmy.world
    link
    fedilink
    arrow-up
    70
    arrow-down
    2
    ·
    1 year ago

    What was that? We’re going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?

    Yeah, that sounds pretty reasonable, we can work with that.

        • Amanduh@kbin.social
          link
          fedilink
          arrow-up
          11
          arrow-down
          1
          ·
          1 year ago

          I’m thoroughly confused by most of what is going on. I’m using kbin on mobile. The interface isn’t terrible and it fills the random surfing void that I am used to.

          • Mr_Jabroni@kbin.social
            link
            fedilink
            arrow-up
            24
            arrow-down
            1
            ·
            1 year ago

            Just a heads up, you’re using kbin which is a different “website” than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other’s content, but the website UI is different between them so if you read about some UI element and can’t find it in kbin that might be one of the reasons.

            That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn’t fit to the side in such a small screen.

            • Amanduh@kbin.social
              link
              fedilink
              arrow-up
              16
              ·
              edit-2
              1 year ago

              Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines

              • BrerChicken @lemmy.world
                link
                fedilink
                arrow-up
                5
                ·
                1 year ago

                Yesss! Lemmy was difficult to use until I did that. Then eventually I had a long enough list of communities/magazines that I could browse subscribed only and that really supposed the issue. I’m busy not a fan of random porn and bigoted memes y’know?

                • Amanduh@kbin.social
                  link
                  fedilink
                  arrow-up
                  2
                  ·
                  1 year ago

                  I mean if a random nude crosses my feed I’m not bothered by it but I really just want to read some cool stuff maybe news etc for like 15 mins at a time throughout the day and this is scratching that itch for me.

              • T156@lemmy.world
                link
                fedilink
                arrow-up
                3
                ·
                1 year ago

                If you’re on Kbin mobile, you can also tap the hamburger bar on the top left to bring up that menu directly, instead of having to scroll all the way down.

                Also great for if you want to fiddle with dark mode and all of that.

      • ImFresh3x@sh.itjust.works
        link
        fedilink
        arrow-up
        7
        ·
        1 year ago

        Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”

        If you’re using a 3rd party app it’s different for each one in terms of how to see it.

      • Drunemeton@lemmy.world
        link
        fedilink
        arrow-up
        4
        ·
        1 year ago

        It’s here!

        (Please do note that the blue checkmark is for our 1€ “Bean” tier subscribers.)

        We’re very pleased to provide you with our premium deluxe green checkmark for your generous 8€ subscription to our “Bacon” tier!

        Feel free to reach out to our Lemmy Cares team @ 1-800-FUC-SPEZ for any questions or concerns that you may have.

  • Jonas Nordlund@social.lol
    link
    fedilink
    arrow-up
    60
    arrow-down
    2
    ·
    1 year ago

    @ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%

    Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻

      • thews@lemmy.world
        link
        fedilink
        arrow-up
        5
        arrow-down
        1
        ·
        1 year ago

        I haven’t checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.

          • thews@lemmy.world
            link
            fedilink
            arrow-up
            2
            ·
            1 year ago

            I read down the list afterwards and found it was using Rust. I skimmed through the source and it is well organized, but would still take quite a while to get up to speed on.

            I saw unit and integration tests. It might be beneficial to generate or capture some data to replay to simulate the load and add debugging. I don’t know much about the abstraction layers. I did see opentelemetry, which is a project I got frustrated with on the lack of stability (fast changes on api).

            I have only dabbled with Rust to test the waters. The largest thing I’ve made was a GUI snake game, and made it portable so it could be compiled for cross platform.

  • Deez@lemm.ee
    link
    fedilink
    arrow-up
    53
    ·
    1 year ago

    Thanks for all of your effort. Even though we are on different instances, it’s important for the Fediverse community that you succeed. You are doing valuable work, and I appreciate it.

    • TheSaneWriter@vlemmy.net
      link
      fedilink
      arrow-up
      7
      ·
      1 year ago

      Not just that, but the code contributed to Lemmy by this debugging will make Lemmy run faster for everyone on every instance, which is makes the ecosystem that much better.

  • Sausage_Mahoney@lemmy.world
    link
    fedilink
    arrow-up
    53
    ·
    1 year ago

    The work you’re doing is greatly appreciated! It’s like you invited half the internet into your house. I feel like I should’ve brought a cake or something

    • Action [email protected]@lemmy.world
      link
      fedilink
      arrow-up
      18
      ·
      1 year ago

      You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you’re not only covering your own hosting fees, but covering for some of the folks who can’t afford to as well. And maybe Ruud gets to buy some beans.

  • SeaMauFive@lemm.ee
    link
    fedilink
    arrow-up
    46
    ·
    edit-2
    1 year ago

    Huge respect for what you’ve built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here