Note: It seems my original post from last week didn’t get posted on lemmy.world from kbin (I can’t seem to find it) so I’m reposting it. Apologies to those who may have already seen this.

I’m looking to deploy some form of monitoring across my selhosted servers and I’m a bit confused about the different options.

I have a small network of three machines that I would like to monitor. I am not looking for a solution that lets me monitor tens, hundreds, or thousands of nodes. Furthermore, I am more interested in being able to observe metrics for each node individually rather than in aggregate. Each of these machines performs a different task so aggregate metrics from these machines are not particularly meaningful. However, collecting all the metrics centrally so that I can have a single dashboard to view them all in one convenient place is definitely something I would like.

With that said, I have been trying to understand the different (popular) options that are available and I would like to hear what the community’s experience is with these options and if anybody has any advice on any of these in light of my requirements above.

Prometheus seems like the default go-to for monitoring. This would require deploying a node_exporter on each node, a prometheus service, and a grafana dashboard. That’s all fine, I can do that. However, from all that I’m reading it doesn’t seem like Prometheus is optimised for my use case of monitoring each node individually. I’m sure it’s possible, but I’m concerned that because this is not what it’s meant for, it would take me ages to set it up such that I’m happy with it.

Netdata seems like a comprehensive single-device monitoring solution. It also appears that it is possible to run your own registry to help with distributed monitoring. Not gonna lie, the netdata dashboard looks slick. An important additional advantage is that it comes packaged on Debian (all my machines run Debian). However, it looks like it does not store the metrics for very long. To solve that I could also set up InfluxDB and Grafana for long-term metrics. I could use Prometheus instead of InfluxDB in this arrangement, but I’m more likely to deploy a bunch of IoT devices than I am to deploy servers needing monitoring which means InfluxDB is a bit more future-proof for me as it could be reused for IoT data.

Cockpit is another single-device solution which additionally provides direct control of the system. The direct control is probably not so much of a plus as then I would never let Cockpit be accessible from outside my home network whereas I wouldn’t mind that so much for dashboards with read-only data (still behind some authentication of course). It’s also probably not built for monitoring specifically, but I included this in the list in case somebody has something interesting to say about it.

What’s everybody’s experience with the above solutions and does anybody have advice specific to my situation? I’m currently leaning to netdata with my own registry at first and later add InfluxDB and Grafana for long-term metrics.

  • dditty@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    1 year ago

    I’m running netdata on each of my servers and it has every feature I need. If u choose netdata, make sure not to install the nightly builds since they get updated all the time and sometimes break features. One annoying thing with netdata is you have to pay a subscription for the option to disable individual alert types. I have a nearly full hard drive and there’s an alert for that which won’t go away. Same thing for temporary inbound packet drops which seems to happen everytime one particular Plex user forcibly transcodes content (they’re old and remote and won’t change their Plex client settings 😡). Each error they send you an email.

    • vegetaaaaaaa@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      1 year ago

      you have to pay a subscription for the option to disable individual alert types

      Never heard of that. You can disable individual alarms by setting to: silent in the relevant health.d configuration file (e.g. health.d/disks.conf). This is exactly what I do for packet drops.

      • dditty@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        Ooh sweet thnx 4 the tip - will try this later! I was trying via the web dashboard which I’m pretty sure requires a subscription