NEMS Linux – Nagios Enterprise Monitoring Server for Raspberry Pi
Important Note: NEMS started as a small project here on my blog, but since has grown into a full-fledged distro! The blog therefore is here for historical purposes, but for the most current information, please visit the NEMS Linux web site: nemslinux.com
NEMS is a modern pre-configured, customized and ready-to-deploy Nagios Core image designed to run on the Raspberry Pi 3 micro computer. At its core it is a lightweight Debian Stretch deployment optimized for performance, reliability and ease of use.
NEMS is free to download, deploy, and use. Its development however is supported by its community of users. Please consider contributing if you can.
Please Note: NEMS is a very ambitious project, and I’m just one guy. Please consider throwing a little gift in my Tip Jar if you find NEMS saves you time or money. Thanks!
Support
[NEMS Documentation]
[NEMS Community Forum]
[NEMS User Comments]
Index
- What Is NEMS Linux and Nagios Core?
- Why Choose NEMS Linux?
- About NEMS
- NEMS’ Out Of The Box Experience
- System Requirements and Supported Platforms
- Instructions
- Buy The Needed Hardware
- Download
- Upgrade Instructions
- Changelog
NEMS 1.1 Featured on Category5 Technology TV
If you like NEMS, please donate: donate.category5.tv
The Out-Of-The-Box NEMS Experience:
-
- The first screen you see when connecting to an initiated NEMS 1.2 server.
-
- The quick statistics you see when logging into the NEMS terminal.
-
- The NEMS Monitorix screen shows you statistics about the performance of your NEMS server itself.
-
- NEMS NConf allows you to configure Nagios in a much more intuitive way.
-
- Nagios3 Tactical Overview and screens have been improved for NEMS.
-
- NagVis lets you see a graphical map of your network assets, including basic uptime information and current status.
-
- Check_MK offers a beautiful overview of your current and historical Nagios3 reports. A direct replacement for Nagios3 Core’s interface.
-
- The Host Info screen in Check_MK shows you the current status of your network.
-
- Another way to view the system uptime and other valuable information for your NEMS server itself.
-
- Historical graphs in RPi-Monitor let you know at a glance if your NEMS server is experiencing high load or temperatures and other important information.
-
- Webmin gives NEMS users a centralized interface for maintaining and monitoring your NEMS server.
-
- The notifications help you spot issues, and the built-in Software Package Updates feature of Webmin helps you keep your NEMS server up to date.
Raspberry Pi 3 are very affordable, and using our Micro SD image, you simply buy the device, “burn” the image to the Micro SD card, and boot it up.
Here’s our link to buy the device you’ll need, complete with the Micro SD card, a power adapter, a good solid case, and more: shop.category5.tv
Please buy it through that link, or let me know if you need a customized link to a different model. We get a small percentage of the sale, and it helps to make it possible to offer this as a free download.
Who Creates NEMS:
Robbie Ferguson is the host of Category5 Technology TV. He’s the kind of guy who when he figures stuff out, he likes to share it with others. That’s part of what makes his show so popular, but also what makes NEMS possible.
Support What I Do:
This project is a part of something much bigger than itself, and we’re all volunteers. Please see our Patreon page for information about our network.
– Please support us by simply purchasing your Raspberry Pi at https://cat5.tv/pi
– We have some support links on the NEMS menu, such as buying from Amazon using our partner link. Please use these every time you use those stores. A small percentage of your purchase will go toward our projects.
– Your donations are VERY MUCH appreciated – https://donate.category5.tv – Please consider how many hours (and hours) of work this project has saved you, and how much you’ll save on hardware and even electrical costs as you consider contributing
– Our network also has a Patreon page – Please consider becoming a patron – https://patreon.com/Category5
Hi Robbie,
Is there a way to upload custom image files for different OS ? As I would like to add CentOS logo via NConf. Many thanks in advance.
If there is an icon set you’d like added, it’d be best to submit it as a feature request so it is a permanent change. Adding it manually would mean if you ever restore from backup, you’ll lose that change. Please use one of the current support methods (see https://nemslinux.com under “Get Help”) rather than commenting on my blog since this post is from the very first release 🙂
Thanks for the prompt reply & noted, will post it there. Many thanks again.
How too add monitoring router or switch?
Hi Robbie,`works fine but I can not access http://192.168.33.10/Check_MK -“The requested URL /Check_MK was not found on this server.”
Check_MK was removed from NEMS Linux with the 1.4 branch, as per the release notes. Please try the included Adagios interface, which is much more current. Check_MK fell too far behind and the developer was taking a different direction.
Hi Robbie, I get the same messages here. I was running 1.2.3 for a couple weeks, backed up configs, created a new sd with NEMS 1.3 and walked throug the nems-init side, no errors. I can access the MOTD page and can log in into Nagios but any attempt to go into config ends with SSL errors. Can i help more to track this issue?
Thank you Omar. Please post in the community forum. There is a thread going about this issue: http://forum.category5.tv/thread-105.html
Yes, the more users’ systems I have access to (be it directly or vicariously through instructions), the faster I can write a patch for the fix. Thanks!
Robbie
Try this:
sudo nems-update && sudo nems-quickfix && sudo nems-certI have the same problem as Dave. I cannot access NEMS 1.3 with Chrome, Edge or IE. I have had NEMS installed before but this is a new install with new IP.
Hi Richard,
Okay thanks. Good to know Dave isn’t alone. I’ll escalate this. Please join the discussion: https://forum.category5.tv/thread-105.html
I cannot access the System settings tool, SSL config was filled out completely. Is it possible to disable SSL for the page somehow?
Hey Dave,
What’s the actual trouble accessing it? Is your browser allowing you to add an exception?
Did you add an exception for the self-signed certificate?
Did you read the documentation about SSL on NEMS? Any of it apply? https://docs.nemslinux.com/self-signed-certs
No, SSL cannot be disabled for NEMS-SST for security reasons. It transmits highly sensitive information. But I’m here to help figure out what’s going on.
Can you answer for me:
Worst case scenario, are you able to open things up to me so I can SSH in and see what’s up?
Also, you may try re-running nems-init if you feel you might have messed up the cert creation. You could PM me the output of nems-init on the community forum if you like to help me see what settings you’re using. Note: re-running nems WILL generate errors, but they won’t break things (eg., user already exists when creating the NEMS user).
Robbie,
This is a new deployment.
New IP, since it’s a new install
Your connection is not private
Attackers might be trying to steal your information from nems (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_INVALID
Subject: *.nems.local
Issuer: *.nems.local
Expires on: Sep 16, 2027
Current date: Nov 8, 2017
It looks like all pages fail with a similar error.
then only cert creation entry I’m not sure about is state, should it be PA or Pennsylvania?
Thanks,
Dave
Hi Dave,
This is covered in the docs. Please read https://docs.nemslinux.com/self-signed-certs
Essentially, because it is a self-signed certificate, you need to add an exception. Does your browser provide this option? Guessing this is Chrome?
If the “Add an exception” option isn’t there, that wouldn’t make sense as that only happens if you already had a NEMS server (and added an exception previously). But you said you didn’t. If that’s incorrect, it’s in the “I added a permanent exception, then reinstalled or upgraded NEMS, and now I can’t connect.” section of the doc.
Robbie,
I have tried Vivaldi, Chrome and MS Edge, none of the browsers offer to add an exception when you click advanced.
Full Text from Chrome and Vivaldi:
Your connection is not private
Attackers might be trying to steal your information from nems (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_INVALID
ReloadHIDE ADVANCED
nems normally uses encryption to protect your information. When Vivaldi tried to connect to nems this time, the website sent back unusual and incorrect credentials. This may happen when an attacker is trying to pretend to be nems, or a Wi-Fi sign-in screen has interrupted the connection. Your information is still secure because Vivaldi stopped the connection before any data was exchanged.
You cannot visit nems right now because the website sent scrambled credentials that Vivaldi cannot process. Network errors and attacks are usually temporary, so this page will probably work later.
What browser are you using that you are able to add an exception? I can also provide screenshots if that will help.
Thanks,
Dave
Thanks Dave.
I think it’d help to see what you’re entering during the nems-init process for the cert values. Could you kindly open a bug report in the community forum and then PM me the copy-and-paste from nems-init (run it again)?
We’ll figure it out!
Doing some searching online, apparently the error you’re receiving can happen if either your computer or the NEMS server’s clocks (date and time) are set incorrectly. Can you confirm both are set correctly?
Hi,
I wanted to try out NEMS, so I got myself a Kingston 64GB micro SDXC card and installed the image on it.
After booting I ran top and saw that the cpu load average was over 8. This slowly decreased and after a day(!) it was down to 1.8. The system behaved extremely sluggish when I ran nems-init. The web interface was almost unusable.
I had another 8GB card lying around and installed NEMS on that one. The system worked flawlessly with that card (cpu load around 0.2).
So I wonder if the rpi and NEMS can cope with large SD cards?
Is there anything I can check to find out what is going on?
Thanks.
Unfortunately once you get into > 32 GB SD cards on the Raspberry Pi, it’s going to be hit or miss whether they’ll work or not. I am a software guy, not hardware, but as I understand it it’s because the Pi’s Card Reader is SDHC, not SDXC… or something like that – so essentially it’s only meant for up to 32 GB cards, and some larger will work, but only if you’re lucky. Try replacing that beautiful little Kingston disk with another… pick a good fast one, and go with either 16GB or 32GB. I usually go with a UHS card, even though I don’t think the Pi can take advantage of the full speed… but I still prefer to lean toward the fastest card I can get.
OK, Ill get me a 32GB card.
Thanks Robbie.
My pleasure. Hope you enjoy NEMS!
Robbie,
Thanks so much for this open source project! I haven’t spent much time with it yet, but I do have a lot of experience with nagios and system administrator and am looking forward to getting my hands dirty with this. Here is some initial feedback I have from my first 24 hours with NEMS:
1) PLEASE include a configuration step in the documentation or add it to the nems-init script, an option to set the locale / keyboard. After I set my passwords, I couldn’t login from my workstation to anything (nconf, webmin, ssh, etc.) except for nagvis (obviously because that password isn’t set during nems-init), but I could login to the pi using a directly connected keyboard. Come to find out, the locale/keyboard is set to GB! Not a big deal, so long as you give the option to change it during setup is all. I didn’t even think to check that at first.
2) Alllll of the login pages except for webmin are over http. Please change everything to be using https by default and have any http requests redirected to https (using mod_rewrite, as an example). It doesn’t matter if people need to generate their own certificates afterwards or maybe you could even have “let’s encrypt” be a part of the configuration process. Logins shouldn’t happen over http *ever*, even if it’s on your local network.
3) Please change the line in your nems-init script where you’re using htpasswd to add the switch “-B”. By not specifying the algorithm, you are allowing the default of MD5 to be used, which is insecure.
That’s all for now. Thanks again!
-Steve
Hi Steve,
Thanks for the great suggestions. 1 & 3 will be added to NEMS v1.3 verbatim. So glad you caught that (and shared it) with the keyboard locale. I’d never have thought of that since everything works OOTB here … but someone elsewhere in the world may have trouble – thanks! Re. #2 (SSL), I agree and disagree. First, Let’s Encrypt is for WAN only. Can’t use it for LAN, as far as I know. If you know otherwise, please share. I will add SSL as an optional method of connection, but because the certs will be self-signed for each deployment of NEMS, the user must add exceptions to their browser. I wish there was a better way to force SSL with a valid certificate without needing to setup a hostname, but fact is, this is indeed LAN and unless you’re in a large network where some users may be untrustworthy (eg., MITM) http is perfectly safe / fine in LAN environments. But yes, you’re right: security through obscurity is not security at all. Knowing that “nobody on my network knows how to packet sniff” is not really a safe mindset overall. So it really just boils down to 1) do we force SSL (no, because it’ll cause warnings in the users’ browser) or 2) do we just add SSL as an option and allow the user to select it as the default?
Thanks!
Update: bcrypt encryption and keyboard locale settings have been rolled out. All NEMS 1.1-1.2.2 systems will receive this update within the next 24 hours.
How can I change the call NEMS.local to a DynDNS name, so that I can access the web front end from remote ?
Thx in advacne for your help and your great support.
This is ill-advised. I would recommend instead that you connect to your LAN via a SOCKS Proxy, and then access NEMS through a local IP. See https://category5.tv/shows/clips_tech/episode/507-ssh-socks-proxy/
How Linux-savvy are you? Based on your question itself, and the way you have posed it, I cannot answer in good conscience because I do not have confidence you understand the extreme risk in what you are requesting.
Please use best security practices.
Dear Robbie, thx for the quick reply. I‘m quite a bit savvy, but not a deep expert… 😉
Well, you’re on the right track! I just don’t want to go giving you dangerous advice that really are bad practice. Please give the Proxy suggestion a try (connect to your NEMS Server by first establishing an SSH SOCKS Proxy to say, your server or other fortified Linux system). I’d suggest using CSF/LFD on the system you are using for the proxy and only allow your own IP access.
good idea. will give it a try
Hey Robbie,
Sorry but how do i go about monitoring another RaspberryPi on the network with NEMS?
I googled and installed SNMP on RaspberryPi but the advanced services like CPU Load , / Disk Space , Uptime SNMP
do not work 🙁
Please see the documentation: http://www.baldnerd.com/installing-the-nagios-nrpe-client-agent-on-debian-ubuntu/
Thank you for the reply,
I’ve followed those, but I still get errors in NEMS
CPU Load, connect to address 192.168.2.17 and port 12489: Connection refused
Memory Usage, connect to address 192.168.2.17 and port 12489: Connection refused
Uptime,CRITICAL – Plugin timed out while executing system call .
Both Pis have been restarted
Any chance there’s a firewall running on the Pi’s you are trying to monitor? What distro is running on them? We could run a command line argument to test. Want to email me SSH access so I can have a look for you? Otherwise, I can write a quick tool to test it, but likely will take a few days to get to.
As a test I have installed “nagios-nrpe-server nagios-plugins” and edited “allowed_hosts” with 192.168.2.0/24 on another pi with same result.
I assume that CPU Load, Check the root filesystem disk space, Uptime inbuilt NEMS services should work with no modifications? I’m very green at the Nagios, I’ve just went in to hosts in NConf –> Show –> Show Services –> Advanced Services (directly linked) and moved “/ Deisk Space, CPU Load, Memory Usage, Swap Usage, Uptime SNMP (up time)”
The 2nd test pi has fail2ban but all nothing special in iptables to block.
Yeha i’m sorry i dont really want to give out SSH access to my pi, I would prefer the quick test tool. Thank you so much!!
Linux RaspberryPi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
Any luck mate 🙂 ?
I’ve never used Nagios for monitoring before, of all the tools bundled in NEMS, how much of it is accessible with a REST api?
I added the API based on user request, but have not used it myself (beyond deploying and ensuring it works, of course) – so hopefully this helps? https://github.com/zorkian/nagios-api/blob/master/README.md (Of course, the installation and config is already done for you on NEMS, so start reading at “API”).
Is there any information about how many hosts or clients that NEMS is designed to manage?
Hi Nick,
We’ve yet to encounter a scenario where NEMS is “maxed out” as far as number of hosts/clients/services go, so there are no numbers readily available. We have users running 100-200 hosts with nothing but praise for NEMS. Theoretically, as long as your MicroSD card is large enough, reliable and fast, NEMS should be able to handle several hundred (or more?) hosts. I look forward to someone trying with 500-1000 or more, because then we’ll really know more about how much it can handle.
I might just have to try it out and see what it can do 🙂 in our network infrastructure we could potentially hit that higher range of hosts.
I’d greatly appreciate you giving it a shot and reporting back. If you do encounter any issues, it’d be a great opportunity for me to optimize NEMS for such environments. Perhaps open a thread in the Community Forum for something like “NEMS managing 500+ hosts” or whatever it is. We can move the discussion there. Thanks!
I have several different vlans on my network. I setup NEMS on my 10.10.60.x server vlan and I can’t access it from my 10.10.50.x workstation vlan. If I start a ping from my workstation to my NEMS server it will reply once eth0 enables for about 5 ping then it starts dropping. Is there a internal firewall that I need to disable on the system so all my networks can talk to it?
Nope. Since NEMS is intended to be run on a LAN (not WAN) I opted out of preinstalling a firewall. My suggestion for those wanting WAN is to instead use a SOCKS proxy over SSH. So that said… hmmm… I’ve never tried such a setup, so this sounds like a networking configuration issue. Is your netmask on the NEMS server 255.255.0.0? By default it would be 255.255.255.0 which would probably cause what you’re describing. I assume you’ve changed it to match your network, but if not, that’d be the first thing I’d try. Not officially supported, of course 🙂
Really think we need a forum and/or wiki…
Anyways, I used Webmin to set static IPs, IPv4 and IPv6 addresses.. They don’t take…
Mind you I had this same issue on Raspbian, I was hoping I didn’t with NEMS.. What am I missing?
Hi Kevin,
I like your idea about a forum. It has come up before and you’re not the first to mention it, but I too feel that NEMS has now grown to the point where a WordPress comment system is insufficient.
I have therefore begun work on a new forum for the NEMS community. Please give it a try: http://forum.category5.tv/forum-8.html
Re. your problem with setting IP’s in Webmin: can you walk me through the steps you’re taking (perhaps in the new forum?) and I’ll try to replicate and resolve this for you? Thanks!
Here’s how i resolved the ‘secondary eth0’ concern…
Using PuTTY, I ran the ‘ip -4 addr show dev eth0 | grep inet’ command and it initially displayed…
@NEMS:~# ip -4 addr show dev eth0 | grep inet
inet 192.168.xxx.xxx/24 brd 192.168.1.255 scope global eth0
inet 192.168.xxx.xxx/24 brd 192.168.1.255 scope global secondary eth0
Afterwhich I ran the ‘sudo nano /etc/network/interfaces’ command and COMMENTed out all STATIC IP Interface references.
# auto eth0
# iface eth0 inet static
# address 192.168.xxx.xxx
# netmask 255.255.255.0
# gateway 192.168.xxx.xxx
… then SAVEd that file, and proceeded to do a ‘sudo nano /etc/dhcpcd.conf’ and add the entries at the bottom of that file…
interface eth0
static ip_address=192.168.xxx.xxx/24
static routers=192.168.1.1
static domain_name_servers=192.168.xxx.xxx 8.8.8.8
… then SAVEd the file, ran a ‘sudo shutdown -r now’.
When NEMS rebooted, I re-ran the ‘ip -4 addr show dev eth0 | grep inet’ command from PuTTY once more, and it then displayed, minus the ‘secondary eth0’ reference …
@NEMS:~# ip -4 addr show dev eth0 | grep inet
inet 192.168.xxx.xxx/24 brd 192.168.1.255 scope global eth0
The reference material used to come to this conclusion was…
https://raspberrypi.stackexchange.com/questions/37920/how-do-i-set-up-networking-wifi-static-ip-address
… and
https://raspberrypi.stackexchange.com/questions/32516/multiple-ip-addresses-being-assigned
Thanks for the post Hesh. I’ve not encountered this. Can you please post some info about it in the forum if it’s a bug and something I should be patching in NEMS 1.3?
How can I check a non standard port using check_http?
Simply add the port to the domain. Eg., baldnerd.com:2222
If however you want to check a https service (check_http vs check_https) you’ll want to create a new command for check_https which simply includes -S like this: $USER1$/check_http -I $HOSTADDRESS$ -S $ARG1$
I’ll be including check_https in a future release as well, but this is Nagios 101 so you can get lots of help already by simply checking Google. I’m happy to help though, if I can. Lemme know where you’re stuck.
Thanks robbie, its very hard going on the old brain. I’v had a manual install of nagios 4.something on a rpi for probably a year mainly to monitor my cctv system. I did the usual, ammend the windows.cfg with my pc & same with the printer…couldn’t get to grips with it. updated it to 4.3.2 the other day, struggled with lilac-reloaded..doesn’t work. Found your distro, wow.. within a few hours practically every device configured!
I’m still learning linux in general but i find the nagios help pages very difficult to digest.
In my nagios4 (can’t remember location, think /usr/local/nagios/??) like the printers.cfg, windows.cfg and the files in in the folder above.. Where are they now? Can nconf create new commands? Otherwise where are the services.cfg?
Thankyou.