Homelab Party 2
I don’t think it’s coming across well that I’m making a Houseparty 2 reference, but I’m sure it’s going to land much better now that I’ve explained it. You can check out the first post, No Lab Like 127.0.0.1 if you’re curious about the early stages. When you’re ready, keep going because we’re gonna start monitoring the configs on this router.
Table Of Contents
A Brief Look At The MikroTik Experience So Far
As it’s been said, many times, you get a lot of bang for your buck with MikroTik products, and RouterOS. But the documentation online is somewhat scattershot. You’ll find conflicting information, incomplete information, and information that, from all accounts, is just blatantly incorrect but it seems to be working for somebody. Some of the documentation for some areas of the RouterOS config are very very scarce on the ground, and it feels like you need a guru to discover what this field or checkbox actually does.
Starting over was really important. To illustrate the above: if you read the last post you know that the “autoconfig” VPN stuff is really rudimentary, and not particularly helpful. Could there have been more parameters programmed in? Sure. But there aren’t. It’s just a cold world out there, and if your wants or needs don’t exactly match someone else’s then you’re going to have to piece it together. But I’ll outline my journey here, not only for my reference but for yours. I don’t really feel like I “hit the ground running” as I’ve had a couple of runs at this, but I think I’ve got it going well enough.
RouterOS Upgrade
I should have done this from the beginning. My fault. I was on the latest RouterOS 6, and didn’t realize there’s an upgrade update channel. Live and learn, I suppose, but I’m putting this here for whoever needs it. Sometimes there are silly little gotchas that can derail your project.
# 1. Download and install RouterOS 7
/system package update
set channel=upgrade
check-for-updates
download
# 2. After reboot, upgrade the RouterBOARD bootloader
/system routerboard upgrade
/system reboot
RouterOS 7 has some fixes, and notable to me are:
- support for L2TPv3
- support for REST API
- support for WireGuard
but you can read the upgrade notes for yourself, if you like. I didn’t have any concerns upgrading as I was going to reset the config to default, anyway. Moving forward with the immediate project, it didn’t seem like being on RouterOS 6 had any impact.
Keeping Track Of The Configses
You don’t need to keep up with the Joneses as much as you need to keep track of your configs. I looked for a network configuration tracker that had broad support, and especially support for MikroTik devices, and found Oxidized. It’s a bit sparse, in terms of documentation, but this is a homelab so I won’t need the most elaborate configurations. The more interesting part of it, is the integration with LibreNMS.
LibreNMS Installation
It didn’t seem like rocket surgery, but I did find a script that would make it a 5-minute affair from a Tall Paul Tech video, except that wasn’t quite right.
- Spin up an Ubuntu 24.04 CT in proxmox (more on proxmox in another post).
- Configure it and grab the script linked in the video.
- Realize that LibreNMS requires PHP 8.4 now, and the Ubuntu packages aren’t quite there yet.
Upgrading to PHP 8.4
This is classic Linux stuff, I must say. And it’s not without a reasonable cause, but it can be frustrating. Usually, it’s worth it to take a deep breath, and prepare to do a little bit of research. I came across the solution easily enough. PHPWatch has the exact post that one might need, and with Ondřej Surý’s repository which makes this all the better. I’ll include the instructions for Ubuntu below:
# Add and update the repo
sudo LC_ALL=C.UTF-8 add-apt-repository ppa:ondrej/php
sudo apt update
# nginx uses FPM to handle the calls
sudo apt install php8.4-cli php8.4-fpm
# make sure everything went well enough
sudo systemctl status php8.4-fpm
# I don't recall what extensions LibreNMS may need
sudo apt install php8.4-common php8.4-{bcmath,bz2,curl,gd,gmp,intl,mbstring,opcache,readline,xml,zip,json,mysql,snmp}
# if you don't need PHP 8.3 anymore
sudo apt purge '^php8.3.*'
I hadn’t hosted a PHP website in a very long time, so I ran into some issues with migrating LibreNMS, specifically, to PHP 8.4 and it involved a few files.
So, when you get the script, you obviously change DATABASEPASSWORD and WEBSERVERHOSTNAME. You’ll want to change your timezone, too, on lines 66-79. Then you’ll want to make sure to copy /etc/php/8.4/fpm/pool.d/librenms.conf to your php/8.4/ directory. No other changes necessary, from what I recall.
But you can also download my updated script and just run that.
While We’re Here Let’s Monitor Proxmox
Once you’re logged in and setup with LibreNMS, you can start adding devices. Let’s start with Proxmox first, as it’s pretty straightforward. LibreNMS has a native Linux agent called Check_MK, and we can install that on Proxmox easily as Proxmox is Debian.
LibreNMS likes to be able to reach things via SNMP, however. I got an error adding my Proxmox server even after enabling the check_mk agent, because it stated that it wasn’t responding. You can skip this and add it by leaving SNMP enabled on that screen, but also enabling the “Force add” toggle, and then waiting 5 minutes, which is the default for LibreNMS. It will ping, but that’s it. We need to do some more config.
apt install -y git #not installed by default, get the files however you want
cd /opt/
git clone https://github.com/librenms/librenms-agent.git
cd librenms-agent
cp check_mk_agent /usr/bin/check_mk_agent
chmod +x /usr/bin/check_mk_agent
cp check_mk@.service check_mk.socket /etc/systemd/system
# there's a specific Proxmox file to copy
mkdir -p /usr/lib/check_mk_agent/plugins /usr/lib/check_mk_agent/local
cp agent-local/proxmox /usr/lib/check_mk_agent/local/
chmod +x /usr/lib/check_mk_agent/local/proxmox
systemctl enable check_mk.socket && systemctl start check_mk.socket
Log in to the web UI for LibreNMS UI and add your device. Once you’ve got your device added, click into it, and then click the gear at the top right. There’s a horizontal menu for Device Settings, and then one we’re interested in at the moment is Applications. You’ll see that LibreNMS has a vast array of support built-in, and if your agent was communicating correctly then you probably already had a banner saying that the Proxmox application was detected. In that long list of applications, go ahead and enable Proxmox. Go into Modules and find UNIX Agent and enable that, and you should be all set.
Except Maybe You’re Not
I certainly wasn’t. I had seen this in my research, but it seems like LibreNMS really likes SNMP. In terms of how I’m using it, this might not be the best fit, but it’s certainly a quicker installation than Wazuh, which I had nuked when I transitioned from Hyper-V 2019 to Proxmox on this box. So let’s enable SNMP. Below is a simple configuration I cribbed from somewhere. SNMP is a deeper subject than I want to get into right now, so consider this a Fisher-Price config.
# =============================================================================
# AGENT BEHAVIOR
# =============================================================================
# Listen on standard UDP port 161 across all local network interfaces
agentaddress udp:161
# =============================================================================
# SYSTEM INFORMATION
# =============================================================================
sysLocation UnderDeskSlot B
sysContact Mark <markd@marktoso.com>
sysServices 72
# =============================================================================
# ACCESS CONTROL (Security)
# =============================================================================
# 1. Define an explicit view that grants access to the entire OID tree
view all_tree included .1
# 2. Configure community access (Restricted to your monitoring server IP)
# Syntax: rocommunity COMMUNITY_STRING SOURCE_IP -V VIEW_NAME
rocommunity proxmox 192.168.1.5/32 -V all_tree
# community string is *not* a password, it's a poorly kept secret at best
# =============================================================================
# PROCESS MONITORING (Proxmox Daemons)
# =============================================================================
proc pvestatd 1 1
proc pveproxy 50 1
proc pvedaemon 50 1
proc corosync 1 1
proc qemu-system-x86
proc lxc-start
# =============================================================================
# SYSTEM MONITORING IMPROVEMENTS (Disk & Memory)
# =============================================================================
# Monitor root filesystem storage usage; triggers error if less than 10% free
includeAllDisks 10%
# Track system load averages (Triggers warning if 1, 5, or 15 min loads spike)
load 12 10 8
So I dropped this onto my box with nano, and then made a backup of the original snmpd.conf and slotted this one in, and restarted snmpd.
apt install snmpd -y
# copy the above config in here, or however you want
nano snmpd-marktoso.conf
cp /etc/snmp/snmpd.conf /etc/snmp/snmpd-og.conf
cp snmpd-marktoso.conf /etc/snmp/snmpd.conf
systemctl restart snmpd
I’m not going to say that this is a good, or even a secure configuration, but it’s fine for homelabbing. Excuses, I know, but I can’t get caught in the weeds. It makes LibreNMS happy. I don’t love SNMP for this, but I also don’t want to spend too much time fooling around with alternatives just yet. LibreNMS has some benefits and some drawbacks, but the benefits are working for me at the moment and the drawbacks are trivial. I really wanted it for just one thing, and figured I might as well take advantage of the additional monitoring capabilities it has.
Oxidized, or is it 0xidized?
I’ve heard it both ways, but the command is oxidized so I’ll go with that. Oxidized is a network device configuration backup tool. I’d heard of RANCID many years ago, but my career path didn’t take me down this route at the time, so I never investigated it. Now that I’ve finally taken up the sword of the homelab, I found that RANCID would not be an ideal tool for my MikroTik devices, but I found Oxidized.
Installation is fairly straightforward from the GitHub README:
add-apt-repository universe
apt install ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libssh2-1-dev libicu-dev zlib1g-dev g++ libyaml-dev libzstd-dev
gem install oxidized
gem install oxidized-web # Web interface and rest API
gem install oxidized-script # Script-based input/output extensions
Also, from the README, I recommend that you do not run Oxidized yet. Do this first:
useradd -s /bin/bash -m oxidized
sudo -i -u oxidized
mkdir -p ~/.config/oxidized/configs
sudo cp extra/oxidized.service /etc/systemd/system
sudo mkdir /run/oxidized
sudo chown oxidized:oxidized /run/oxidized
sudo systemctl enable oxidized.service # enable on startup
# only now, as the oxidized user, will you run the command
oxidized
That will get oxidized going, but we still need to configure Oxidized and then hook it into LibreNMS.
Configure Oxidized
Oxidized has two main files that we’ll be concerned with: /home/oxidized/.config/oxidized/config and /home/oxidized/.config/oxidized/router.db. These will be, surprising no one, the main config file for oxidized, and then the list of network devices that it should be checking on. Let’s start with router.db as it’s very simple:
# address, device type, device username, device password
192.168.88.1:routeros:oxidized:password
So you can see that oxidized is going to reach out and log in to the device in question to grab the config. We list the address, type, username, and password so that it knows what to do and how to do it. Very straightforward, but we’re going to ensure that we have ssh access from Oxidized (or LibreNMS if you co-located them like I did). It’s important that you protect Oxidized, and limit access to it as it will have your network secrets in the config. You can add remove secrets: true to the vars in the config file to prevent that from happening, but you’ll need to verify that and not take it on faith.
Right now I have one management network that’s basically where the brains of the operation is, and that’s the default network behind the MikroTik 4011, so for the 5 ports I have on that network I don’t really need to create a rule, but if you do you can do:
/ip firewall filter
add action=accept chain=input comment="SSH from LAN" dst-port=22 \
in-interface-list=LAN protocol=tcp src-address=192.168.88.0/24
The way that my setup is, I will always need to add new devices into router.db to ensure that Oxidized is picking up the configurations for them. There is further configuration where LibreNMS will feed Oxidized, but that’s out of scope at the moment.
The Oxidized config file is a little more involved. There is an example config that you should use as a base.
Configure LibreNMS for Oxidized
Use that configuration as your baseline. You probably don’t want telnet to be an option, so remove that from the inputs, but other than that it’s pretty much what we want. Add the following line at the bottom to ensure that mikrotik is mapped to routeros:
model_map:
juniper: junos
cisco: ios
mikrotik: routeros
You might want to have your Oxidized listen on 127.0.0.1 if it’s colocated with LibreNMS. Then in LibreNMS you’re going to go to Global Settings -> External - Oxidized and there you will:
- Enable Oxidized Support
- Populate the URL to your Oxidized instance (with port number)
and I believe that’s it. If you had already seen your Oxidized devices showing up in its own web interfaces, you’ll very soon be able to go to your network device in LibreNMS and see a Config item in the top horizontal menu that will take you to the Oxidized configs.
Here you will have versioning and everything you’ll need to manage your configs in an orderly fashion. Also you get a bunch of cool graphs. You can view diffs between the configs as well, which will be very helpful as I continue to troubleshoot my VPN issue.
./content/posts/homelab-party-2.md 1:1 suggestion You averaged 1.27 complex marktoso.Kiss words per sentence 10:1 warning Use first person (such as 'I Microsoft.FirstPerson ') sparingly. 10:1 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 10:44 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 10:132 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 10:200 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 10:384 warning Try to avoid using Microsoft.We first-person plural like 'we'. 14:4 suggestion 'A Brief Look At The MikroTik Microsoft.Headings Experience So Far' should use sentence-style capitalization. 16:9 suggestion 'been said' looks like passive Microsoft.Passive voice. 16:199 error More than 3 commas! marktoso.TresComas 16:284 suggestion Consider using 'wrong' instead Microsoft.ComplexWords of 'incorrect'. 16:335 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 16:335 warning Consider using 'some' instead Microsoft.Wordiness of 'Some of the'. 16:403 warning Consider removing 'very'. Microsoft.Adverbs 16:409 warning Consider removing 'very'. Microsoft.Adverbs 18:19 warning Consider removing 'really'. Microsoft.Adverbs 18:55 suggestion Verify your use of 'above' Microsoft.Vocab with the A-Z word list. 18:119 suggestion 'VPN' has no definition. Microsoft.Acronyms 18:132 warning Consider removing 'really'. Microsoft.Adverbs 18:259 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 18:407 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 18:420 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 18:450 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 18:486 warning Consider removing 'really'. Microsoft.Adverbs 18:571 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 20:4 suggestion 'RouterOS Upgrade' should use Microsoft.Headings sentence-style capitalization. 22:1 warning Use first person (such as 'I Microsoft.FirstPerson ') sparingly. 22:45 warning Use first person (such as Microsoft.FirstPerson 'My') sparingly. 22:54 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 22:158 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 22:174 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 36:43 warning Use first person (such as Microsoft.FirstPerson 'me') sparingly. 39:15 suggestion 'REST' has no definition. Microsoft.Acronyms 42:179 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 44:4 suggestion 'Keeping Track Of The Microsoft.Headings Configses' should use sentence-style capitalization. 48:5 suggestion 'LibreNMS Installation' Microsoft.Headings should use sentence-style capitalization. 50:1 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 50:40 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 56:19 warning Avoid using acronyms in a Microsoft.HeadingAcronyms title or heading. 58:29 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 58:199 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 58:227 warning Consider removing 'easily'. Microsoft.Adverbs 78:1 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 78:1 warning Use first person (such as 'I Microsoft.FirstPerson ') sparingly. 78:37 warning Consider removing 'very'. Microsoft.Adverbs 78:56 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 80:146 warning In most cases, use 'from' or Microsoft.Ranges 'through' to describe a range of numbers. 80:146 error Use an en dash in a range of Microsoft.RangeFormat numbers. 80:298 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 82:27 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 84:5 suggestion 'While We're Here Let's Microsoft.Headings Monitor Proxmox' should use sentence-style capitalization. 84:11 warning Try to avoid using Microsoft.We first-person plural like 'We'. 84:22 warning Try to avoid using Microsoft.We first-person plural like 'Let's'. 84:28 suggestion Consider using 'check' or Microsoft.ComplexWords 'watch' instead of 'Monitor'. 86:78 warning Try to avoid using Microsoft.We first-person plural like 'Let's'. 86:223 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 86:250 warning Try to avoid using Microsoft.We first-person plural like 'we'. 86:281 warning Consider removing 'easily'. Microsoft.Adverbs 88:47 suggestion 'SNMP' has no definition. Microsoft.Acronyms 88:61 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 88:84 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 88:135 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 88:187 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 88:227 suggestion 'SNMP' has no definition. Microsoft.Acronyms 88:390 warning Try to avoid using Microsoft.We first-person plural like 'We'. 109:267 warning Try to avoid using Microsoft.We first-person plural like 'we'. 109:320 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 109:397 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 109:508 suggestion 'was detected' looks like Microsoft.Passive passive voice. 111:5 suggestion 'Except Maybe You're Not' Microsoft.Headings should use sentence-style capitalization. 113:1 warning Use first person (such as 'I Microsoft.FirstPerson ') sparingly. 113:40 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 113:80 warning Consider removing 'really'. Microsoft.Adverbs 113:93 suggestion 'SNMP' has no definition. Microsoft.Acronyms 113:99 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 113:115 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 113:222 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 113:239 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 113:300 warning Try to avoid using Microsoft.We first-person plural like 'let's'. 113:313 suggestion 'SNMP' has no definition. Microsoft.Acronyms 113:350 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 113:377 suggestion 'SNMP' has no definition. Microsoft.Acronyms 113:406 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 160:3 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 160:24 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 173:1 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 173:128 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 173:200 suggestion 'SNMP' has no definition. Microsoft.Acronyms 173:218 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 173:280 suggestion Consider using 'choices' Microsoft.ComplexWords or 'options' instead of 'alternatives'. 173:383 warning Use first person (such as Microsoft.FirstPerson 'me') sparingly. 173:433 warning Consider removing 'really'. Microsoft.Adverbs 173:481 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 173:520 suggestion Consider using 'more' Microsoft.ComplexWords or 'extra' instead of 'additional'. 175:30 warning Don't use end punctuation in Microsoft.HeadingPunctuation headings. 177:210 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 177:237 warning Use first person (such as Microsoft.FirstPerson 'me') sparingly. 177:271 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 177:405 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 177:429 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 179:17 warning Consider removing 'fairly'. Microsoft.Adverbs 189:23 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 189:47 error Use 'don't' instead of 'do Microsoft.Contractions not'. 203:35 warning Try to avoid using Microsoft.We first-person plural like 'we'. 205:5 suggestion 'Configure Oxidized' Microsoft.Headings should use sentence-style capitalization. 207:34 warning Try to avoid using Microsoft.We first-person plural like 'we'. 207:40 suggestion 'be concerned' looks like Microsoft.Passive passive voice. 207:286 warning Try to avoid using Microsoft.We first-person plural like 'Let's'. 207:321 warning Consider removing 'very'. Microsoft.Adverbs 214:109 warning Try to avoid using Microsoft.We first-person plural like 'We'. 214:121 suggestion Consider using 'discuss' Microsoft.ComplexWords instead of 'address'. 214:205 warning Consider removing 'Very'. Microsoft.Adverbs 214:231 warning Try to avoid using Microsoft.We first-person plural like 'we'. 214:246 suggestion Verify your use of 'ensure' Microsoft.Vocab with the A-Z word list. 214:258 error Use 'we've' instead of 'we Microsoft.Contractions have'. 214:258 warning Try to avoid using Microsoft.We first-person plural like 'we'. 214:331 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 216:1 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 216:10 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 216:171 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 216:194 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 216:203 warning Consider removing 'really'. Microsoft.Adverbs 224:14 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 224:26 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 224:83 suggestion Verify your use of 'ensure' Microsoft.Vocab with the A-Z word list. 228:5 suggestion 'Configure LibreNMS for Microsoft.Headings Oxidized' should use sentence-style capitalization. 230:164 warning Try to avoid using Microsoft.We first-person plural like 'we'. 230:213 suggestion Verify your use of 'ensure' Microsoft.Vocab with the A-Z word list. 230:236 suggestion 'is mapped' looks like passive Microsoft.Passive voice. 243:16 warning For a general audience, use Microsoft.GeneralURL 'address' rather than 'URL'. 245:4 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 245:26 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 245:117 warning Consider removing 'very'. Microsoft.Adverbs 247:203 warning Consider removing 'very'. Microsoft.Adverbs 247:218 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 247:246 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 247:249 suggestion 'VPN' has no definition. Microsoft.Acronyms 250:40 suggestion 'was checked' looks like Microsoft.Passive passive voice. 250:146 suggestion 'was checked' looks like Microsoft.Passive passive voice. 250:184 suggestion Verify your use of 'as well Microsoft.Vocab as' with the A-Z word list. 250:210 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 250:284 suggestion 'was put' looks like passive Microsoft.Passive voice.✖ 4 errors, 91 warnings and 47 suggestions in 1 file.