Log Aggregation for Fun (Not Profit)
People of a certain age might remember The Log Song. It plays in my head every time I think, write, or say the word “log”. So this will be the soundtrack of the post. Now that the tone is set, let’s get logging.
Table Of Contents
Wazuuuuuuup!
So when I think, write, or say the word Wazuh I, along with people of a certain age, think about this Budweiser commercial. It also brings about a particular fondness for the communal nature of landlines. But Wazuh is not affiliated, associated, does not endorse, nor is endorsed by Budweiser. I haven’t looked into the proper pronunciation of “Wazuh” and, at this point, I’m not terribly interested in doing that because I want a little bit of mystery in my life.
Wazuh is well documented as being a good free and open-source SIEM and XDR product. You can get training and support and consultation from them for a price, so their product is not their software, but their services. A good model, in my opinion. You can only sell the software to someone once. Making good software and selling services on the reputation of it feels right. So let’s make some use of this good software.
Installation
This is not an installation post. It’s relatively straightforward, and is supported on Ubuntu 22.04 LTS, so that’s what I went with. There are a lot of considerations on how much storage and all that you’ll want, and those are genuine concerns, but in this homelab situation it’s not too complicated. I gave the VM a 512GB disk with 8GB of RAM, and I’ve followed the instructions. I dropped everything on the same host. I would say that it would be best to work with several people on just the care and feeding of Wazuh, as it may be a bit much for a solo effort in a production environment.
Configuration
Wazuh, by default, is not configured to intake syslog but there are agents. Installing said agents is generally a matter of clicking next on the UI and providing some info about your Wazuh server. Jump in to /var/ossec/etc/ossec.conf and add the following to listen for syslog:
<remote>
<connection>syslog</connection>
<port>514</port>
<protocol>tcp</protocol>
<allowed-ips>192.168.100.1/24</allowed-ips>
<local_ip>192.168.100.130</local_ip>
</remote>
The tag local_ip is going to be an interface on your Wazuh box. If you’re on Ubuntu, like I am, hit it with a systemctl restart wazuh-manager while you’re at it. For OPNsense we don’t actually need this, as there’s a plugin, but we’ll return to this later. First thing’s first is to get OPNsense hooked up.
Double 0 Agent
The agent for Wazuh is a very straightforward installation. In the Agent Management screen there is a wizard that will help you along when you click Deploy new agent.
This is definitely one of the better UI experiences I've had with security products.
Just one of various built-in compliance targets.
Plug Me In Coach
Installation is really straightforward. Don’t leave the browser window until it reports that it’s done. I don’t think having the browser window open actually matters, but the firmware UI is a little strange. At this point we are fully collecting info from the OPNsense router, which is very cool.
You can pick the logs and all the stuff. I’m not sure if the IDS logs will get picked up, but I’m still learning about that feature in OPNsense anyway.This reminds me to go through my DC and make sure I’m pickup up the AD DS, DHCP, and DNS logs. But right now I’m seeing a lot of blocks on the firewall from my ISP’s gateway. Lovely. It’s sending out multicast packets–probably looking for other associated devices. I appreciate the effort, but I don’t want this trivial activity showing up in Wazuh.
Suppressing Rules
From the event, we’re going to get the rule ID, which in this case is 87702. There’s a lot of info here, but we’re going to see if we can add conditions to suppress this rule. This may be inadvisable in a production environment, but we’re doing it for the labs. There are different ways to go about this, and I’ll take the most straightforward way first.
In local_rules.xml (in Wazuh), I’m going to add a new group for the rule IDs for blocks. Those IDs are 87700, 87701, and 87702. I’ll define the source IP as 192.168.1.1, the ISP router, and I’ll define the destination CIDR address of 224.0.0.0/4 which should handle all multicast space. I’ll then add this group to the local_rules.xml file and click “Restart” to cycle Wazuh Manager.
<group name="local,suricata_ignore,">
<rule id="100010" level="0">
<if_sid>87700, 87701, 87702</if_sid>
<srcip>192.168.1.1</srcip>
<dstip type="cidr">224.0.0.0/4</dstip>
<description>Ignoring Suricata stream events for multicast traffic from 192.168.1.1.</description>
</rule>
</group>
This will remove some of that noise from my OPNsense logs, but it won’t go back and remove the events that were already recorded. This is an ongoing project so I’m not too bent out of shape about that. And I’m seeing a few other messages that I may want to suppress in the future, and I’ll use the same technique as it’s working fine for me, for the moment.
It’s Big. It’s Heavy. It’s Wood.
So let’s get some more big heavy logs in here. I’m jumping into C:\Program Files (x86)\ossec-agent\ossec.conf which is a beefy XML file that configures the agent. There’s a metric ton of things–files to monitor for integrity, registry keys, the whole 9. But there are a few entries that look like this that we want to replicate, to add more logging for our Domain Controller:
<localfile>
<location>Security</location>
<log_format>eventchannel</log_format>
<query>Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and
EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and
EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and
EventID != 5152 and EventID != 5157]</query>
</localfile>
<localfile>
<location>System</location>
<log_format>eventchannel</log_format>
</localfile>
So the log format being eventchannel is a giveaway. We’re getting the whole of the System log but we’re filtering the Security log. And that’s good–the Security log is already obnoxious even with that filter in place. I am going to follow this guide to do that.
I’ve added several logs, and I’ll drop what I’ve added here:
<!-- added by mark -->
<localfile>
<location>Microsoft-Windows-DHCP Server Events\Admin</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-DHCP Server Events\FilterNotifications</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-DHCP Server Events\Operational</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-Windows Defender\Operational</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-DNS-Server\Audit</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-Sysmon\Operational</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Microsoft-Windows-Security-Netlogon\Operational</location>
<log_format>eventchannel</log_format>
</localfile>
<localfile>
<location>Directory Service</location>
<log_format>eventchannel</log_format>
</localfile>
<!-- done added by mark-->
It might have been a bit aggressive, as I’m seeing a significant increase in idle CPU on the DC, but it’s not so much that it’s a problem. This is a 3 vCPU VM on an Athlon FX. We’re not boiling the ocean here. Single-threaded performance is nothing to write home about. At the end of the day, we’re OK on CPU but not doing too hot on memory.
640k ought to be enough for anyone.
The volume of events has increased, and I’ll continue to monitor it. I’ve been having some issues with my workstation VM, WS01, and the fact that I’m on a MacBook Air most of the time, and not on a Windows computer to be able to connect to the console for that VM, but I’m definitely seeing an increase of events now that I’ve enabled all of these logs.
Yeah, that line goes up.
Denoument Is Probably Not How You Spell It
I’ll monitor this, and see how it goes. But this was a fun romp into log aggregation and really getting into the weeds on what needs to be done to make this collection process productive. There’s another side of this coin that is reporting and alerting, and we’ll get into that. Until then!
./content/posts/log-aggregation-for-fun-not-profit/index.md 1:1 suggestion You averaged 1.28 complex marktoso.Kiss words per sentence 9:118 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 9:136 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 9:169 error Punctuation should be inside Microsoft.Quotes the quotes. 9:238 suggestion 'is set' looks like passive Microsoft.Passive voice. 9:246 warning Try to avoid using Microsoft.We first-person plural like 'let's'. 12:14 warning Don't use end punctuation in Microsoft.HeadingPunctuation headings. 13:8 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 13:16 error More than 3 commas! marktoso.TresComas 13:68 warning Use first person (such as ' I, Microsoft.FirstPerson ') sparingly. 13:285 error Use 'isn't' instead of 'is Microsoft.Contractions not'. 13:316 error Use 'doesn't' instead of 'does Microsoft.Contractions not'. 13:338 suggestion 'is endorsed' looks like Microsoft.Passive passive voice. 13:364 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 13:442 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 13:450 warning Consider removing 'terribly'. Microsoft.Adverbs 13:491 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 13:526 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 15:65 suggestion 'SIEM' has no definition. Microsoft.Acronyms 15:74 suggestion 'XDR' has no definition. Microsoft.Acronyms 15:177 error Use 'isn't' instead of 'is Microsoft.Contractions not'. 15:237 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 15:379 warning Try to avoid using Microsoft.We first-person plural like 'let's'. 18:6 error Use 'isn't' instead of 'is Microsoft.Contractions not'. 18:72 suggestion 'is supported' looks like Microsoft.Passive passive voice. 18:101 suggestion 'LTS' has no definition. Microsoft.Acronyms 18:120 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 18:134 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 18:303 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 18:442 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 18:481 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 18:482 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 21:20 error Use 'isn't' instead of 'is Microsoft.Contractions not'. 21:103 warning Consider removing 'generally'. Microsoft.Adverbs 31:92 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 31:180 warning Try to avoid using Microsoft.We first-person plural like 'we'. 31:234 warning Try to avoid using Microsoft.We first-person plural like 'we'. 33:4 suggestion 'Double 0 Agent' should use Microsoft.Headings sentence-style capitalization. 33:13 warning Prefer 'personal digital Microsoft.Terms assistant' over 'Agent'. 34:5 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 34:26 warning Consider removing 'very'. Microsoft.Adverbs 34:68 warning Prefer 'personal digital Microsoft.Terms assistant' over 'Agent'. 34:162 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 36:39 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 36:136 suggestion Verify your use of Microsoft.Vocab 'actionable' with the A-Z word list. 36:187 suggestion Consider using 'part' instead Microsoft.ComplexWords of 'portion'. 36:203 suggestion 'GRC' has no definition. Microsoft.Acronyms 37:17 suggestion 'CIS' has no definition. Microsoft.Acronyms 40:4 suggestion 'Plug Me In Coach' should use Microsoft.Headings sentence-style capitalization. 40:9 warning Use first person (such as Microsoft.FirstPerson 'Me') sparingly. 41:17 warning Consider removing 'really'. Microsoft.Adverbs 41:104 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 41:223 warning Try to avoid using Microsoft.We first-person plural like 'we'. 41:223 error Use 'we're' instead of 'we Microsoft.Contractions are'. 41:287 warning Consider removing 'very'. Microsoft.Adverbs 43:42 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 43:42 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 43:62 suggestion 'IDS' has no definition. Microsoft.Acronyms 43:95 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 43:150 error 'y.T' should have one space. Microsoft.Spacing 43:165 warning Use first person (such as Microsoft.FirstPerson 'me') sparingly. 43:182 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 43:202 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 43:227 suggestion 'DHCP' has no definition. Microsoft.Acronyms 43:237 suggestion 'DNS' has no definition. Microsoft.Acronyms 43:261 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 43:309 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 43:312 suggestion 'ISP' has no definition. Microsoft.Acronyms 43:446 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 45:5 suggestion 'Suppressing Rules' should use Microsoft.Headings sentence-style capitalization. 46:17 warning Try to avoid using Microsoft.We first-person plural like 'we'. 46:112 warning Try to avoid using Microsoft.We first-person plural like 'we'. 46:134 warning Try to avoid using Microsoft.We first-person plural like 'we'. 46:236 warning Try to avoid using Microsoft.We first-person plural like 'we'. 48:34 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 48:185 suggestion 'ISP' has no definition. Microsoft.Acronyms 48:229 suggestion 'CIDR' has no definition. Microsoft.Acronyms 48:234 suggestion Consider using 'discuss' Microsoft.ComplexWords instead of 'address'. 59:42 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 59:161 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 59:203 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 59:207 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 59:243 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 59:339 warning Use first person (such as Microsoft.FirstPerson 'me') sparingly. 61:4 suggestion 'It's Big. It's Heavy. Microsoft.Headings It's Wood.' should use sentence-style capitalization. 61:11 warning Don't use end punctuation in Microsoft.HeadingPunctuation headings. 61:23 warning Don't use end punctuation in Microsoft.HeadingPunctuation headings. 61:34 warning Don't use end punctuation in Microsoft.HeadingPunctuation headings. 62:4 warning Try to avoid using Microsoft.We first-person plural like 'let's'. 62:48 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 62:95 warning Prefer 'personal digital Microsoft.Terms assistant' over 'agent'. 62:207 suggestion Consider using 'check' or Microsoft.ComplexWords 'watch' instead of 'monitor'. 62:311 warning Try to avoid using Microsoft.We first-person plural like 'we'. 62:357 warning Try to avoid using Microsoft.We first-person plural like 'our'. 78:55 warning Try to avoid using Microsoft.We first-person plural like 'We'. 78:101 warning Try to avoid using Microsoft.We first-person plural like 'we'. 78:221 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 117:1 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 117:41 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 117:177 warning Try to avoid using Microsoft.We first-person plural like 'We'. 117:294 warning Try to avoid using Microsoft.We first-person plural like 'we'. 120:58 suggestion Consider using 'check' or Microsoft.ComplexWords 'watch' instead of 'monitor'. 120:70 suggestion Try to keep sentences short (< Microsoft.SentenceLength 30 words). 120:104 warning Use first person (such as Microsoft.FirstPerson 'my') sparingly. 120:121 error More than 3 commas! marktoso.TresComas 120:147 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 120:270 warning Use first person (such as Microsoft.FirstPerson 'I'm') sparingly. 120:336 warning Consider using 'these' instead Microsoft.Wordiness of 'all of these'. 123:6 suggestion 'Denoument Is Probably Not Microsoft.Headings How You Spell It' should use sentence-style capitalization. 124:6 suggestion Consider using 'check' or Microsoft.ComplexWords 'watch' instead of 'monitor'. 124:90 warning Consider removing 'really'. Microsoft.Adverbs 124:137 suggestion 'be done' looks like passive Microsoft.Passive voice. 124:223 error Use 'that's' instead of 'that Microsoft.Contractions is'. 124:259 warning Try to avoid using Microsoft.We first-person plural like 'we'. 126:40 suggestion 'was checked' looks like Microsoft.Passive passive voice. 126:146 suggestion 'was checked' looks like Microsoft.Passive passive voice. 126:184 suggestion Verify your use of 'as well Microsoft.Vocab as' with the A-Z word list. 126:210 warning Use first person (such as ' I Microsoft.FirstPerson ') sparingly. 126:284 suggestion 'was put' looks like passive Microsoft.Passive voice.✖ 11 errors, 70 warnings and 38 suggestions in 1 file.