Recently a colleague of mine dropped a piece of work with [WIP] tag „on my desk” and run away to celebrate at company’s Kick-Off. Last thing I’ve heard was: „rsyslog is beating me into submission”. Turns out it wasn’t rsyslog problem at all! Note: vm is running Centos 6 and rsyslog 7.x.

vShield Edge Gateway logs

There’s a Skyscape KB article on how to collect aforementioned logs. Content is as follows:

The vShield Edge is globally configured to send syslog messages to a specific IP address. By configuring a new network within your tenancy and deploying a collector with the specified IP address you can access your gateway syslog data.

Step 1
Configure a new routed network with following details.

Parameter Value
Gateway 100.127.255.249
Subnet Mask 255.255.255.248
DNS <customer defined>
Static IP pool 100.127.255.250-100.127.255.254

Step 2
Add new server or an additional interface on your monitoring server with the address:

100.127.255.250

vShield gateway syslog entries can now be captured.

Odd that dest IP is hardcoded into what’s looking like totally random value, but instructions are easy to follow – do that. At this stage you should have two important questions rambling around in your head – TCP or UDP and is it on default port?
Docs don’t say that, you have to either trust me or check yourself but they are sent using UDP on default 514 port.

Ok, you’ve got the vSE configured to send logs, and you’ve got the server attached to proper network that’s ready to receive. Time to set up rsyslog. Let’s assume it’s vanilla rsyslog config for the sake of simplicity.

Create new conf file in /etc/rsyslog.d/ – I’ll use 1-vsecapture.conf

module(load="imudp")
ruleset(name="vselogs"){
action(type="omfile" file="/var/log/vse_logs.log")
}
input(type="imudp" port="514" ruleset="vselogs")

Above loads UDP input module, defines ruleset that writes everything to a specified file and then defines input stream using UDP on port 514 that will use it.

# service rsyslog restart

You’d expect to see them lines in /var/log/vse_logs.log, do you? I did too, but nothing showed up.

Troubleshooting phase #1

I had a secound of doubt in my rsyslog-fu, so I’ve run rsyslog in debug mode, but still nothing coming from remote was there.
That’s the part when I realised it has nothing to do with rsyslog itself.

What if there is nothing coming in from that Edge Gateway? Tcpdump should be able to tell.

# tcpdump -p -c 5 -i eth1 -w dump.pcap

Definitely receiving traffic and as it’s on a dedicated network created according to KB I was confident it’s from VSE even without looking at the dump. But here’s a sample entry:

Frame 10: 334 bytes on wire (2672 bits), 334 bytes captured (2672 bits)
Ethernet II, Src: Vmware_9c:2e:2a (00:50:56:9c:2e:2a), Dst: Vmware_01:db:5f (00:50:56:01:db:5f)
Internet Protocol Version 4, Src: 172.26.60.177 (172.26.60.177), Dst: 100.127.255.250 (100.127.255.250)
User Datagram Protocol, Src Port: 42968 (42968), Dst Port: 514 (514)
[truncated]Syslog message: KERN.WARNING: May 20 16:33:20 vse-LONG_ID_HERE firewall[]: [LONG_ID_HERE]: ACCEPT_131073IN= OUT=vNic_1 SRC=10.20.20.60 DST=10.20.10.111 LEN=60 TOS=0x00 PREC=0x00

Next thought was: maybe it’s IPtables? – I am sure Joe have thought about that and added appropriate rules, but it doesn’t hurt to check.

# iptables -v -n -L

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * * 100.127.255.248/29 0.0.0.0/0 multiport ports 514 /* 210 allow rsyslog in on default port 514 */
0 0 ACCEPT udp -- * * 100.127.255.248/29 0.0.0.0/0 multiport ports 514 /* 211 allow rsyslog in on default port 514 */
0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 999 drop all */ reject-with icmp-port-unreachable

Bingo! There’s the rule that allows UDP on 514 but only from 100.127.255.248/29 subnet.

Troubleshooting, phase #2

Hey, what’s wrong with that? KB said to create such network, so why is it wrong?
Let’s take a look again at the tcpdump output:

Internet Protocol Version 4, Src: 172.26.60.177 (172.26.60.177), Dst: 100.127.255.250 (100.127.255.250)

Destination address is correct, but what is the source IP? 172.26.60.177 – yes, it’s not in the same dedicated subnet, like one would expect.
Logs are being sent from the same network that vSE is attached to. Oh Skyscape, why, oh why?! But anyway, let’s turn off firewall at all and see if that’s the case.

# service iptables stop

Where are my logs?! Still nothing, the file hasn’t been even created.

Epiphany

Time to act like a proper IT pro – what does the Google say? But what to search for? I know that tcpdump is closer to the wire than iptables, but is there anything else in front of user application like rsyslog? I stumbled upon this question on ServerFault, which mentioned rp_filter (check this link for more details) .

The same way Joe couldn’t know that iptables rule has to allow different IP in, he couldn’t know that routing table would also have to be altered as:

# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
100.127.255.240 0.0.0.0 255.255.255.240 U 0 0 0 eth1
10.20.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
10.0.0.0 10.20.10.1 255.0.0.0 UG 0 0 0 eth0
0.0.0.0 10.20.10.1 0.0.0.0 UG 0 0 0 eth0

there indeed is different route assigned for going back, hence packets were discarded.
It all boils down to two settings in /etc/sysctl:

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2

They were set to 1 previously, even though the default is 0. That’s because of hardening policies we apply to all our boxes in that environment. Because there’s quite a lot of them it’s easy to forget that some were set.
After putting in place aforementioned solution
(you can do this without rebooting by issuing:

# echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
# echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

)

Success

I got the logs flowing like there’s no tomorrow! Voila. I’d also recommend is to change this setting from 1 to 2 only on the interface you’re going to receive the logs.

As stated in the first paragraph there was not a single rsyslog problem :-) I’ve also sent an email to Skyscape cloud support with my suggestions on how to improve that KB article everything started with.