Traffic not flowing for hosts behind NAT - Edge Router Lite

[EddieJennings]

This weekend I failed to deploy my ERLs, because of a traffic issue occurring with one of the two.

ERL1 = local office ERL. I was able to test the configuration and found it to function with no problems. Traffic passed fine to the devices with 1-to-1 NAT

ERL2 = data center ERL. Due to [reasons] it was not possible to test this prior to the deployment attempt; thus, I mimicked the configuration of the known, working ERL1 as much as I could, obviously changing IP addresses as necessary. Traffic for hosts that weren't involved with 1-to-1 NAT worked fine. Traffic for hosts that had 1-to-1 NAT failed, which makes me think the issue is with how I've configured NAT.

Configuration caveats:

• Security through obscurity: I've altered the IP addresses of the hosts and interfaces, but they should be valid for the /24 and /28 networks involved.
• The old Cisco ASA had two interfaces in the outside-VLAN with one address assigned to them. eth1 will eventually be that second interface.
• I was so focused on just getting Internet connectivity to all of the hosts, I did not test the success of the site-to-site VPN.
• All configurations (except the obscuring of the IP addresses) were done using the GUI.

This is the configuration of the ERL2 as it was Saturday night.

Here's the current question: Do you folks see a problem with the NAT configuration? There should be 1-to-1 NAT for the IIS server and PostFix server. All other hosts (SQL server, etc.) use NAT-overload as its destination NAT.

firewall {
    all-ping enable
    broadcast-ping disable
    ipv6-receive-redirects disable
    ipv6-src-route disable
    ip-src-route disable
    log-martians enable
    name WAN_IN {
        default-action drop
        description "WAN to internal"
        rule 10 {
            action drop
            description "Deny traffic from 74.6.168.184"
            log disable
            protocol all
            source {
                address 74.6.168.184
            }
        }
        rule 20 {
            action drop
            description "Deny traffic from 23.253.40.182"
            log disable
            protocol all
            source {
                address 23.253.40.182
            }
        }
        rule 30 {
            action drop
            description "Deny traffic from 72.30.14.0/24"
            log disable
            protocol all
            source {
                address 72.30.14.0/24
            }
        }
        rule 40 {
            action accept
            description "Allow HTTP to IIS"
            destination {
                address 172.16.254.12
                port 80
            }
            log disable
            protocol tcp
        }
        rule 50 {
            action accept
            description "Allow HTTPS to IIS"
            destination {
                address 172.16.254.12
                port 443
            }
            log disable
            protocol tcp
        }
        rule 60 {
            action accept
            description "Allow FTP to IIS"
            destination {
                address 172.16.254.12
                port 21
            }
            log disable
            protocol tcp
        }
        rule 70 {
            action accept
            description "Allow SMTP to PostFix"
            destination {
                address 172.16.254.14
                port 25
            }
            log disable
            protocol tcp
        }
        rule 80 {
            action accept
            description "Allow SMTP-SSL to PostFix"
            destination {
                address 172.16.254.14
                port 465
            }
            log disable
            protocol tcp
        }
        rule 90 {
            action accept
            description "Allow SMTP-MSA to PostFix"
            destination {
                address 172.16.254.14
                port 587
            }
            log disable
            protocol tcp
        }
        rule 100 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 110 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    name WAN_LOCAL {
        default-action drop
        description "WAN to router"
        rule 10 {
            action accept
            description "Allow established/related"
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action drop
            description "Drop invalid state"
            state {
                invalid enable
            }
        }
    }
    receive-redirects disable
    send-redirects enable
    source-validation disable
    syn-cookies enable
}
interfaces {
    ethernet eth0 {
        address 65.254.219.164/28
        address 65.254.219.165/28
        address 65.254.219.166/28
        description Internet
        duplex auto
        firewall {
            in {
                name WAN_IN
            }
            local {
                name WAN_LOCAL
            }
        }
        speed auto
    }
    ethernet eth1 {
        address 172.16.100.1/24
        description "Local 2"
        duplex auto
        speed auto
    }
    ethernet eth2 {
        address 172.16.254.1/24
        description Local
        duplex auto
        speed auto
    }
    loopback lo {
    }
}
service {
    dns {
        forwarding {
            cache-size 150
            listen-on eth1
            listen-on eth2
        }
    }
    gui {
        http-port 80
        https-port 443
        older-ciphers enable
    }
    nat {
        rule 1 {
            description "Outside to IIS"
            destination {
                address 65.254.219.165
            }
            inbound-interface eth0
            inside-address {
                address 172.16.254.12
            }
            log disable
            protocol all
            type destination
        }
        rule 2 {
            description "Outside to PostFix"
            destination {
                address 65.254.219.166
            }
            inbound-interface eth0
            inside-address {
                address 172.16.254.14
            }
            log disable
            type destination
        }
        rule 5000 {
            description "IIS to Outside"
            log disable
            outbound-interface eth0
            outside-address {
                address 65.254.219.165
            }
            protocol all
            source {
                address 172.16.254.12
            }
            type source
        }
        rule 5001 {
            description "PostFix to Outside"
            log disable
            outbound-interface eth0
            outside-address {
                address 65.254.219.166
            }
            protocol all
            source {
                address 172.16.254.14
            }
            type source
        }
        rule 5002 {
            description "masquerade for WAN"
            outbound-interface eth0
            type masquerade
        }
    }
    ssh {
        port 22
        protocol-version v2
    }
    unms {
        disable
    }
}
system {
    gateway-address 65.254.219.161
    host-name erl-peak10
    login {
        user SomeUser {
            authentication {
                encrypted-password GIBBERISH.
            }
            level admin
        }
    }
    name-server 8.8.8.8
    ntp {
        server 0.ubnt.pool.ntp.org {
        }
        server 1.ubnt.pool.ntp.org {
        }
        server 2.ubnt.pool.ntp.org {
        }
        server 3.ubnt.pool.ntp.org {
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone America/New_York
    traffic-analysis {
        dpi enable
        export enable
    }
}
vpn {
    ipsec {
        auto-firewall-nat-exclude enable
        esp-group FOO0 {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        }
        ike-group FOO0 {
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 14
                encryption aes128
                hash sha1
            }
        }
        site-to-site {
            peer 50.241.10.49 {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret GIBBERISH
                }
                connection-type initiate
                description Office
                ike-group FOO0
                ikev2-reauth inherit
                local-address 172.16.254.1
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group FOO0
                    local {
                        prefix 172.16.254.0/24
                    }
                    remote {
                        prefix 172.16.253.0/24
                    }
                }
            }
        }
    }
}


/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "config-management@1:conntrack@1:cron@1:dhcp-relay@1:dhcp-server@4:firewall@5:ipsec@5:nat@3:qos@1:quagga@2:system@4:ubnt-pptp@1:ubnt-unms@1:ubnt-util@1:vrrp@1:webgui@1:webproxy@1:zone-policy@1" === */
/* Release version: v1.9.7+hotfix.4.5024004.171005.0403 */

[JaredBusch]

This means you messed up the source and/or destination NAT rules.

I am on my phone and cannot look at this carefully.

I can also post examples of how to configure this from a working site.

[EddieJennings]

@jaredbusch said in Traffic not flowing for hosts behind NAT - Edge Router Lite:

This means you messed up the source and/or destination NAT rules.

I am on my phone and cannot look at this carefully.

I can also post examples of how to configure this from a working site.

I agree. I’ll look at it again after teaching tonight. I’m sure it’s staring back at me and for some reason I’m blind.

[EddieJennings]

To check my understanding of NAT.

IIS Server

• Public: 65.254.219.165
• Private: 172.16.254.12

PostFix Server

• Public: 65.254.219.166
• Private: 172.16.254.14

Static 1:1 NAT is what's needed, and I'd have two rules for each host.

Rule 1 would be destination NAT. A packet enters the eth0 interface bound for the public IP address for the intended server. The ERL translates the destination address of this packet to the private IP address of the intended server.

Rule 2 would be a source NAT. A pack exits the eth0 interface bound for the Internet from a server with the source address as the server's private IP address. The ERL translates the source address of the pack to the public IP address of the server.

So the above example for the IIS server would look like this.

Rule 1. If a packet's destination is 65.254.219.165 is ingress to eth0, translate that packet's destination to 172.16.254.12, then send packet on to have firewall rules checked.

Rule 2. If a packet's source is 172.16.254.12 is to egress from eth0, translate that packet's source to 65.254.219.165, then send the packet on its way.

[JaredBusch]

going to my office in a few minutes. will be able to help.

[JaredBusch]

Delayed by kids. Putting them to bed now though

[EddieJennings]

@jaredbusch said in Traffic not flowing for hosts behind NAT - Edge Router Lite:

Delayed by kids. Putting them to bed now though

Worry not. I’m turning early myself. As always, I appreciate the wisdom whenever it comes.

[JaredBusch]

@eddiejennings said in Traffic not flowing for hosts behind NAT - Edge Router Lite:

@jaredbusch said in Traffic not flowing for hosts behind NAT - Edge Router Lite:

Delayed by kids. Putting them to bed now though

Worry not. I’m turning early myself. As always, I appreciate the wisdom whenever it comes.

They are going to bed. Not me.

anyway. post deluge incoming.

[JaredBusch]

Dashboard, view showing eth0 (WAN port)

[JaredBusch]

The Port forwarding rules only work against the first IP listed on the specified port.

In this case the .42 on eth0
FYI: .1.14 is the internal Exchange server and .0.20 is the HAProxy server for Exchange. I am only proxying SSL.

[JaredBusch]

I have a few firewall groups setup

[JaredBusch]

The only LAN firewall rules are to block SMTP.

[JaredBusch]

For anything you are going to NAT in to something other than the first IP address on the default masquerade, you have to have both firewall rules and NAT destination rules.

For anything that you want to go out something other than the default masquerade, you have to have a NAT source rule.

This is the WAN_IN and is likely part of your problem.

This is the NAT (source and destination) rules that are the other part of your problem.

[JaredBusch]

For the screenshots above, the default NAT traffic will show a "What is my IP" as 12.XXX.XXX.42 as that is the first IP listed in the config of the eth0 port.

All traffic goes out through the default NAT masquerade (shown as order 4 in the Source NAT section) unless there is a Source NAT rule to override it.

As I have more than one thing happening here, we will just focus on the Nginx ones as the stuff not going through the default masquerade.

A Source NAT is used for outbound traffic. You are the source of the traffic.
A Destination NAT is used for inbound traffic. You are the destination of the traffic.

So to send traffic from the server with an internal IP address of 10.202.1.16 out the external IP address of 12.XXX.XXX.43 instead of the default masquerade external IP of 12.XXX.XXX.42 we need to make a Source NAT rule.

Make a rule and fill it out like this. Obviously, there are a lot more options here than I am going over. If you have multiple machines that you need to use this IP outbound, then create a firewall group and select the Src Address Group drop down instead of specifying the IP address as I did here.

Now if you check your What is my IP from the specified server, it should return the 12.XX.XXX.43

[JaredBusch]

The inbound is harder because you have to setup both a Destination NAT rule as well as allow it on the firewall's WAN_IN rule.

First make the Destination NAT rule as you are already on this screen from making the Source NAT rule.

As you can see you simply reverse what you did in the Source NAT rule. This time the local IP goes in the Translation and the WAN IP goes in the Dest Address field. Anything coming in on this destination IP will be translated to this local IP.

As you can also see, I further restricted this translation to only be TCP/UDP and only ports included in a firewall port group.

In case you are curious, here are the ports in that firewall port group.

[JaredBusch]

Now make a firewall rule in the policy assigned to the IN direction of your WAN interface.

The wizards name this rule WAN_IN by default.

You want to make the settings match when it comes to the protocol and port settings. But the destination is now the internal IP address as the translation has already happened by the NAT rules before the firewall rules see it.

[JaredBusch]

Now you should have traffic properly flowing to and from your alternate IP addresses.
And yes, I noticed..

[Mike Davis]

Well done Jared.

[Dashrender]

When I tried this last night, my new NAT rules were all below the default masquerade one. I tried moving (click and drag) above but it wouldn't actually move.

I then added a third rule (just some fake crap), then upon having three rules I was able to move my desired rule above the default one.

I'm on firmware v1.9.7-hotfix.4

[EddieJennings]

@jaredbusch said in Traffic not flowing for hosts behind NAT - Edge Router Lite:

Now you should have traffic properly flowing to and from your alternate IP addresses.

Thanks for the above. I'm comparing that to my configuration now.

And yes, I noticed..

Ah, then you know the commercials.