How to Setup Brocade L3 Switch to Work with PfSense Router

 In this post, I will explain how to setup a Brocade ICX 6610 L3 switch to route traffic in LAN/VLAN at full line rate (1G/10G/40Gbps) without going up to the PfSense firewall/router, and when the connected devices need to get on the Internet, the Brocade switch will forward traffic to PfSense for it to handle Internet traffic.

 If you are familiar with VLANs, you probably know that in order to route traffic between VLANs, you'd need a router. A normal consumer grade router can typically route traffic at 1Gbps, or 125MB/s. For normal home network, this is more than adequate, since home connections rarely exceed 1Gbps, and connected devices often don't have a NIC that's capable in excess of 1G. Although nowadays 2.5G on-board NICs are becoming more than more common, 2.5G switches are still quite expensive.

 Last summer I got into this "homelab" lobby, and I enjoy every minute of tinkering my network and VMs. After reading this STH post, I decided to get myself one of these Brocade ICX 6610 switch. It packs 48 POE 1G ports, 8 x 10G SFP+ ports, and 2 x 40G QSFP+ ports. According to the spec sheet, this switch can route traffic at line speed, regardless of features enabled - meaning, if configured correctly, 10G ports will be running at 10G speed, 40G ports will be running at 40G speed, etc.

 So, when I got my hands on the switch, I immediately begins to tinker with VLANs, since I desperately need to separate all the VMs from my main network. Also, with security cameras installed, I don't want them to be on my main network, either - they should be locked down, accepting no connection besides from a few allowed client devices, and absolutely forbids internet connection.

 This is what my network looked like:

 As you can see, I connected my Brocade switch to PfSense using 1G link (black arrows). Also, I created 5 VLANs on both my switch and PfSense. VLAN 10 is my main network, connecting with a QNAP switch that powers all the computers. VLAN 20 is my VM network, where my servers and containers reside. VLAN 30 is my storage network, with two Synology NAS and an Unraid NAS. These two networks are utilizing that 40G connection (green arrows). VLAN 40 is my camera network, and VLAN 99 is my IOT network. Both are what I consider attack surfaces and they are not allowed to initiate connection to any other network, and their firewall policies are pretty aggressively locked down to provide the absolute minimum amount of connection while maintaining functionality of those devices.

 This setup worked flawlessly, at least that's what I thought, until I found out that none of these VLANs can communicate with each other at "line speed". After some research, I quickly realized that 1G link between my switch and PfSense is limiting bandwidth. However, there's more to that.

 Remember I mentioned that this Brocade switch can do line speed if configured correctly? It turns out, L3 switches are capable of routing within the switch itself, all I have to do is to setup "Virtual Interfaces" on each of the VLAN that I created. I quickly googled all the procedures to get them up and running and pointed all of my servers and workstations to the virtual interfaces inside of the switch. Viola! my network is running at line speed!

 But things took a turn from there. You see, with Brocade's virtual interfaces turned on, I basically got myself two sets of routers, one is for the VLANs, and the other is PfSense, which serves a very important purpose - providing Internet access.

 I quickly realized that if I point my computer's default gateway to the Brocade, I will lose internet connection, unless I provide a default route to the PfSense router. However, if this default route is set, the traffic returning from my other VLANs will be blocked by the PfSense firewall because of something called Asymmetric Routes. I tried all the fixes I can find for this problem, even installed some packages on my PfSense, none of which seemed to work. So, I started to look for a solution.

 After some extensive research, I learned from this post from STH that I need to setup something called a "transit" network. This network is nothing special, it is just a middleman between the PfSense and the Brocade.

 Please note, if you are modifying your existing network, rather than starting from scratch, you have to mindful that you might lock yourself out. To prevent this, I suggest the following:

   a. Always keep a backup connection that you can talk to the switch/firewall.

   b. If you have to remove interfaces from PfSense to set up static routing,     move your existing connection to one of the already configured network     first.

   c. the biggest risk in my opinion, is being locked out by PfSense. If your     PfSense instance is virtualized, you can always revert to the last backup. If     you don't have one, make one before continuing on.

 If you are prepared to carry on the task, here are the steps I took.

1. Setup a dedicated link (preferably a new one) from PfSense to the switch. I used a random 1G port to connect to one of the ethernet ports my server has, passed through to my virtualized PfSense.

 In the switch, create a VLAN.

brocade$ enable 			#just like sudo -i
brocade$ conf t  			#enter config terminal
brocade(config)$ vlan 200 		#create VLAN 200

3. Create a Virtual Interface for this VLAN.

(config-vlan-200)$ router-interface ve 200
(config-vlan-200)$ exit

##I strongly recommend setting the VE# same as your VLAN#

4. Assign an IP address to this Virtual Interface.

(config)$ int ve 200	  		 #VE 200 configuration
(config-vif-10)$ ip address 172.16.1.1/24  #Assign an IP address to VE 200
(config-vif-10)$ exit

5. Now let's go back to PfSense. Under "Interfaces-Assignments", add an interface attached to the Brocade connection and name it as "Transit". Then, give it an IP address of 172.16.1.2/24. After that, try to ping the switch from PfSense. If everything goes right, you should be able to get a response from the switch, indicating a good connection. Also, you may or may not need to adjust your firewall policy on the "Transit" interface to allow traffic between PfSense and Brocade. With the transit part done, we will move on to the next part, which is traffic segregation and routing.

6. This is step, we will setup a gateway so PfSense can route traffic to the Brocade. In PfSense, go to "System - Routing", you should be in "Gateways" tab. Click "Add", select "Transit" as the interface, give it a meaningful name like "Transit_Gateway", give it an IP address of 172.16.1.1, which is pointing to our switch. With all that done, click "Save".

7.  We need to create "Static Routes" on PfSense so PfSense can be aware of how to get to our VLANs. Go to "System-Routing", select "Static Routes" tab, then create static routes going to all the VLAN subnets. Click "Add", in "Destination Network" box, put in your VLAN network such as 192.168.10.0/24, select "Transit_Gateway" as the gateway, and give it a descriptive description.

8.  We will go back to Brocade terminal for this part. We will now setup a default route, which is used when the switch is not aware of the destination address network, so it will have to forward the packets to someone that knows how to handle them, which is PfSense. The goal here is to segregate local traffic and outbound Internet traffic so the local traffic doesn't go to PfSense, while sending Internet traffic to it. With the help of one single command, we can now route local traffic within the switch at line speed, while flawlessly utilizing the transit network to avoid asymmetric routing problems.

(back to config terminal)
(config)$ ip default-network 172.16.1.2

## This is the PfSense interface on Transit Network

9. We are very close to the end of the tunnel. To complete the last step, we need to adjust some firewall settings in PfSense. Just for testing purposes, we will make our transit network wide open. PfSense, being a stateful firewall, it knows where a packet came from, and where it is going to. So, in PfSense, go to "Firewall - Rules", select "Transit" interface. The first rule I'd add is to allow from your core VLAN to access PfSense.

[Source] 192.168.10.0/24
[Port] *
[Destination] This Firewall
[Port] "your PfSense Web Port"

## Anti Lockout

10. Then, we will allow traffic from VLAN subnets to use Transit interface to get out to the Internet.

 That's all the configuration I've done to get my Brocade and PfSense working together happily.

 If you followed all the steps to here, congratulations! You have set up a transit network between your router and L3 switch with a default route to segregate local and Internet traffic, keeping the best from both worlds. To validate your setup is working, use a device in the completed subnet, and try a traceroute to 1.1.1.1, Cloudflare's DNS server:

 As you can see, Internet traffic is going through the PfSense router in the transit network.

 Let's validate local traffic, too:

 No external hop, just routing inside the switch.

 Finally, if you are happy with the results, write the configuration to memory so it persists:

(config)$ write memory

## writes config to ROM

 This is just a general guide to get my Brocade ICX 6610 to operate at its best while keeping the benefit of having PfSense standing between me and the outside world. Configurations and methods can vary from brand to brand, and my implementation is definitely by no means the best, but what worked for me.

 I hope this guide is helpful to you. If you like to see more content like this, you can subscribe for free, so you will be notified when new articles are posted.