Stuck routing a vlan over Same interface with netplan in Ubuntu


Recommended Posts

I have a kubernetes server running ubuntu 20.04, using a 10Gbit link to connect to my switch.  It has an ip address of 192.168.4.x, and the switch is an access port for vlan 4 (.4.).   This works great, but All my NFS traffic goes across my firewall, bogging it down.  I'd like to  move NFS traffic off to 192.16.1.x, leaving 4.x for kubernetes traffic only.  If it helps, I'm using BGP as well... here are my subnets:

 

Default Lan: 192.168.1.0/24
Kubernetes hosts: 192.168.4.0/27
Kubernetes pods: 192.168.50.0/24

 
Storage: 192.168.1.110
Kubernetes host: 192.168.4.10

I am using BGP courtesy of metallb to advertise 192.168.5.0/24 out to the rest of my network via 192.168.4.10.   I'd like to maintain this, while having only storage-nfs traffic go over a 192.168.1.x address to the NFS server at 192.168.1.110.  

I am having trouble wrapping my head around how to do this with netplan, would something like this work?

 

network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0f0:
      dhcp4: no
      addresses: [192.168.1.112/24]
      gateway4: 192.168.1.1
      nameservers:
          addresses: ["1.1.1.1", "8.8.8.8"]
  vlan4:
      id: 4
      link:  enp3s0f0
      dhcp4: no
      addresses: [192.168.4.10/27]
      routes:
        - to: 192.168.4.0/27
          via: 192.168.4.1
          on-link: true


?  If yes, would I then need to also add a static route in my firewall to tell everything else that if it wants to talk to 192.168.4.10, it needs to go to 192.168.1.112 (the Ip assigned to the non-tagged part of the interface)?  or would i need to do something like this:

 

network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0f0:
      dhcp4: no
      addresses: [192.168.1.112/24]
      gateway4: 192.168.1.1
      nameservers:
          addresses: ["1.1.1.1", "8.8.8.8"]
      routes:
             - to: 192.168.1.0/24
                 via: 192.168.1.1
                 table: 101
            routing-policy:
             - from: 192.168.1.0/24
                 table: 101
  vlan4:
      id: 4
      link:  enp3s0f0
      dhcp4: no
      addresses: [192.168.4.10/27]
      gateway4: 192.168.4.1
      routes:
       - to: 0.0.0.0/0
         via: 9.9.9.9
          on-link: true

?

Basically I'm trying to have the vlan 4 interface be the default route and have all inbound/outbound traffic go over it, with the untagged part of the interface be only for storage traffic (NFS), since .1 is a protected vlan I don't think i can assign that to a vlan interface.  These two IPs will exist on the same physical 10Gb interface.     Thanks in advance!

Not exactly sure what your trying to accomplish here.. Oh I get not wanting nfs traffic to route.. So you want a san (storage area network)..

 

So create a san - not sure what your trying to do with routing.. If .110 and .112 are in the same L2.. Then they would use that address and interface to talk to the other IP in their network.

 

If you want this L2 your using to be native (untagged) that is fine.. You just need to make sure that .110 and .112 are in the same L2.   And this same network can also carry your tagged vlan 4 traffic..  There is no reason to route this 192.168.1 network for devices directly attached to it.

 

Normally when setting up a san, you would not set a default gw on this interface on the devices.  If you want other other networks to talk to the 192.168.1 network that are not actual members of the 192.168.1 network - then you could setup up routes on the host they want to talk to telling them how to get back to the source network.

 

But normally you would just talk to the host that is multihomed via its non san IP.. So that san would be left only isolated and not have any way to route to or from..

 

I have a somewhat related setup... My PC normal network is 192.168.9/24 and my NAS is also on this 192.168.9 network... But this is limited to gig..  I do not have any switch capable of more than gig interfaces.  But I wanted my pc and nas to talk at 2.5gig.. This was done via simple usb interface added to my pc and my nas.. This uses a different network, or san - 192.168.10/24

 

All of the other networks use the nas 192.168.9 network to talk to it.  But my pc uses the 192.168.10.x address - so all traffic to and from my PC talking to the nas via file sharing protocols use 192.168.10 address.  But when I want to manage the nas, talk to it on other protocols I use the 192.168.9 address.

 

There is no routing at all of the 192.168.10 no gateways on either the pc or the nas for this network.. My router doesn't even know about.. etc..

  • Like 1

@BudMan  Thanks for the reply, yeah basically I'm trying to not route NFS traffic inter-vlan

 

I have the following:
VLAN 1(default) = normal LAN traffic
VLAN 4 = kubernetes traffic   

 

Although irrelevant, Storage/K8s/Firewall all link to switch at 10Gb.     As is right now, any pod (or the K8s Node) doing any NFS traffic mounts a 192.168.1.110:/whatever mount to it, meaning that NFS comes across 192.168.1.x <->192.168.4.x, traversing the firewall.  With the hardware firewall I have (Palo Alto networks PA-3020), this isn't an issue since the ASIC in charge of the dataplane can handle it, but I'm moving to a virtual firewall (Palo Alto VM series), which will rely on a Off-the shelf X86 cpu (Atom C3758).  The atom, while capable of gigabit routing, likely can't handle all the NFS packets going across it, and it sends the 1 core that the dataplane uses up to like 85% usage.  so the goal is to get NFS to travel INTRA-vlan.    As I see it, I have a few options.  

Option 1.) As in original post, set up netplan on kubernetes to try and route over separate interfaces (probably the best option, since 10G can be for NFS and 1G  can be for K8s traffic)  This would also let me configure the nics with an MTU of 9k, since they're the only things talking over it.
Option 2.) Add a VLAN4 IP to the storage appliance, have pods/host mount this.  Keeps traffic Intra-vlan

Option 3.) Move K8s to a 192.168.1.x address. Since Native interface of K8s host was .4, I'd have to redeploy K8s with a .1 address, not to mention doing BGP (for advertising pod addresses) Intra-vlan isn't advised, this might pose an issue.

 

Given that I don't saturate 1Gb with pure-pod traffic, I think option 1 is the easiest, keeping NFS traffic off the firewall, so I think, like your statement said about setting up proper routing, would be some thing like this:
 


network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:         #### 1G link on vlan 4 for Kubernetes traffic
      dhcp4: no
      addresses: [192.168.4.10/27]
      gateway4: 192.168.4.1
      nameservers:
          addresses: ["1.1.1.1", "8.8.8.8"]
   en3s0f0:           #### 10G link on Vlan 1 for NFS traffic
      dhcp4: no
      addresses: [192.168.1.112/24]
      mtu: 9000
      routes:
        - to: 192.168.1.0/24
          via: 192.168.1.1
          on-link: true

 

  On 29/12/2020 at 00:35, SirEvan said:

This would also let me configure the nics with an MTU of 9k, since they're the only things talking over it.

Expand  

You will for sure want to test doing that.. Jumbo doesn't make sense most of the time - depending on what exactly sort of traffic your sending..

 

I leave mine set at standard 1500..  When I bump it to 9k.. Get worse speed..

 

Here with 9k set on nas and windows machine

$ iperf3.exe -c 192.168.10.10 -V
iperf 3.9
CYGWIN_NT-10.0-19042 I5-Win 3.1.6-340.x86_64 2020-07-09 08:20 UTC x86_64
Control connection MSS 8960
Time: Tue, 29 Dec 2020 15:46:09 GMT
Connecting to host 192.168.10.10, port 5201
      Cookie: nucme5bf7bctijjl46sbkp5yuax33vled34f
      TCP MSS: 8960 (default)
[  5] local 192.168.10.9 port 1054 connected to 192.168.10.10 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   242 MBytes  2.03 Gbits/sec
[  5]   1.00-2.00   sec   251 MBytes  2.10 Gbits/sec
[  5]   2.00-3.00   sec   250 MBytes  2.09 Gbits/sec
[  5]   3.00-4.00   sec   252 MBytes  2.11 Gbits/sec
[  5]   4.00-5.00   sec   251 MBytes  2.11 Gbits/sec
[  5]   5.00-6.00   sec   253 MBytes  2.13 Gbits/sec
[  5]   6.00-7.00   sec   252 MBytes  2.11 Gbits/sec
[  5]   7.00-8.00   sec   251 MBytes  2.11 Gbits/sec
[  5]   8.00-9.00   sec   250 MBytes  2.10 Gbits/sec
[  5]   9.00-10.00  sec   252 MBytes  2.11 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.45 GBytes  2.10 Gbits/sec                  sender
[  5]   0.00-10.01  sec  2.44 GBytes  2.10 Gbits/sec                  receiver
CPU Utilization: local/sender 23.2% (7.7%u/15.4%s), remote/receiver 18.1% (1.2%u/16.8%s)
rcv_tcp_congestion cubic

iperf Done.

 

Here with standard..

$ iperf3.exe -c 192.168.10.10 -V                                                                                
iperf 3.9                                                                                                       
CYGWIN_NT-10.0-19042 I5-Win 3.1.6-340.x86_64 2020-07-09 08:20 UTC x86_64                                        
Control connection MSS 1460                                                                                     
Time: Tue, 29 Dec 2020 15:43:12 GMT                                                                             
Connecting to host 192.168.10.10, port 5201                                                                     
      Cookie: 62tq5fnfv7vfdg65ve35uv3eobnwqqvu6d5g                                                              
      TCP MSS: 1460 (default)                                                                                   
[  5] local 192.168.10.9 port 35558 connected to 192.168.10.10 port 5201                                        
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0          
[ ID] Interval           Transfer     Bitrate                                                                   
[  5]   0.00-1.00   sec   267 MBytes  2.24 Gbits/sec                                                            
[  5]   1.00-2.00   sec   282 MBytes  2.36 Gbits/sec                                                            
[  5]   2.00-3.00   sec   282 MBytes  2.36 Gbits/sec                                                            
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec                                                            
[  5]   4.00-5.00   sec   283 MBytes  2.38 Gbits/sec                                                            
[  5]   5.00-6.00   sec   282 MBytes  2.36 Gbits/sec                                                            
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec                                                            
[  5]   7.00-8.00   sec   281 MBytes  2.36 Gbits/sec                                                            
[  5]   8.00-9.00   sec   282 MBytes  2.37 Gbits/sec                                                            
[  5]   9.00-10.00  sec   278 MBytes  2.33 Gbits/sec                                                            
- - - - - - - - - - - - - - - - - - - - - - - - -                                                               
Test Complete. Summary Results:                                                                                 
[ ID] Interval           Transfer     Bitrate                                                                   
[  5]   0.00-10.00  sec  2.73 GBytes  2.35 Gbits/sec                  sender                                    
[  5]   0.00-10.01  sec  2.73 GBytes  2.34 Gbits/sec                  receiver                                  
CPU Utilization: local/sender 31.8% (11.0%u/20.8%s), remote/receiver 10.8% (0.5%u/10.3%s)                       
rcv_tcp_congestion cubic                                                                                        
                                                                                                                
iperf Done.                                                                                                     

 

I will stick to just standard MTU of 1500..

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Yet the best laptop for all day battery life is a Mac, hands down, no contest. Windows is bloated and power management is rubbish. Search indexer. Defender. Malicious Software Removal Tool. Windows Update (+DISM). Office CTR. Telemetry. Disclaimer: I own a surface laptop studio, multiple gaming desktops, server, and a macbook pro.
    • And bring back taskbar deskbands which were removed, and full and total customization of notification area icons like earlier
    • This has all the markings of a thinly veiled mechanism for force users of older versions of Windows to upgrade...and trash perfectly good hardware. Microsoft, your history easily supports this line of reasoning.
    • Plasma 6.5 brings improved Emoji Selector, better performance in Activity manager, and more by David Uzondu This week saw the long-awaited release of KDE Plasma 6.4, bringing better window management and a whole lot of color management features. Apart from the release of 6.4, the KDE team was able to get other work done, and this was all outlined in the latest issue of This Week in Plasma, which details what is coming down the pipe for future versions. Looking ahead to Plasma 6.5, the developers are making some notable changes to improve performance and general usability. To prevent its database from growing endlessly and causing performance problems over time, the Activity Manager service will now only store the last four months of your history by default. The Emoji selector app is also getting a much-needed redesign that makes the window more compact and moves the sidebar button to the header for a cleaner look. Other little details for 6.5 are being polished up too. The unpopular vertical line separating the date and time on the horizontal Digital Clock widget is gone; if you want it back, you can add it yourself with a custom date format. The "Add New" button has also been moved to the top toolbar in the Shortcuts page within Settings, freeing up some valuable screen real estate. The devs also reduced the minimum size of custom tiling tiles, a small but significant fix for anyone with an ultrawide monitor. In addition to that, the Networks widget's captive portal banner now uses inline header styling, so it doesn't look like a bunch of frames stacked inside each other anymore. Of course, before we get to 6.5, the current release needs some attention. Plasma 6.4's first bug fix release, 6.4.1, addresses issues like broken item selection in the Folder View widget and a bug that could cause the system to lock or suspend faster than intended. Keyboard navigation in list views in Discover feels smoother now, and text is easier to read in certain list items in KRunner. The devs also cleaned up how list items look when you press or click them in Discover. 6.4.1 also fixes a bug where the clipboard history popup would fail to select the top item, and makes the Earth Science Picture of The Day wallpaper plugin work again after its data source changed. Here's the full list of fixes: Fixed several issues in the Folder View widget that caused selecting or opening items to not work when using certain non-default view settings, when the view was scrollable, or when using a touchscreen. Fixed a bug in the Meta+V clipboard popup that sometimes failed to pre-select the top-most item. The Clipboard settings window’s shortcuts page no longer shows columns for local shortcuts that don’t do anything, since the clipboard is global in scope. Fixed the Earth Science Picture of the Day wallpaper plugin after the source data changed formatting again. Made a few fixes to the “Missing Backends” section of Discover’s settings window that kept it from working properly. Fixed a bug that prevented direct scan-out (and its performance benefits) from activating on rotated screens. Fixed an issue where the system could lock or suspend sooner than expected after an app stopped blocking those actions. Installing a new wallpaper plugin no longer causes the plugin list combobox to appear blank. The team even went back to squash some bugs in the older 6.3.6, tackling an issue that could cause keyboard shortcuts to get lost during a system upgrade and fixing an overflow bug with KRunner's faded completion text. Plasma 6.4.1 is set to arrive on June 24th, with 6.3.6 following on July 8th.
    • UniGetUI 3.2.1 Beta 1 by Razvan Serea UniGetUI is an application whose main goal is to create an intuitive GUI for the most common CLI package managers for Windows 10 and Windows 11, such as Winget, Scoop and Chocolatey. With UniGetUI, you'll be able to download, install, update and uninstall any software that's published on the supported package managers — and so much more. UniGetUI features Install, update and remove software from your system easily at one click: UniGetUI combines the packages from the most used package managers for windows: WinGet, Chocolatey, Scoop, Pip, Npm and .NET Tool. Discover new packages and filter them to easily find the package you want. View detailed metadata about any package before installing it. Get the direct download URL or the name of the publisher, as well as the size of the download. Easily bulk-install, update or uninstall multiple packages at once selecting multiple packages before performing an operation Automatically update packages, or be notified when updates become available. Skip versions or completely ignore updates in a per-package basis. Manage your available updates at the touch of a button from the Widgets pane or from Dev Home pane with UniGetUI Widgets. The system tray icon will also show the available updates and installed package, to efficiently update a program or remove a package from your system. Easily customize how and where packages are installed. Select different installation options and switches for each package. Install an older version or force to install a 32bit architecture. [But don't worry, those options will be saved for future updates for this package] Share packages with your friends to show them off that program you found. Here is an example: Hey @friend, Check out this program! Export custom lists of packages to then import them to another machine and install those packages with previously-specified, custom installation parameters. Setting up machines or configuring a specific software setup has never been easier. Backup your packages to a local file to easily recover your setup in a matter of seconds when migrating to a new machine UniGetUI 3.2.1 Beta 1 changelog: Added per-package-manager default install options Added the ability to run pre-install/update/uninstall and post-install/update/uninstall commands Added the ability to kill process(es) before a package is installed/updated/uninstalled Block custom command-line arguments (and pre/post commands) behind a SecureSetting switch SecureSettings are settings that require administrator privileges to be switched. Bundles won't import custom arguments & custom commands by default Bundles will show a security report when importing potentially dangerous settings Added a better crash message for when UniGetUI files are missing Deep improvements to how bundles and InstallOptions are loaded from disk. Improvements to WebView lifecycle Download: UniGetUI 3.2.1 Beta 1 | 51.7 MB (Open Source) Links: WingetUI Home Page | GitHub | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • Rising Star
      aphanic went up a rank
      Rising Star
    • Contributor
      GravityDead went up a rank
      Contributor
    • Week One Done
      BlakeBringer earned a badge
      Week One Done
    • Week One Done
      Helen Shafer earned a badge
      Week One Done
    • First Post
      emptyother earned a badge
      First Post
  • Popular Contributors

    1. 1
      +primortal
      665
    2. 2
      ATLien_0
      264
    3. 3
      Michael Scrip
      235
    4. 4
      Steven P.
      162
    5. 5
      +FloatingFatMan
      152
  • Tell a friend

    Love Neowin? Tell a friend!