Introduction

Welcome to the User Guide for Flock Networks IP Routing Suite 20.3.

The Flock Networks IP Routing Suite will run on any version of Linux, including the SONiC NOS.

At Flock Networks we believe networks should be fast, secure and uniform.

Performance is achieved by:
- Running in a single operating system process, with each component having its own thread. This makes the communication between components dramatically faster.
- Using a thread pool in each component to achieve linear scale vs available CPU cores.
- Keeping the entire implementation lock free. This means each work flow can progress unhindered by any external interruptions.
- Handling network protocol updates in batches. This means route calculation can be performed less frequently, and keeps the CPU's instruction cache hot.
- Only using data structures that are CPU data cache friendly.
Security is achieved by the routing suite being exclusively written in the Rust programming language. Rust has a similar performance to C / C++, but is memory safe at compile time.
Uniformity is achieved by having a similar configuration across all routers in the network. Device specific and in particular interface specific configuration is to be avoided.

If you have any questions, suggestions or issues, please email support@flocknetworks.com.

Installation

The Flock Networks Routing Suite is shipped as a Debian package. However the Routing Suite can run on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc). The following instructions are for installing on a Debian based Linux distribution (Debian, Ubuntu, Mint, etc). To install on other systems, see Manual Install. The Flock Networks Routing Suite can also be used as a drop in replacement on the SONiC Network Operating System, see SONiC.

Download the latest Debian package.

Install the application from the Debian package (as root or using sudo);

# dpkg -i flockd_20.3.x_amd64.deb

The Flock Routing Suite Licence is appended to;

/usr/share/doc/flockd/copyright

The Flock Routing Suite Client flockc is copied to;

/usr/bin/flockc

The Flock Routing Suite Daemon flockd is copied to;

/usr/sbin/flockd

flockd config is stored under this directory;

/etc/flockd

As part of a fresh install:

A default ospfv2.toml file is copied to /etc/flockd. The presence of /etc/flockd/ospfv2.toml causes flockd to enable OSPFv2 on startup.
An example bgpv4.toml.example file is copied to /etc/flockd. Until there is a /etc/flockd/bgpv4.toml file, BGPv4 will remain disabled on startup.

flockd is controlled by systemd;

$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd

logging can be viewed via journalctl;

# journalctl -u flockd --boot

Example successful installation

# dpkg -i flockd_20.3.0_amd64.deb
Selecting previously unselected package flockd.
(Reading database ... 27695 files and directories currently installed.)
Preparing to unpack flockd_20.3.0_amd64.deb ...
Unpacking flockd (20.3.0) ...
Setting up flockd (20.3.0) ...

# systemctl start flockd

$ systemctl status flockd
● flockd.service - Flock Networks Routing Suite Daemon
Loaded: loaded (/lib/systemd/system/flockd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-24 12:39:24 GMT; 6s ago
Main PID: 825 (flockd)
    Tasks: 1 (limit: 1150)
Memory: 908.0K
CGroup: /system.slice/flockd.service
        └─785 /usr/sbin/flockd
Jan 24 12:39:24 flocknet flockd[825]: [INFO  flockd] START: PID=825, Compile Mode=Release, Log Level="info"
Jan 24 12:39:24 flocknet flockd[825]: [INFO  flockd] OSPFv2 derived RouterId(192.168.122.171) from IPv4 Address
Jan 24 12:39:28 flocknet flockd[825]: [INFO  flockd::ospf_intf] IntfId(2) 10.0.1.168/24 state change Wait -> Backup
Jan 24 12:39:28 flocknet flockd[825]: [INFO  flockd::ospf_neigh] RouterId(10.0.100.2) V4(10.0.1.249) neigh state change Exchange -> Full

$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.3.0"
"model": "Multi-threaded"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
"fpm_state": "disabled"
$

Troubleshooting Installation

`flockd` fails to start

systemd PolicyKit / polkit failure

PolicyKit / polkit is the "sudo of systemd". If you have PolicyKit / polkit active on the install host, and you want to start flockd as an unprivileged user, you will need to add PolicyKit / polkit configuration for flockd. Alternatively you can just run systemctl start flockd as root or under sudo.

$ systemctl start flockd
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'flockd.service'.
Authenticating as: www-data
Password:
Failed to start flockd.service: Connection timed out
See system logs and 'systemctl status flockd.service' for details.
polkit-agent-helper-1: pam_authenticate failed: Authentication failure

Flock Routing Suite Daemon `flockd`

Overview

The flockd process is controlled by systemd.

$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd

flockd is a single process that is split into separate components.

System Component

The system component manages the other components and presents a unified API to the outside world.

The system component monitors external signals and updates the other components as required. Example external signals would be the addition of an IP address to an interface, or the addition of a route to the Linux Kernel.

The system component provides the configuration and operations API.

Routing Information Base (RIB) Component

The RIB component receives route updates from other components and from the Linux Kernel. The RIB component selects the best update for each route and redistributes it to other components and to the Linux Kernel. This redistribution can be controlled by configuration. By default the RIB will send the best route to the Linux Kernel and will accept any routes from the Linux Kernel that are not sourced by flockd.

OSPFv2 Component

The OSPFv2 component runs the OSPFv2 protocol. This component gets interface state information and connected IP subnet information from the System Component. This component gets external state information by connecting to OSPFv2 neighbors via Linux Kernel raw IP sockets. This information is combined and each best route is derived and sent to the RIB component.

BGPv4 Component

The BGPv4 component runs the BGPv4 protocol. This component gets IP route information from the RIB component and external state information by connecting to BGP neighbors via Linux Kernel TCP sockets. This information is combined and each best route is derived and sent to the RIB component.

Multithreading

The routing suite runs in a single process and each component runs in its own thread. One way to view the active threads is via the Linux /proc virtual file system. The System and RIB components currently share the main thread. OSPFv2 and BGPv4 have a thread each. If a component is not enabled, it will not have any threads.

The BGPv4 master thread travels with a BGP thread pool. The BGP thread pool will have a thread for each logical CPU core on the router.

flock@flocknet:~$ grep Name /proc/`pidof flockd`/task/*/status
/proc/409/task/409/status:Name: flockd
/proc/409/task/432/status:Name: OSPFv2 Master
/proc/409/task/433/status:Name: BGPv4
/proc/409/task/436/status:Name: BGPv4
/proc/409/task/437/status:Name: BGPv4
/proc/409/task/438/status:Name: BGPv4
/proc/409/task/439/status:Name: BGPv4
flock@flocknet:~$

In the example above the thread with id 433 is the BGP Master thread. The threads with id's 436, 437, 438 and 439 belong to the BGP thread pool. There are 4 threads in the BGP thread pool as the router has 4 logical CPU's.

flock@flocknet:~$ cat /proc/cpuinfo | grep -c processor
4
flock@flocknet:~$

Logging

Logging can be viewed using journalctl.

View flockd logs since router was booted.

$ journalctl -u flockd --boot
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:38:44 BST. --
Jun 29 08:49:36 r61 flockd[455]: [INFO  flockd] START: PID=455, Compile Mode=Release, Log Level="info"
...

View flockd logs since a certain time.

$ journalctl -u flockd --since "2020-06-26 17:19:20"
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:40:55 BST. --
Jun 26 17:19:20 r61 flockd[431]: [INFO  flockd::bgp::bgp_neigh] 90.0.93.70/Some(BgpId(70.0.100.70)) BgpAsId2(70) Outgoing FSM state change OpenConfirm -> Established
...

Monitor for the latest flockd logs.

$ journalctl -u flockd --follow
-- Logs begin at Fri 2020-01-10 14:01:47 GMT. --
Jun 29 09:44:34 r61 flockd[455]: [INFO  flockd::bgp::bgp_neigh] 60.0.60.60/Some(BgpId(60.0.100.60)) BgpAsId2(60) Incoming FSM state change OpenConfirm -> Established
...

Log Levels

The default log level is info. At this level all info and higher priority levels will be logged. Supported log levels in descending priority order are error, warn, info, debug and trace.

Error

Log level error / [ERROR] is used for unexpected events signalled from inside the router. These are never expected to be seen and indicate a bug. Please email a bug report to: support@flocknetworks.com.

Warning

Log level warn / [WARN] is used for unexpected events signalled from outside the router. It is normal to see warnings whilst the network is converging. Warnings should never be seen after the network has converged and remains stable.

[WARN  flockd::ospf_neigh] RouterId(10.0.100.2), V4(10.0.3.157) neigh state change Full -> Down

Information

Log level info / [INFO] is used for expected events of note

[INFO  flockd] START: PID 385 Compile Mode Release Log Level "debug"
[INFO  flockd::sys::sys_intf] Update IntfId(2)] Broadcast Mtu(1500) Up [] event DownToUp
[INFO  flockd::ospf_intf] IntfId(2), 10.0.1.168/24 state change Wait -> DrOther
[INFO  flockd::ospf_neigh] RouterId(10.0.100.3), V4(10.0.1.152) neigh state change Loading -> Full

Debug

Log level debug / [DEBUG] is used for common expected events.

Trace

Log level trace / [TRACE] is used for very common expected events.

Changing the default log level

The default log level can be changed by setting the RUST_LOG environment variable in the flockd systemd service file.

grep RUST_LOG /lib/systemd/system/flockd.service
    Environment=RUST_LOG="info"

When the systemd service file has changed, systemd needs to be told to reload the new flockd configuration.

# systemctl daemon-reload

To enable the new log level, flockd needs to be restarted.

# systemctl restart flockd

Flock Routing Suite Client `flockc`

flockc is the client provided to query the current state of flockd. By design flockc can be run by a user with no special privileges. flockc connects to a Read-Only connection on flockd. flockd state can be viewed but cannot be changed. All output is in JSON format, making it suitable for consumption by humans or machines.

flockc connects to a REST API on flockd. This REST API will talk to any HTTP based client. This means any HTTP based client can be used to monitor flockd. Just connect the client to the same URL that flockc uses. Use the flockc --show-url option to display the URL associated with the flockc command.

For example:

$ flockc system --show-url
http://127.0.0.1:8000/system/sort/json-lines

See here for more information on using the REST API.

Local Connectivity

flockc uses the REST API via the local IP Looback address to retrieve state information from the local flockd.

This command shows all the IPv4 Prefixes in the RIB of the local flocknet1 router.

flock@flocknet1:~$ flockc ribv4 --prefix
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,  "ip_addr":"10.0.1.168"},{"intf_id":3,"ip_addr":"10.0.2.203"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
...

Remote Connectivity

flockc can connect and retrieve the same state information from a remote router. Just add the --host option to any command. The host parameter can be a hostname or an IP Address.

This command shows all the IPv4 Prefixes in the RIB of the remote flocknet2 router.

flock@flocknet1:~$ flockc ribv4 --prefix --host flocknet2
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,  "ip_addr":"10.0.11.168"},{"intf_id":3,"ip_addr":"10.0.22.203"}]}
{"ip_net":"10.0.11.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.22.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
...

flockc is available on its own. Install this package if you want to remotely manage your network from a host.

# dpkg -i flockc_20.3.x_amd64.deb

General Command Flags

General Flags can be included in any flockc command.

-d, --detail
-h, --help
-J, --json           Output in JSON
-j, --json-pretty    Output in Pretty Print JSON
-u, --unsorted       Output in unsorted order

Changing the output format

By default the output is in JSON Lines format, which is JSON with newlines added to aid reading by a human. The output can also be changed to JSON Pretty or vanilla JSON.

JSON Lines format

flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3
{"ospf_area_id":"0.0.0.0"}
{"ospf_intf":"enp1s0"}
{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.225","bdr":"10.0.5.204"}

JSON Pretty format

Using the -j, --json-pretty flag the output is pretty printed JSON.

flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3 -j
[
  {
    "ospf_area_id": "0.0.0.0",
    "ospf_intfs": [
      {
        "ospf_intf": "enp1s0",
        "ospf_neighs": [
          {
            "id": "10.0.100.3",
            "ip": "10.0.5.225",
            "state": "Full",
            "dr": "10.0.5.225",
            "bdr": "10.0.5.204"
          }
        ]
      }
    ]
  }
]

JSON format

Using the -J, --json flag the output is JSON. This is the flag to use when talking to another application which wants to deserialize the JSON string into its own representation.

flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3 -J
[{"ospf_area_id":"0.0.0.0","ospf_intfs":[{"ospf_intf":"enp1s0","ospf_neighs":[{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.225","bdr":"10.0.5.204"}]}]}]

System Component

Configuration Overview

/etc/flockd/system.toml is the system configuration file. The file must start by specifying the format_version it uses. Having a format_version allows configuration to be amended or deprecated in the future. Currently version 20 is the only supported version.

format_version = 20

Enable the forwarding plane manager (FPM) connection. FPM is used to program a forwarding plane that is not the Kernel. In particular SONiC requires FPM to be enabled. To enable FPM add these two lines to /etc/flockd/system.toml

[fpm]
tcp_port = 2620

Check status of `flockd`

flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.3.0"
"model": "Multi-threaded"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
"fpm_state": "disabled"
flock@flocknet:~$

Show all system interfaces

flock@flocknet:~$ flockc system -i
{"name":"dummy0","id":7,"ip_prefixes":["60.0.20.61/32"],"state":"Up"}
{"name":"enp10s0","id":4,"ip_prefixes":[],"state":"Down"}
{"name":"enp1s0","id":2,"ip_prefixes":["90.0.93.61/24"],"state":"Up"}
{"name":"enp7s0","id":5,"ip_prefixes":[],"state":"Down"}
{"name":"enp8s0","id":6,"ip_prefixes":["60.0.60.61/24"],"state":"Up"}
{"name":"enp9s0","id":3,"ip_prefixes":["90.0.91.61/24"],"state":"Up"}
{"name":"lo","id":1,"ip_prefixes":["127.0.0.1/8"],"state":"Up"}
flock@flocknet:~$

Show single system interface

flock@flocknet:~$ flockc system -i enp1s0
{"name":"enp1s0","id":2,"ip_prefixes":["90.0.93.61/24"],"state":"Up"}
flock@flocknet:~$

System Operation

Help

flockc system -h

Overview

flockc system

All system interfaces

flockc system -i

Single interface

flockc system -i <interface-name>

RIBv4 Component

Show RIB Overview

flock@flocknet:~$ flockc ribv4
{"route_count":2328}
flock@flocknet:~$

Longest Path Match (LPM) for destination

flock@flocknet:~$ flockc ribv4 -l 10.0.2.34
{"origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
flock@flocknet:~$

Show single RIB prefix entry

flock@flocknet:~$ flockc ribv4 -p 10.0.2.0/24
{"origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
flock@flocknet:~$

Show all prefix entries in the RIB

flock@flocknet:~$ flockc ribv4 -p
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.22.1"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
{"ip_net":"42.0.0.0/16","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.68.122.1"}]}
{"ip_net":"192.168.122.0/24","origin":"Kernel","next_hops":[{"intf_id":5}]}
flock@flocknet:~$

RIBv4 Operation

Help

flockc ribv4 -h

Overview

flockc ribv4

Prefixes

flockc ribv4 -p [<ipv4-network>]

All prefixes by route origin. Origin can be bgp, kernel or ospfv2.

flockc ribv4 -p -o ospfv2

Longest Path Match (LPM)

flockc ribv4 -l <ipv4-address>

OSPFv2 Component

Configuration Overview

/etc/flockd/ospfv2.toml is the configuration file. If this file exists OSPFv2 is enabled, otherwise OSPFv2 remains disabled. The default flockd debian package install will create this file, so OSPFv2 will be enabled as part of the install.

The file must start by specifying the format_version it uses. Having a format_version allows configuration to be amended or deprecated in the future. Currently version 20 is the only supported version.

The Flock Networks Routing Suite is designed for massive scale so placing all routers in a single OSPF area is recommended. (If you are adding a device to an existing multi-area OSPF Autonomous System, multiple areas are fully supported)

flock@r01:~$ cat /etc/flockd/ospfv2.toml
format_version = 20
# Place all ethernet interfaces into area 0.0.0.0
[[area]]
area_id = "0.0.0.0"
[[area.intf]]
# Match any interfaces that have name starting with "en"
name = "^en"

This is all the OSPFv2 configuration you need, to create an OSPF network as large as you like. Each device has an identical configuration which simplifies the operation of the network. A management station can easily determine all the Router Id's in the network by querying a single device for all of its Router LSA's. But anyway if you like complexity (we don't) then read on, otherwise skip to the Operational State section.

Redistribution of Kernel routes into OSPF

You may wish to redistribute static routes from the RIB into OSPFv2.

[[redistribute]]
origin = "kernel-static"
metric_type = 2
metric = 1000

As a minimum we may want a default route added to the kernel of each ASBR router. This route will appear in the RIB and then be redistributed into OSPFv2. OSPFv2 will advertise this route across the AS, so all nodes learn the route to exit the network. Static routes are added using the Linux command line.

# ip route add 0.0.0.0/0 via 192.168.122.171 dev enp9s0

In Debian make this permanent by adding the following lines in /etc/network/interfaces under the iface enp9s0 entry.

post-up /bin/ip route add 0.0.0.0/0 via 192.168.122.171 dev enp9s0

Explicit Router Id

By default the highest IPv4 Address is used as the Router Id. To explicitly set the Router Id to 10.0.100.1 place this line at the top of ospfv2.toml

router_id = "10.0.100.1"

An alternative approach that can be useful when operating the network, is to create a dummy interface with an IP Address that becomes the Router ID.

First create the dummy interface. On Debian add this entry in /etc/network/interfaces.

flock@r01:~$ cat /etc/network/interfaces
auto dummy0
iface dummy0 inet static
address 10.0.100.1/32
pre-up /bin/ip link add dummy0 type dummy
post-down /bin/ip link del dummy0

flock@r01:~$ sudo systemctl restart networking
flock@r01:~$

Second place the dummy0 interface in OSPF area 0. The Flock Networks Routing Suite will select the Router Id from IP Addresses on dummy interfaces in preference to other interface types. So with this method you do not need the explicit router_id configuration shown above.

flock@r01:~$ cat /etc/flockd/ospfv2.toml
[[area]]
area_id = "0.0.0.0"
[[area.intf]]
name = "^en"
[[area.intf]]
name = "dummy0" # Add this line

flock@r01:~$ sudo systemctl restart flockd
flock@r01:~$

For detailed configuration information see OSPFv2 Configuration.

Operational State Overview

Check OSPFv2 is enabled

Check OSPFv2 is listed in the enabled_protocols field.

flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.3.0"
"model": "Multi-threaded"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
"fpm_state": "disabled"
flock@flocknet:~$

Show OSPFv2 Overview

flock@r01:~$ flockc ospfv2
{"router_id":"10.0.100.4","class":"IR","redistribute":[]}

Show all neighbors (out of all interfaces, in all areas)

flock@r01:~$ flockc ospfv2 -n
{"ospf_area_id":"0.0.0.0"}
{"ospf_intf":"enp1s0"}
{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.204","bdr":"10.0.5.225"}
{"ospf_area_id":"0.0.0.20"}
{"ospf_intf":"enp7s0"}
{"id":"10.20.100.20","ip":"10.20.20.189","state":"Full","dr":"10.20.20.189","bdr":"10.20.20.214"}

Show Area 0 Link State Database

flock@flocknet:~$ flockc ospfv2 -a 0 -l
{"lsa_age":279,"lsa_opts":{"bits":2},"lsa_type":"Router","lsa_id":"10.0.100.4","lsa_router_id":"10.0.100.4","lsa_seq":-2147483646,"lsa_checksum":28411,"lsa_len":36}
{"lsa_age":266,"lsa_opts":{"bits":2},"lsa_type":"Router","lsa_id":"10.0.100.5","lsa_router_id":"10.0.100.5","lsa_seq":-2147483646,"lsa_checksum":22802,"lsa_len":36}
...

For detailed operational state information see OSPFv2 Operation.

OSPFv2 Configuration

All Flock Network configuration files are in the toml format. If the configuration file exists the protocol is enabled, otherwise it remains disabled.

/etc/flockd/ospfv2.toml

# OSPFv2 Instance level configuration
# -----------------------------------
# RFC2328 1.2 Router ID
# Optional: If not specified highest IPv4 Address is used.
router_id = "String in dotted decimal format"

# Array of 'redistribute' toml tables
[[redistribute]]
# Origin of the Routes in the RIB to be redistributed
origin = ("kernel-static" | "kernel-connected")
# RFC2328 2.3 Type 1 / Type 2 external metrics
metric_type = ( 1 | 2 )
# OSPF metric to reach redistributed routes, from this router.
# RFC2328 B. LSInfinity => 16777215
metric = ( 0..16777215 )

# OSPFv2 Area level configuration
# -------------------------------
[[area]] # Array of 'area' toml tables
# RFC2328 C.2 Area ID
area_id = "String in dotted decimal format"

# OSPFv2 Interface level configuration
# ------------------------------------
[[area.intf]]
# Specify the entire interface name i.e. "eno1"
# or use ^ to match the start of interface names
#   e.g. "^en" will match all interfaces that start with "en"
name = "Interface Name"
# RFC2328 C.3 Interface output cost
# Optional: Default is 10
cost = ( 1..65,535 )
# RFC2328 C.3 Router Priority
# Optional: Default is 1
priority = ( 0..255 )
# RFC2328 C.3 HelloInterval
# Optional: Default is 10s
hello_interval = ( 1..65,535 seconds)
# RFC2328 C.3 RouterDeadInterval
# Optional: Default is 40s
dead_interval = ( 1..65,535 seconds)

Example Exhaustive OSPFv2 Configuration

router_id = "10.0.1.100"

[[redistribute]]
origin = "kernel-static"
metric_type = 1
metric = 100
[[redistribute]]
origin = "kernel-connected"
metric_type = 2
metric = 1000

[[area]]
area_id = "0.0.0.0"
[[area.intf]]
name = "enp7s0"
cost = 20
priority = 10
hello_interval = 1
dead_interval = 4
[[area.intf]]
name = "enp8s0"
cost = 40
priority = 20
hello_interval = 2
dead_interval = 8

[[area]]
area_id = "0.0.0.1"
[[area.intf]]
name = "enp9s0"
cost = 30
priority = 30
hello_interval = 20
dead_interval = 80

OSPFv2 Operation

Help

flockc ospfv2 -h

Overview

flockc ospfv2

Overview of areas

flockc ospfv2 -a [<area-id>]

All interfaces in Area 20

flockc ospfv2 -i -a 20

All neighbors on interface enp1s0 in Area 0

flockc ospfv2 -n -i enp1s0 -a 0

All neighbors on all interfaces in all areas

flockc ospfv2 -n

Autonomous System Link State Database

flockc ospfv2 -l

Area 0.0.0.0 Link State Database

flockc ospfv2 -a 0 -l

Network route table prefixes

flockc ospfv2 -p [<ipv4-network>]

Router route table prefixes

flockc ospfv2 -P [<router-id>]

BGPv4 Component

Configuration Overview

/etc/flockd/bgpv4.toml is the configuration file. If this file exists BGPv4 is enabled, otherwise BGPv4 remains disabled. The default flockd debian package install will not create this file, so BGPv4 will not be enabled as part of the install. During the package install, an example configuration file is copied to /etc/flockd/bgpv4.toml.example.

In the minimal configuration file below;

The router is in AS 65016 with a BGP router id of 172.16.10.1
The router originates the 172.16.0.0/16 network
The router has a single iBGP neighbor 172.16.10.2
- The iBGP connection source is 172.16.10.1
- Routes are advertised over iBGP with a next hop of 172.16.10.1 (next_hop_self)

The router has a single eBGP neighbor 172.17.20.1 in remote AS 65017

  [local]
  id = "172.16.10.1" # BGP ID of this router
  as_id = 65016 # Autonomous System this router is part of
  # Originate networks into BGP
  [[local.network]]
  ip_prefix = "172.16.0.0/16"

  # iBGP neighbors (as_id == local.as_id)
  [[as]]
  as_id = 65016
  # Allow iBGP neighbors to not have eBGP subnets in their IPv4 RIB
  next_hop_self = true
  [[as.neighbor]]
  ip = "172.16.10.2"
  local_ip = "172.16.10.1"

  # AS65017 eBGP neighbors (as_id != local.as_id)
  [[as]]
  as_id = 65017
  [[as.neighbor]]
  ip = "172.17.20.1"

Operational State Overview

Check BGPv4 is enabled

Check BGPv4 is listed in the enabled_protocols field.

flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.3.0"
"model": "Multi-threaded"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["BGPv4"]
"fpm_state": "disabled"
flock@flocknet:~$

Show BGPv4 Overview

flock@r01:~$ flockc bgp
{"id":"60.0.100.61","as_id":60,"routes":{"ipv4_unicast":4271}}

Show all neighbors (in all Autonomous Systems)

BGPv4 runs two Finite State Machines (FSM's) per neighbor. One FSM handles the Outgoing TCP connection and the other handles the Incoming TCP connection. The Flock Routing Suite does not hide this from the operator. In the final working state each neighbor should have one FSM in the Established state, and one FSM in the Idle state.

The last error to cause a BGP Notify Message is held in each FSM's last_notify field. This field is never cleared, it is only overwritten with the last error. So a value of null means there have been no errors that have caused a notify message since flockd was started.

flock@r61:~$ flockc bgp -n
{"as_id":50}
{"ipv4_addr":"50.0.20.50","local_ipv4_addr":"60.0.20.61","as_id":50,"bgp_id":"50.0.100.50","neigh_type":"External","outgoing":{"state":"Established","last_notify":null},"incoming":{"state":"Idle","last_notify":null}}
{"as_id":60}
{"ipv4_addr":"60.0.60.60","as_id":60,"bgp_id":"60.0.100.60","neigh_type":"Internal","outgoing":{"state":"Established","last_notify":null},"incoming":{"state":"Idle","last_notify":null}}

Show BGPv4 RIB prefix's

Note that this is not the System RIB. The BGPv4 RIB records routes from all neighbors and sends the 'best entry' route to the System RIB.

Show all prefix's. Only the 'best entry' for each prefix is shown, along with the reason why it was the best.

flock@r01:~$ flockc bgp -p
{"ip_net":"50.0.0.0/8","best_entry":{"neigh":{"neigh_ipv4_addr":"50.0.20.50","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}},"reason":"OnlyValidPeer"}}
{"ip_net":"60.0.0.0/8","best_entry":{"reason":"SelfOriginated"}}

Show a specific prefix. The 'best entry' and all the candidate entries are shown.

flock@r61:~$ flockc bgp -p 50.0.0.0/8
{"best_entry":{"neigh":{"neigh_ipv4_addr":"50.0.20.50","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}},"reason":"OnlyValidPeer"},"neighboring_as":[{"med_origin_as_id":50,"via_neighs":[{"neigh_ipv4_addr":"50.0.20.50","neigh_bgp_id":"50.0.100.50","neigh_type":"External","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}}]}]}

BGPv4 Configuration

All Flock Network configuration files are in the toml format. If the configuration file exists the protocol is enabled, otherwise it remains disabled.

Example /etc/flockd/bgpv4.toml

[local]
id = "172.16.10.1" # BGP ID of this router
as_id = 65016 # Autonomous System this router is part of
# Originate networks into BGP
[[local.network]]
ip_prefix = "172.16.0.0/16"
[[local.network]]
ip_prefix = "172.17.0.0/16"

# iBGP neighbors (as_id == local.as_id)
[[as]]
as_id = 65016
# Allow iBGP neighbors to not have eBGP subnets in their IPv4 RIB
next_hop_self = true
[[as.neighbor]]
ip = "172.16.10.2"
local_ip = "172.16.10.1"
# Set non-default timers for this neighbor
[[as.neighbor]]
ip = "172.16.10.3"
local_ip = "172.16.10.1"
[as.neighbor.timers]
keep_alive = 30
hold_time = 90

# AS65017 eBGP neighbors (as_id != local.as_id)
[[as]]
as_id = 65017
[[as.neighbor]]
ip = "172.17.20.1"
[[as.neighbor]]
ip = "172.17.30.1"

BGPv4 Active / Passive Neighbors

By default BGPv4 will try to create two TCP transport connections to each neighbor. One outgoing to the neighbors remote BGP TCP port 179, and one allowing incoming connections from the neighbor to the local BGP TCP port 179. A tie break is used to enusure only one connection remains when the BGP neighbor moves to the 'Established' state.

The router can be configured to only form a single TCP transport connection to each neighbor using the connect_mode neighbor configuration parameter.

# Only create the outgoing connection to this neighbor, refuse any incoming connection.
[[as.neighbor]]
connect_mode = "active"

# Only allow the incoming connection from this neighbor, do not create any outgoing connection.
[[as.neighbor]]
connect_mode = "passive"

BGPv4 Route Reflectors

To configure a router as a BGP Route Reflector, specify which neighbors are Route Relector clients using the route_reflector_client configuration boolean.

# Reflect iBGP routes to and from neighbor 172.16.10.2
[[as.neighbor]]
ip = "172.16.10.2"
route_reflector_client = true

# Do not reflect iBGP routes to or from neighbor 172.16.10.3
# Route reflection is off by default.
[[as.neighbor]]
ip = "172.16.10.3"

To deploy redundant Route Reflectors a Route Relector Cluster Id can optionally be configured.

[local]
cluster_id = "1.2.3.4"

BGPv4 Operation

Help

flockc bgp -h

Overview

flockc bgp

Autonomous System

flockc bgp -a [<as-d>]

Neighbors

flockc bgp -n [<ip-addr>]

Prefixes

flockc bgp -p [<ip-network>]

Linux Kernel Settings

IP forwarding

For a Linux host to operate as an IP Router, IP forwarding must be enabled.

View IP Forwarding state

# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

Turn on IP Forwarding

# sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

Turn on IP forwarding permanently / on system boot

# nano /etc/sysctl.conf
net.ipv4.ip_forward = 1

BGP / TCP Termination at scale

Some protocols (most notably BGPv4) and the Operation API rely on a TCP transport. The Linux kernel has two parameters to control how many TCP connections can simultaneously be formed.

tcp_max_syn_backlog: Max TCP connections waiting for final ACK (of the TCP three way handshake)

flock@flocknet:/proc/sys/net/ipv4$ cat tcp_max_syn_backlog
256

somaxconn: Max TCP connections with completed TCP three way handshake waiting for accept() to be called.

flock@flocknet:/proc/sys/net$ sudo cat core/somaxconn
128

If these limits are exceeded the Linux kernel decides it is under a SYN DoS attack and will prevent further connections. Under these conditions this message is logged in /var/log/messages

"TCP: request_sock_TCP: Possible SYN flooding on port 179. Sending cookies."

In a production network it is very unlikely these limits will be reached (unless the router is under a SYN DoS attack). Even with 1000's of BGP neighbors it is unlikely that there will be greater than 128 TCP connections waiting to be accepted. However in the lab using a traffic generator this limit can be hit.

By default the Flock Networks Routing Suite is configured to be able to handle up to 1024 simultaneous BGPv4 TCP connections. To reach this scale the Linux kernel defaults need to be updated to match.

flock@flocknet:~$ sudo sysctl -w net.ipv4.tcp_max_syn_backlog=1024
flock@flocknet:~$ sudo sysctl -w net.core.somaxconn=1024
flock@flocknet:~$

Monitoring via the REST API

flockd supports a REST API to allow monitoring of the Router. The REST API is accessed over HTTP and delivers a JSON payload. The REST API is Read-Only so a connecting client can only query state in the router, never change it. In HTTP terms, by design, only the GET method is supported. The flockc client uses the REST API.

Local Connection within a Router

On router R01 and we use the client to retrieve local state within R01. The JSON is delivered using the REST API via R01's IP loopback address.

flock@R01:~$ flockc ribv4 --prefix
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.246"}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}

Remote Connection from one Router to another

From R01 we can view information on another router using the REST API. The JSON is delivered via HTTP. The command is the same, except we append the host name / IP Address of the target router.

flock@R01:~$ flockc ribv4 --prefix --host R02
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}

Remote Connection from Host to any Router

Now moving to a host H01. We can install only flockc, and we can monitor all routers, without running flockd. NB: This is the flockc package being installed, which only includes the client, not the flockd package which includes the client and the daemon.

flock@H01# dpkg -i flockc_20.3.x_amd64.deb
flock@H01:~$ flockc ribv4 --prefix --host R01
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
...
flock@H01:~$ flockc ribv4 --prefix --host R02
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}
...

3rd Party Clients

Since the REST API is simply a JSON payload inside HTTP, there are many ways to connect. The Host Operating System can be your choice of Linux, Windows, Mac, Redox, etc. The HTTP client application can be any one that the Operating System supports. For instance, the venerable curl will run on all the above Operating Systems.

To use curl you just need to know the URL to connect to. If you run flockc with the --show-url option, then the URL associated with that command will be displayed rather than connected to.

flock@H01:~$ flockc ribv4 --prefix --host R01 --show-url
http://R01:8000/ribv4/prefix?ipv4_net=*/sort/json-lines
flock@H01:~$

you@your-host:~$ curl -s -X GET "http://R01:8000/ribv4/prefix?ipv4_net=*/sort/json-lines"
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
...

You can then use the language of your choice to consume and manipulate the JSON information. When feeding into a program we want vanilla JSON (rather than the default of JSON with extra line breaks), so we include the --json option to discover the URL.

flock@H01:~$ flockc ribv4 --prefix --json --host R01 --show-url
http://R01:8000/ribv4/prefix?ipv4_net=*/sort/json
flock@H01:~$

you@your-host:~$ curl -s -X GET "http://R01:8000/ribv4/prefix?ipv4_net=*/sort/json" | python -m json.tool
[
    {
        "ip_net": "0.0.0.0/0",
        "next_hops": [
            {
                "intf_id": 5,
                "ip_addr": "192.168.122.1"
            }
        ],
        "origin": "Kernel"
    },
...

Example: Connecting to the REST API using Python

flock@H01:~$ flockc ribv4 --prefix --host r02 --json --show-url
http://r02:8000/ribv4/prefix?ipv4_net=*/sort/json

you@your-host:~$ python3
Python 3.7.3 (default, Apr  3 2019, 05:39:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('http://r02:8000/ribv4/prefix?ipv4_net=*/sort/json')
>>> r.json()
[{'ip_net': '0.0.0.0/0', 'origin': 'Ospfv2', 'next_hops': [{'intf_id': 2, 'ip_addr': '10.0.1.168'}, {'intf_id': 4, 'ip_addr': '10.0.3.176'}]}, {'ip_net': '10.0.1.0/24', 'origin': 'Kernel', 'next_hops': [{'intf_id': 2}]}, {'ip_net': '192.168.122.0/24', 'origin': 'Ospfv2', 'next_hops': [{'intf_id': 2, 'ip_addr': '10.0.1.168'}, {'intf_id': 4, 'ip_addr': '10.0.3.176'}]}]

Manually Building REST API URL's

The available REST API URL's can be discovered using the flockc --show-url option. They can also be built by following these rules.

The URL's mirror the flockc long options, for example:

flockc ribv4 --prefix maps to the URL /ribv4/prefix

Sorting

Append /sort if you want the output sorted.

/ribv4/prefix/sort

Specify the output JSON format

A json format must be specified at the end of the URL.

Append /json if you want vanilla JSON (no extra line breaks).

/ribv4/prefix/sort/json

Append /json-pretty if you want JSON pretty printed.

/ribv4/prefix/sort/json-pretty

Append /json-lines if you want JSON Lines.

/ribv4/prefix/sort/json-lines

Wildcards

In a URL query string a * is treated as wild. prefix?ipv4_net=* will get all IPv4 Networks.

/ribv4/prefix?ipv4_net=*/sort/json

Escaping the `/` character

To escape the / character in a URL use %2F (0x2F is the ASCII value for /). So to specify an IPv4 network use ipv4_net=10.20.30.0%2F16. If you do not escape the / character this URL will not be found ipv4_net=10.20.30.0/16. It specifies a URL path ending in /16 which is not what we want.

/ribv4/prefix?ipv4_net=10.20.30.0%2F24/sort/json

Commonly used REST API URL's

# BGP Overview
/bgp/json
# BGP Autonomous System
/bgp/as?as_id=*/json
/bgp/as?as_id=65073/json
# BGP Prefixes
/bgp/prefix?ipv4_net=*/json
/bgp/prefix?ipv4_net=4.3.0.0%2F16/json
# BGP Neighbors
/bgp/as?as_id=65056/neigh?ipv4_addr=10.20.30.42/json
/bgp/as?as_id=65056/neigh?ipv4_addr=*/json

# OSPFv2 Overview
/ospfv2/json
# OSPFv2 Autonomous System Link State Database (LSA Type 5)
/ospfv2/lsdb/json
# OSPFv2 Area Overview
/ospfv2/area?area_id=0.0.0.0/json
# OSPFv2 Area Link State Database (LSA Type 1 -> Type 4)
/ospfv2/area?area_id=0.0.0.0/lsdb/json
# OSPFv2 Interfaces
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=*/json
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=eth0/json
# OSPFv2 Neighbors
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=*/neigh?id=5.6.7.8/sort/json
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=eth0/neigh?id=5.6.7.8/sort/json
# OSPFv2 Network Prefixes
/ospfv2/prefix-network?ipv4_net=*/json
/ospfv2/prefix-network?ipv4_net=20.30.0.0%2F16/json

# RIBv4 Overview
/ribv4/json
# RIBv4 Longest path match lookup
/ribv4/lpm?ipv4_addr=1.2.3.4/json
# RIBv4 Prefixes
/ribv4/prefix?ipv4_net=*/json
/ribv4/prefix?ipv4_net=10.20.30.0%2F24/json
/ribv4/prefix?origin=ospfv2/json

# System Overview
/system/json
# System Interfaces
/system/intf?intf_name=*/json
/system/intf?intf_name=eth0/json

RFC Compliance

Flock Networks Routing Suite implements the following RFC's.

BGPv4

RFC 1997 BGP Communities Attribute
RFC 4271 A Border Gateway Protocol 4 (BGP-4)
RFC 4360 BGP Extended Communities Attribute
RFC 4456 BGP Route Reflection
RFC 4760 Multiprotocol Extensions for BGP-4
- Currently only IPv4 (AFI 1) and Unicast (SAFI 1) are supported
RFC 5492 Capabilities Advertisement with BGP-4
RFC 5668 4-Octet AS Specific Extended Community
RFC 6793 BGP Support for Four-Octet Autonomous System (AS) Number Space
RFC 7300 Reservation of Last Autonomous System (AS) Numbers
RFC 8092 BGP Large Communities Attribute
RFC 8654 Extended Message Support for BGP

OSPFv2

RFC 2328 OSPF Version 2

Inter-operating with Cisco OSFPv2

Cisco routers enable RFC 5613 "Link Local Signaling" (LLS) by default. Flock Networks OSPFv2 does not yet support RFC5613. To inter-operate LLS needs to be disabled on the Cisco device.

router ospf <n>
  no capability lls

If you are hitting this problem, the Flock Networks router will be logging a warning.

[WARN  ospf::ospf_intf] Rx OSPF Header sanity failed: "Rx Pak Hello ignore, Options mismatch, expected [E] got [E | L] from RouterId(192.168.0.4)

Manual Install

Flock Networks Routing Suite can be manually installed on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc).

Extract the contents of the Debian package

You will need the ar archive utility from binutils.

For example on Fedora;

# yum install binutils

Then extract the contents of the debian package

$ ar x flockd_20.3.x_amd64.deb
$ ls
control.tar.gz  data.tar.xz  debian-binary  flockd_20.3.x_amd64.deb
$ tar -xf data.tar.xz
$ ls
control.tar.gz  debian-binary  flockd_20.3.x_amd64.deb  usr data.tar.xz etc lib

Install Files

The copyright / software licence file is here;

less usr/share/doc/flockd/copyright

Copy the flockd daemon config files to /etc/flockd

# mkdir /etc/flockd
# cp etc/flockd/* /etc/flockd

Copy the flockd daemon to /usr/sbin

# cp usr/sbin/flockd /usr/sbin

Copy the flockc client to /usr/bin

# cp usr/bin/flockc /usr/bin

Start the daemon

# RUST_LOG=info /usr/sbin/flockd &

Check the daemon is running

$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.3.0"
"model": "Multi-threaded"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
"fpm_state": "disabled"
$

SONiC Support

The Flock Networks Routing Suite can act as the IP Routing control plane for the SONiC Network Operating System. SONiC uses the forwarding plane manager (FPM) to program fpmsyncd. fpmsyncd then sends the routing updates into the SONiC APPL_DB database.

To program fpmsyncd the system configuration file /etc/flockd/system.toml requires these two lines of configuration.

[fpm]
tcp_port = 2620

Then check FPM is enabled by using the REST API.

$ flockc system | grep fpm_state
"fpm_state": "enabled"
$

NB: The default IP Routing suite shipped with SONiC will need to be disabled, or it will conflict with the Flock Networks IP Routing Suite.

Debugging SONiC

An overview of the SONiC routing-state interactions is here.

Check the `fpmsyncd` logs for a successful connection from the Flock Networks IP Routing Suite.

# grep "fpmsyncd Connected" /var/log/messages
2020-08-25T08:44:49.707069+00:00 ad2e7e7845ad supervisord: fpmsyncd Connected!

Check the ASIC's have been programmed

The entries below match this SONiC P4 example.

host1 (Ubuntu, 192.168.1.2/24) <--> switch1 (SONiC) <--> switch2 (SONiC) <--> host2 (Ubuntu, 192.168.2.2/24)

Use the SONiC saidump utility to show the programmed ASIC entries. Below are shown the entries used to forward to the remote BGP subnet 192.168.2.0/24.

switch1$ saidump

# Remote BGP subnet
SAI_OBJECT_TYPE_ROUTE_ENTRY {"dest":"192.168.2.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000066"}
    SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID : oid:0x400000000009c

# Which uses this nexthop
SAI_OBJECT_TYPE_NEXT_HOP oid:0x400000000009c
    SAI_NEXT_HOP_ATTR_IP                  : 10.0.0.1
    SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID : oid:0x6000000000095
    SAI_NEXT_HOP_ATTR_TYPE                : SAI_NEXT_HOP_TYPE_IP

# Which is out of this interface
SAI_OBJECT_TYPE_ROUTER_INTERFACE oid:0x6000000000095
    SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS   : 00:01:04:4C:49:F5
    SAI_ROUTER_INTERFACE_ATTR_TYPE              : SAI_ROUTER_INTERFACE_TYPE_VLAN
    SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID : oid:0x3000000000066
    SAI_ROUTER_INTERFACE_ATTR_VLAN_ID           : oid:0x2600000000008f

User Guide: Flock Networks Routing Suite 20.3 [Copyright 2020]