Introduction
Welcome to the User Guide for Flock Networks IP Routing Suite 21.1.
The Flock Networks IP Routing Suite will run on any version of Linux, including the SONiC NOS.
At Flock Networks we believe networks should be fast, secure and uniform.
- Performance is achieved by:
- Running in a single operating system process, with each component having its own thread. This makes the communication between components dramatically faster.
- Using a thread pool in each component to achieve linear scale vs available CPU cores.
- Keeping the entire implementation lock free. This means each work flow can progress unhindered by any external interruptions.
- Handling network protocol updates in batches. This means route calculation can be performed less frequently, and keeps the CPU's instruction cache hot.
- Only using data structures that are CPU data cache friendly.
- Security is achieved by the routing suite being exclusively written in the Rust programming language. Rust has a similar performance to C / C++, but is memory safe at compile time.
- Uniformity is achieved by having a similar configuration across all routers in the network. Device specific and in particular interface specific configuration is to be avoided.
If you have any questions, suggestions or issues, please email support@flocknetworks.com.
Installation
The Flock Networks Routing Suite is shipped as a Debian package. However the Routing Suite can run on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc). The following instructions are for installing on a Debian based Linux distribution (Debian, Ubuntu, Mint, etc). To install on other systems, see Manual Install. The Flock Networks Routing Suite can also be used as a drop in replacement on the SONiC Network Operating System, see SONiC.
Download the latest Debian package.
Install the application from the Debian package (as root or using sudo);
# dpkg -i flockd_20.4.x_amd64.deb
The Flock Routing Suite Licence is appended to;
/usr/share/doc/flockd/copyright
The Flock Routing Suite Client flockc is copied to;
/usr/bin/flockc
The Flock Routing Suite Daemon flockd is copied to;
/usr/sbin/flockd
All Flock Routing Suite flockd configuration is stored in a single file;
/etc/flockd/flockd.json
flockd is controlled by systemd;
$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd
logging can be viewed via journalctl;
# journalctl -u flockd --boot
Example successful installation
# dpkg -i flockd_20.4.0_amd64.deb
Selecting previously unselected package flockd.
(Reading database ... 27695 files and directories currently installed.)
Preparing to unpack flockd_20.4.0_amd64.deb ...
Unpacking flockd (20.4.0) ...
Setting up flockd (20.4.0) ...
# systemctl start flockd
# systemctl status flockd
● flockd.service - Flock Networks Routing Suite Daemon
Loaded: loaded (/lib/systemd/system/flockd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-24 12:39:24 GMT; 6s ago
Main PID: 368 (flockd)
Tasks: 1 (limit: 1150)
Memory: 3.2M
CGroup: /system.slice/flockd.service
└─785 /usr/sbin/flockd
Nov 17 15:39:15 debian10-clean systemd[1]: Started Flock Networks Routing Suite Daemon.
Nov 17 15:39:15 debian10-clean flockd[368]: [INFO flockd::sys::sys_inst] System Up: version=20.4.0 model="Multi-threaded" pid=3
Nov 17 15:39:15 debian10-clean flockd[368]: [INFO flockd::sys::sys_inst] REST API TcpListener { addr: 0.0.0.0:8000, fd: 4 }
$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 368
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 14 }
"enabled_protocols": []
Troubleshooting Installation
flockd fails to start
- systemd PolicyKit / polkit failure
PolicyKit / polkit is the "sudo of systemd". If you have PolicyKit / polkit active on the install host, and you want to start flockd as an unprivileged user, you will need to add PolicyKit / polkit configuration for flockd. Alternatively you can just run systemctl start flockd as root or under sudo.
$ systemctl start flockd
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'flockd.service'.
Authenticating as: www-data
Password:
Failed to start flockd.service: Connection timed out
See system logs and 'systemctl status flockd.service' for details.
polkit-agent-helper-1: pam_authenticate failed: Authentication failure
Flock Routing Suite Daemon flockd
Overview
The flockd process is controlled by systemd.
$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd
flockd is a single process that is split into separate components.
System Component
The system component manages the other components and presents a unified API to the outside world.
The system component monitors external signals and updates the other components as required. Example external signals would be the addition of an IP address to an interface, or the addition of a route to the Linux Kernel.
The system component provides the configuration and operations API.
Routing Information Base (RIB) Component
The RIB component receives route updates from other components and from the Linux Kernel. The RIB component selects the best update for each route and redistributes it to other components and to the Linux Kernel. This redistribution can be controlled by configuration. By default the RIB will send the best route to the Linux Kernel and will accept any routes from the Linux Kernel that are not sourced by flockd.
OSPFv2 Component
The OSPFv2 component runs the OSPFv2 protocol. This component gets interface state information and connected IP subnet information from the System Component. This component gets external state information by connecting to OSPFv2 neighbors via Linux Kernel raw IP sockets. This information is combined and each best route is derived and sent to the RIB component.
BGPv4 Component
The BGPv4 component runs the BGPv4 protocol. This component gets IP route information from the RIB component and external state information by connecting to BGP neighbors via Linux Kernel TCP sockets. This information is combined and each best route is derived and sent to the RIB component.
Static Component
The Static component manages the configuration of IPv4/IPv6 static routes. Static routes have a default administrative distance of 1 and therefore are preferred over dynamic routes by default. However, Static component allows you to specific a non-default administrative distance and thereby allowing a static route to be used as a backup route to a dynamic route. The configured static routes can be either recursive or non-recursive by specifying an explicit outgoing interface.
Multithreading
The routing suite runs in a single process and each component runs in its own thread. One way to view the active threads is via the Linux /proc virtual file system. The System, RIBv4, RIBv6, OSPFv2 and BGPv4 components are allocated a thread each. If a component is not enabled, it will not have any threads allocated.
The BGPv4 master thread travels with a BGP thread pool. The BGP thread pool will have a thread for each logical CPU core on the router.
flock@flocknet:~$ grep Name /proc/`pidof flockd`/task/*/status
/proc/409/task/409/status:Name: flockd
/proc/409/task/432/status:Name: OSPFv2 Master
/proc/409/task/433/status:Name: BGPv4
/proc/409/task/434/status:Name: RIBv4 Master
/proc/409/task/435/status:Name: RIBv6 Master
/proc/409/task/440/status:Name: BGPv4
/proc/409/task/441/status:Name: BGPv4
/proc/409/task/442/status:Name: BGPv4
/proc/409/task/443/status:Name: BGPv4
flock@flocknet:~$
In the example above:
-
The entire routing suite is running in a single process (process id
409). -
The thread with thread id
409is theflockdsystem thread. -
The thread with thread id
433is the BGP Master thread. The threads with id's440,441,442and443belong to the BGP thread pool. There are 4 threads in the BGP thread pool as the router has 4 logical CPU's.flock@flocknet:~$ cat /proc/cpuinfo | grep -c processor 4 flock@flocknet:~$
Logging
Logging can be viewed using journalctl.
View flockd logs since router was booted.
# journalctl -u flockd --boot
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:38:44 BST. --
Jun 29 08:49:36 r61 flockd[455]: [INFO flockd] START: PID=455, Compile Mode=Release, Log Level="info"
...
View flockd logs since a certain time.
# journalctl -u flockd --since "2020-06-26 17:19:20"
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:40:55 BST. --
Jun 26 17:19:20 r61 flockd[431]: [INFO flockd::bgp::bgp_neigh] 90.0.93.70/Some(BgpId(70.0.100.70)) BgpAsn2(70) Outgoing FSM state change OpenConfirm -> Established
...
Monitor for the latest flockd logs.
# journalctl -u flockd --follow
-- Logs begin at Fri 2020-01-10 14:01:47 GMT. --
Jun 29 09:44:34 r61 flockd[455]: [INFO flockd::bgp::bgp_neigh] 60.0.60.60/Some(BgpId(60.0.100.60)) BgpAsn2(60) Incoming FSM state change OpenConfirm -> Established
...
Log Levels
Log levels can be set using the RUST_LOG environment variable.
Flock Routing Suite Client flockc
flockc is the client provided to query the current state of flockd. By design flockc can be run by a user with no special privileges. flockc connects to a Read-Only connection on flockd. flockd state can be viewed but cannot be changed. All output is in JSON format, making it suitable for consumption by humans or machines.
flockc connects to a REST API on flockd. This REST API will talk to any HTTP based client. This means any HTTP based client can be used to monitor flockd. Just connect the client to the same URL that flockc uses. Use the flockc --show-url option to display the URL associated with the flockc command.
For example:
$ flockc system --show-url
http://::1:8000/system/sort/json-lines
See here for more information on using the REST API.
Local Connectivity
flockc uses the REST API via the local IP Looback address to retrieve state information from the local flockd.
This command shows all the IPv4 Prefixes in the RIB of the local flocknet1 router.
flock@flocknet1:~$ flockc rib --af ipv4 --prefix
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2, "ip_addr":"10.0.1.168"},{"intf_id":3,"ip_addr":"10.0.2.203"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
...
Remote Connectivity
flockc can connect and retrieve the same state information from a remote router. Just add the --host option to any command. The host parameter can be a hostname or an IP Address.
This command shows all the IPv4 Prefixes in the RIB of the remote flocknet2 router.
flock@flocknet1:~$ flockc rib --af ipv4 --prefix --host flocknet2
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2, "ip_addr":"10.0.11.168"},{"intf_id":3,"ip_addr":"10.0.22.203"}]}
{"ip_net":"10.0.11.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.22.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
...
flockc is available on its own. Install this package if you want to remotely manage your network from a host.
# dpkg -i flockc_20.4.x_amd64.deb
General Command Flags
General Flags can be included in any flockc command.
-d, --detail
-h, --help
-J, --json Output in JSON
-j, --json-pretty Output in Pretty Print JSON
-u, --unsorted Output in unsorted order
Changing the output format
By default the output is in JSON Lines format, which is JSON with newlines added to aid reading by a human. The output can also be changed to JSON Pretty or vanilla JSON.
JSON Lines format
flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3
{"ospf_area_id":"0.0.0.0"}
{"ospf_intf":"enp1s0"}
{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.225","bdr":"10.0.5.204"}
JSON Pretty format
Using the -j, --json-pretty flag the output is pretty printed JSON.
flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3 -j
[
{
"ospf_area_id": "0.0.0.0",
"ospf_intfs": [
{
"ospf_intf": "enp1s0",
"ospf_neighs": [
{
"id": "10.0.100.3",
"ip": "10.0.5.225",
"state": "Full",
"dr": "10.0.5.225",
"bdr": "10.0.5.204"
}
]
}
]
}
]
JSON format
Using the -J, --json flag the output is JSON. This is the flag to use when talking to another application which wants to deserialize the JSON string into its own representation.
flock@flocknet:~$ flockc ospfv2 --neigh 10.0.100.3 -J
[{"ospf_area_id":"0.0.0.0","ospf_intfs":[{"ospf_intf":"enp1s0","ospf_neighs":[{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.225","bdr":"10.0.5.204"}]}]}]
System Component
Configuration Overview
The system configuration is held under the top level system object in /etc/flockd/flockd.json. The system object must exist and it must contain an api object. The api object can be left empty.
flock@flocknet:~$ cat /etc/flockd/flockd.json
{
"system": {
"api": {}
}
}
flock@flocknet:~$
Configuration in detail
Operations REST API Configuration
The Operations REST API is used to view flockd internal state. It is a 'read only' API. By design state can be viewed but cannot be changed using this API.
The Operations REST API is configured using the rest_api JSON object. The rest_api object is optional. If the object is not specified the Operations REST API will not be started.
This configuration will bind to all local IPv6 addresses.
"system": {
"api": {
"rest": {
"bind_ip_addr": "::"
}
}
},
This configuration will bind to only the loopback IPv6 address. The Operations REST API will not be available from outside of the router.
"system": {
"api": {
"rest": {
"bind_ip_addr": "::1"
}
}
},
By default a Linux host will operate as an IPv4/IPv6 dual stack node, meaning IPv4 requests will be serviced. If you only want to service IPv6 requests please see Bind IPv6 Only
This configuration will bind to all local IPv4 addresses.
"system": {
"api": {
"rest": {
"bind_ip_addr": "0.0.0.0"
}
}
},
This configuration will bind to only the loopback IPv4 address. The Operations REST API will not be available from outside of the router.
"system": {
"api": {
"rest": {
"bind_ip_addr": "127.0.0.1"
}
}
},
Filtering interfaces in system component
System component receives interfaces from the Linux kernel. Interfaces not used by any routing component can be filtered out at the system component level by using intf_denylist JSON object.
This configuration will ignore interface eth0 and all interfaces starting with en:
"system": {
...
"intf_denylist": [
# Specify the entire interface name i.e. "eth0"
# or use ^ to match the start of interface names
# e.g. "^en" will match all interfaces that start with "en"
# Multiple interface names and patterns can be used
"eth0",
"^en"
]
},
The intf_denylist object is optional. If it is not specified, all interfaces present in kernel will also be present in system component.
The filtered out interfaces will not be present in the flockc system -i command output.
Operational State Overview
Check status of flockd
flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
flock@flocknet:~$
Show all system interfaces
flock@flocknet:~$ flockc system -i
{"name":"dummy0","id":7,"ip_prefixes":["60.0.20.61/32"],"state":"Up"}
{"name":"enp10s0","id":4,"ip_prefixes":[],"state":"Down"}
{"name":"enp1s0","id":2,"ip_prefixes":["90.0.93.61/24"],"state":"Up"}
{"name":"enp7s0","id":5,"ip_prefixes":[],"state":"Down"}
{"name":"enp8s0","id":6,"ip_prefixes":["60.0.60.61/24"],"state":"Up"}
{"name":"enp9s0","id":3,"ip_prefixes":["90.0.91.61/24"],"state":"Up"}
{"name":"lo","id":1,"ip_prefixes":["127.0.0.1/8"],"state":"Up"}
flock@flocknet:~$
Show single system interface
flock@flocknet:~$ flockc system -i enp1s0
{"name":"enp1s0","id":2,"ip_prefixes":["90.0.93.61/24"],"state":"Up"}
flock@flocknet:~$
Show all vrfs
flock@flocknet:~$ flockc system -v
{"vrf_id":0,"vrf_name":"default"}
{"vrf_id":11,"vrf_name":"blue","kernel_intf_name":"blue"}
{"vrf_id":12,"vrf_name":"pink","kernel_intf_name":"pink"}
flock@flocknet:~$
Show single vrf
flock@r70-deb11:~$ flockc system -v pink
{"vrf_id":12,"vrf_name":"pink","kernel_intf_name":"pink"}
flock@r70-deb11:~$
System Operation
Help
flockc system -h
Overview
flockc system
All system interfaces
flockc system -i
Single interface
flockc system -i <interface-name>
All vrfs
flockc system -v
Single vrf
flockc system -v <vrf-name>
RIB Component
The RIB component receives routes from each routing protocol and stores them in the RIB. For each network prefix the 'best' route is sent to the dataplane for use when forwarding packets. By default, flock uses the Linux kernel as dataplane, but it can be configured to use other dataplanes.
Configuring the RIB component
RIB component configuration is under the top level rib object in /etc/flockd/flockd.json. The default configuration uses the Linux kernel as dataplane, and programs the routes via netlink:
{
"rib": {
"dataplane": {
"netlink": true,
}
}
}
If the rib JSON object doesn't exist, or is empty of null, flock falls back the default configuration shown above. These configurations are all equivalent:
{}
{"rib": {}}
{"rib": null}
{"rib": {"dataplane": {}}
{"rib": {"dataplane": null}
{"rib": {"dataplane": {"netlink": true}}}
To use an alternative dataplane via FPM (Forwarding Plan Manager):
{
"rib": {
"dataplane": {
"netlink": false,
"fpm": {
"tcp_port": 2620
}
}
}
}
Operational State
Overview
flock@flocknet:~$ flockc rib --af ipv4
{"route_count":2328}
flock@flocknet:~$
flock@flocknet:~$ flockc rib --af ipv6
{"route_count":5340}
flock@flocknet:~$
In all RIB commands IPv6 is the default address family
flock@flocknet:~$ flockc rib
{"route_count":5340}
flock@flocknet:~$
Longest Path Match (LPM) for destination
flock@flocknet:~$ flockc rib --af ipv4 -l 10.0.2.34
{"origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
flock@flocknet:~$
Single RIB prefix entry
flock@flocknet:~$ flockc --af ipv4 -p 10.0.2.0/24
{"origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
flock@flocknet:~$
All prefix entries in the RIB
flock@flocknet:~$ flockc --af ipv4 -p
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.22.1"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.49"}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
{"ip_net":"42.0.0.0/16","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.68.122.1"}]}
{"ip_net":"192.168.122.0/24","origin":"Kernel","next_hops":[{"intf_id":5}]}
flock@flocknet:~$
RIB Operation
Help
flockc rib -h
RIB Overview
flockc rib --af [ipv4 | ipv6 ]
RIB Prefixes
flockc rib --af [ ipv4 | ipv6 ] -p [<ip-network>]
Prefixes by route origin.
Origin can be bgp, kernel, ospfv2 or static.
flockc rib --af [ipv4 | ipv6 ] -p -o ospfv2
Longest Path Match (LPM)
flockc rib --af [ipv4 | ipv6 ] -l <ip-address>
OSPFv2 Component
Enabling the OSPFv2 Component
The OSPFv2 configuration is held under the top level ospfv2 object in /etc/flockd/flockd.json. If the ospfv2 object exists OSPFv2 will be enabled and the OSPFv2 master thread will be started.
The Flock Networks Routing Suite is designed for massive scale so placing all routers in a single OSPF area is recommended. (If you are adding a device to an existing multi-area OSPF Autonomous System, multiple areas are fully supported).
With this configuration file:
-
The OSPFv2 master thread will be started
-
The OSPFv2 router will advertise a router id of
10.0.100.2 -
All interfaces with names starting with
enwill be placed in OSPF area 0."ospfv2": { "router_id": "10.0.100.2", "area": [ { "area_id": "0.0.0.0", "intf": [ { "name": "^en" } ] } ] }
This is all the OSPFv2 configuration you need, to create an OSPF network as large as you like. Each device has an identical configuration which simplifies the operation of the network. A management station can easily determine all the Router Id's in the network by querying a single device for all of its Router LSA's.
Redistribution of routes into OSPF
You may wish to redistribute routes from the RIB into OSPFv2. Use the redistribute json object. The origin field specifies the protocol that programmed the routes into the RIB.
flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
"redistribute": [
{
"metric": 200,
"metric_type": 2,
"origin": "static"
}
]
}
As a minimum we may want a default route added to the kernel of each ASBR router. This route will appear in the RIB and then be redistributed into OSPFv2. OSPFv2 will advertise this route across the AS, so all nodes learn the route to exit the network. Static routes are added using the flockd static component.
flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"static": {
"static_routes": [
{
"ip_net": "0.0.0.0/0",
"next_hops": [
{
"ip_addr": "192.168.122.171",
"intf_name": "enp8s0"
}
]
}
]
}
Implicit Router Id
With no explicit configuration the highest IPv4 Address is used as the Router Id. IPv4 Addresses on loopback interfaces are always preferred over IPv4 Addresses on physical interfaces.
Explicit Router Id
To explicitly set the Router Id to 10.0.100.1.
flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
"router_id": "10.0.100.1",
...
}
Advertising the IPv4 Address that is being used as the Router Id
When operating a network it can be useful to have the Router Id's advertised as an IPv4 host route in OSPF. This means the Router Id will respond to network operation tools such as ping and traceroute. To do this create a loopback interface and assign the IPv4 host route to it. Then enable OSPF on the interface that is providing the Router Id IPv4 address.
Create a loopback interface and assign the IPv4 host route to it.
The method for permanently adding IP addresses to loopback interfaces is Linux distribution specific. For example Debian uses the /etc/network/interfaces file.
flock@flocknet:~$ cat /etc/network/interfaces
...
# The loopback network interface
auto lo
iface lo inet loopback
# Add IPv4 Address to be used as RouterId
auto lo:20
iface lo:20 inet static
address 70.0.100.71/32
...
flock@flocknet:~$ sudo systemctl restart networking
flock@flocknet:~$
Enable OSPF on the loopback interface
flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
"area": [
{
"area_id": "0.0.0.0",
"intf": [
{
"name": "lo"
}
]
}
]
},
Check the Router Id is as expected
flock@r71:~$ flockc ospfv2 -j | grep router_id
"router_id": "70.0.100.71",
flock@flocknet:~$
Operational State Overview
Check OSPFv2 is enabled
Check OSPFv2 is listed in the enabled_protocols field.
flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
flock@flocknet:~$
Show OSPFv2 Overview
flock@r01:~$ flockc ospfv2
{"router_id":"10.0.100.4","class":"IR","redistribute":[],"neigh_state_count":{"down":0,"attempt":0,"init":0,"two_way":0,"ex_start":0,"exchange":0,"loading":0,"full":6}}
flock@r01:~$
Show all neighbors (out of all interfaces, in all areas)
flock@r01:~$ flockc ospfv2 -n
{"ospf_area_id":"0.0.0.0"}
{"ospf_intf":"enp1s0"}
{"id":"10.0.100.3","ip":"10.0.5.225","state":"Full","dr":"10.0.5.204","bdr":"10.0.5.225"}
{"ospf_area_id":"0.0.0.20"}
{"ospf_intf":"enp7s0"}
{"id":"10.20.100.20","ip":"10.20.20.189","state":"Full","dr":"10.20.20.189","bdr":"10.20.20.214"}
flock@r01:~$
Show Area 0 Link State Database
flock@flocknet:~$ flockc ospfv2 -a 0 -l
{"lsa_age":279,"lsa_opts":{"bits":2},"lsa_type":"Router","lsa_id":"10.0.100.4","lsa_router_id":"10.0.100.4","lsa_seq":-2147483646,"lsa_checksum":28411,"lsa_len":36}
{"lsa_age":266,"lsa_opts":{"bits":2},"lsa_type":"Router","lsa_id":"10.0.100.5","lsa_router_id":"10.0.100.5","lsa_seq":-2147483646,"lsa_checksum":22802,"lsa_len":36}
...
flock@flocknet:~$
Annotated OSPFv2 Configuration
"ospfv2": {
# RFC2328 1.2 Router ID
# Optional: If not specified highest IPv4 Address is used.
"router_id": "String in dotted decimal format"
# Array of 'redistribute' objects
"redistribute": [
{
# Origin of the Routes in the RIB to be redistributed
"origin": ["kernel-static" | "kernel-connected"],
# RFC2328 2.3 Type 1 / Type 2 external metrics
"metric_type": [ 1 | 2 ],
# OSPF metric to reach redistributed routes, from this router.
# RFC2328 B. LSInfinity => 16777215
"metric": ( 0..16777215 )
},
],
# OSPFv2 Area level configuration
# -------------------------------
[[area]] # Array of 'area' objects
"area": [
{
# RFC2328 C.2 Area ID
"area_id": "String in dotted decimal format",
# OSPFv2 Interface level configuration
# ------------------------------------
"intf": [
{
# Specify the entire interface name i.e. "eno1"
# or use ^ to match the start of interface names
# e.g. "^en" will match all interfaces that start with "en"
"name": "Interface Name",
# RFC2328 C.3 Interface output cost
# Optional: Default is 10
"cost": ( 1..65,535 ),
# RFC2328 C.3 Router Priority
# Optional: Default is 1
"priority": ( 0..255 ),
# RFC2328 C.3 HelloInterval
# Optional: Default is 10s
"hello_interval": ( 1..65,535 seconds),
# RFC2328 C.3 RouterDeadInterval
# Optional: Default is 40s
"dead_interval": ( 1..65,535 seconds),
}
]
}
]
}
Example Exhaustive OSPFv2 Configuration
"ospfv2": {
"router_id": "10.0.1.100",
"redistribute": [
{
"metric": 100,
"metric_type": 1,
"origin": "kernel-static"
},
{
"metric": 1000,
"metric_type": 2,
"origin": "kernel-connected"
}
],
"area": [
{
"area_id": "0.0.0.0",
"intf": [
{
"name": "enp0s0",
"cost": 20,
"dead_interval": 4,
"hello_interval": 1,
"name": "enp7s0",
"priority": 10
},
{
"name": "^eth",
"cost": 40,
"dead_interval": 8,
"hello_interval": 2,
"name": "enp8s0",
"priority": 20
}
]
},
{
"area_id": "0.0.0.1",
"intf": [
{
"name": "enp1s0",
"cost": 30,
"dead_interval": 80,
"hello_interval": 20,
"name": "enp9s0",
"priority": 30
}
]
}
]
}
OSPFv2 Operation
Help
flockc ospfv2 -h
Overview
flockc ospfv2
Overview of areas
flockc ospfv2 -a [<area-id>]
All interfaces in Area 20
flockc ospfv2 -i -a 20
All neighbors on interface enp1s0 in Area 0
flockc ospfv2 -n -i enp1s0 -a 0
All neighbors on all interfaces in all areas
flockc ospfv2 -n
Autonomous System Link State Database
flockc ospfv2 -l
Area 0.0.0.0 Link State Database
flockc ospfv2 -a 0 -l
Network route table prefixes
flockc ospfv2 -p [<ipv4-network>]
Router route table prefixes
flockc ospfv2 -P [<router-id>]
BGP Component
Overview
Enabling the BGP Component
The BGP configuration is held under the top level bgp object in /etc/flockd/flockd.json. If the bgp object exists BGP will be enabled and the BGP master thread will be started.
With this configuration file:
-
The BGP master thread will be started.
-
The router is in an Autonomous System identified by the Autonomous System Number
65016 -
The router has a BGP router identifier assigned as
172.16.10.1"bgp": { "local": { "id": "172.16.10.1", "asn": 65016 } }
Show the status of the BGP component
Check BGP is listed in the enabled_protocols field.
flock@r01:~$ flockc system
"hostname": "r01"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["BGP"]
flock@r01:~$
Show BGP Overview
flock@r01:~$ flockc bgp
{"id":"172.16.10.1","asn":65016,"routes":{"ipv4_unicast":4271,"ipv6_unicast":4760},"neighbor_summary":{"count":4,"established":3,"send_converged":2,"recv_converged":1}}
flock@r70:~$
send_converged means all updates have been sent to this neighbor. The neighbor send update queue is empty. The neighbor may not have received all the updates yet, they may still be in the local TCP send buffer (or the neighbors TCP receive buffer)
recv_converged means all available updates from this neighbor have been processed. The neighbor receive TCP buffer is empty. However the neighbor may not have managed to send all updates yet.
- There are 4271 IPv4 Unicast routes in the BGP RIB
- There are 4760 IPv6 Unicast routes in the BGP RIB
- There are 4 neighbors, 3 of which have reached
establishedstate - 2 of the neighbors are
send_converged - 1 of the neighbors is
recv_converged
Originating Networks
With this configuration file:
-
The router originates the
172.16.0.0/16andfc00:46::/32networks"bgp": { "local": { "id": "172.16.10.1", "asn": 65016, "network": [ { "ip_prefix": "172.16.0.0/16" }, { "ip_prefix": "fc00:46::/32" } ] } flock@r70:~$ flockc bgp --af=ipv4 -p 70.0.0.0/8 {"best_entry":{"reason":"SelfOriginated"}} -
If not specified the address family defaults to
--af=ipv6flock@r70:~$ flockc bgp -p {"ip_net":"fc00:46::/32","best_entry":{"reason":"SelfOriginated"}}
Configuring Neighbors
With this configuration file:
-
The router has a single iBGP neighbor
172.16.10.2- The iBGP connection source is
172.16.10.1 - The iBGP connection will advertise IPv4 unicast routes
- Routes are advertised over iBGP with a next hop of
172.16.10.1(next_hop_self)
- The iBGP connection source is
-
The router has a single eBGP neighbor
172.17.20.1in remoteAS 65017"bgp": { "local": { "id": "172.16.10.1", "asn": 65016, } "as": [ { "asn": 65016, "next_hop_self": true, "neighbor": [ { "ip": "172.16.10.2", "local_ip": "172.16.10.1", "af": [ { "afi": "ipv4", "safi": "unicast" } ] } ] }, { "asn": 65017, "neighbor": [ { "ip": "172.17.20.1", "af": [ { "afi": "ipv4", "safi": "unicast" } ] } ] } ] }
Show all neighbors (in all Autonomous Systems)
BGP runs two Finite State Machines (FSM's) per neighbor. One FSM handles the Outgoing TCP connection and the other handles the Incoming TCP connection. The Flock Routing Suite does not hide this from the operator. In the final working state each neighbor should have one FSM in the Established state, and one FSM in the Idle state.
The last error to cause a BGP Notify Message is held in each FSM's last_notify field. This field is never cleared, it is only overwritten with the last error. So a value of null means there have been no errors that have caused a notify message since flockd was started.
flock@r61:~$ flockc bgp -n
{"asn":50}
{"ip_addr":"50.0.20.50","local_ip_addr":"60.0.20.61","asn":50,"bgp_id":"50.0.100.50","neigh_type":"External"}
{"tcp":"Outgoing", "state":"Established","last_notify":null,"updates_sent":42,"updates_queued":0,"recv_converged":true}
{"tcp":"Incoming", "state":"Idle","last_notify":null",updates_sent":0,"updates_queued":0,"recv_converged":false}}
{"asn":60}
{"ip_addr":"60.0.60.60","asn":60,"bgp_id":"60.0.100.60","neigh_type":"Internal"}
{"tcp":"Outgoing","state":"Established","last_notify":null,"updates_sent":73,"updates_queued":0,"recv_converged":true}
{"tcp":"Incoming", "state":"Idle","last_notify":null,"updates_sent":0,"updates_queued":0,"recv_converged":false}
Show BGP RIB prefix's
Note that this is not the RIB held in the RIB component, this is the BGP RIB. The BGP RIB records routes from all neighbors and sends the 'best entry' route to the RIB component. By default, BGP will show the ipv6 routes if the af parameter is not specified.
Show all prefix's. Only the 'best entry' for each prefix is shown, along with the reason why it was the best.
flock@r01:~$ flockc bgp -p --af ipv4
{"ip_net":"50.0.0.0/8","best_entry":{"neigh":{"neigh_ip_addr":"50.0.20.50","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}},"reason":"OnlyValidPeer"}}
{"ip_net":"60.0.0.0/8","best_entry":{"reason":"SelfOriginated"}}
Show a specific prefix. The 'best entry' and all the candidate entries are shown.
flock@r61:~$ flockc bgp -p 50.0.0.0/8 --af ipv4
{"best_entry":{"neigh":{"neigh_ip_addr":"50.0.20.50","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}},"reason":"OnlyValidPeer"},"neighboring_as":[{"med_origin_asn":50,"via_neighs":[{"neigh_ip_addr":"50.0.20.50","neigh_bgp_id":"50.0.100.50","neigh_type":"External","attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[50]}]},"next_hop":"50.0.20.50","med":null,"local_pref":null,"atomic_aggregate":false,"aggregator":null}}]}]}
Configuring BGP Active / Passive Neighbors
By default BGP will try to create two TCP transport connections to each neighbor. One outgoing to the neighbors remote BGP TCP port 179, and one allowing incoming connections from the neighbor to the local BGP TCP port 179. A tie break is used to enusure only one connection remains when the BGP neighbor moves to the 'Established' state.
The router can be configured to only form a single TCP transport connection to each neighbor using the connect_mode neighbor configuration parameter.
bgp {
"as": [
"neighbor": [
# Only create the outgoing connection to this neighbor.
# Refuse any incoming connection.
"connect_mode": "active"
]
]
}
or
# Only allow the incoming connection from this neighbor.
# Do not create any outgoing connection.
"connect_mode": "passive"
Configuring BGP Route Reflectors
To configure a router as a BGP Route Reflector, specify which neighbors are Route Relector clients using the route_reflector_client configuration boolean.
bgp {
"as": [
"neighbor": [
# Reflect iBGP routes to and from this neighbor
"route_reflector_client": "true"
]
]
}
To deploy redundant Route Reflectors a Route Relector Cluster Id can optionally be configured.
bgp {
"local": {
"cluster_id": "1.2.3.4"
}
}
Configuring BGP to act as a Route Server
BGP Route Server functionality is defined in RFC7947. To configure a router as a BGP Route Server use the route_server configuration boolean.
bgp {
"as": [],
"local": {
"asn": 65056,
"id": "192.168.0.14",
"route_server": true
}
}
To check BGP is running as a route server.
flock@r01:~$ flockc bgp -j | grep route_server
"route_server": true,
flock@r01:~$
Configuring Multihop BGP
Multihop BGP is configured by changing the Time to Live (TTL) of the BGP packets that are sent.
The default BGP packet TTL's are iBGP = 64 and eBGP = 1.
Use the neighbor ttl configuration keyword to override the defaults.
bgp {
"as": [
{
"asn": 60,
"neighbor": [
{
"ip": "60.0.20.61",
"ttl": {
"send": 2
}
}
]
}
]
}
BGP Operation Commands Reference
Help
flockc bgp -h
Overview
flockc bgp
Autonomous System
flockc bgp -a [<asn>]
Neighbors
flockc bgp -n [<ip-addr>]
Prefixes
flockc bgp --af=<ipv4|ipv6> -p [<ip-network>]
Static Component
Configuration Overview
The static route configuration is held under the top level static object in /etc/flockd/flockd.json. If the static object exists static routes will be enabled and the static master thread will be spawned.
In the minimal configuration file below;
- The
vrfobject is an array of vrfs, indexed byvrf_name. - The
static_routesobject is an array of static routes, indexed byip_net. - The
ip_netobject is used to specify the static route prefix. The prefix can be either IPv4 or IPv6. - The
next_hopsobject is used to specify a set of next-hop(s) for the static route. - Each next-hop object is defined as either a destination
ip_addror an outgoingintf_nameor both.
In this example there is a single vrf containing a single static route to the 20.20.20.0/24 network. Traffic for that network will be forwarded out of interface ens1 to the neighboring device with the address 10.10.10.2.
"static": {
"vrf": [
{
"vrf_name": "default",
"static_routes": [
{
"ip_net": "20.20.20.0/24",
"next_hops": [
{
"ip_addr": "10.10.10.2",
"intf_name": "ens1"
}
]
}
]
}
]
}
Configuration in detail
If a next_hop only contains an ip_addr the static route is said to be "recursive" as a further lookup is needed to find the outgoing interface. If a next_hop has an outgoing intf_name defined the static route is said to be "non-recursive".
"Floating static" routes can also be defined using the distance object. A "floating static" route is given a high distance (Admin Distance) which means during normal operation the RIB will choose to use another protocol that has a lower distance. If there is a network outage and the preferred route is withdrawn from the RIB, the "floating static" route will take over.
Recursive route configuration
"static": {
"vrf": [
{
"vrf_name": "default",
"static_routes": [
{
"ip_net": "20.20.20.0/24",
"next_hops": [
{
"ip_addr": "10.10.10.2"
}
]
}
]
}
]
}
Non-recursive route configuration
"static": {
"vrf": [
{
"vrf_name": "default",
"static_routes": [
{
"ip_net": "20.20.20.0/24",
"next_hops": [
{
"ip_addr": "10.10.10.2",
"intf_name": "ens1"
}
]
},
{
"ip_net": "10:10::0/64",
"next_hops": [
{
"intf_name": "ens1"
}
]
}
]
}
]
}
Multi-path recursive route configuration
"static": {
"vrf": [
{
"vrf_name": "default",
"static_routes": [
{
"ip_net": "20.20.20.0/24",
"next_hops": [
{
"ip_addr": "10.10.10.2"
},
{
"ip_addr": "11.11.11.2"
}
]
}
]
}
]
}
Floating static route configuration
"static": {
"vrf": [
{
"vrf_name": "default",
"static_routes": [
{
"ip_net": "20.20.20.0/24",
"next_hops": [
{
"ip_addr": "10.10.10.2"
}
],
"distance": 120
}
]
}
]
}
Operational State Overview
Check Static is enabled
Check static is listed in the enabled_protocols field.
flock@flocknet$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["Static"]
flock@flocknet:~$
Show Static Overview
Shows the configured number of IPv4 and IPv6 static routes. It also shows the number of interfaces that the static component is aware of, these include both the system interfaces that are currently enabled as well as the interfaces that are not enabled in the system but are referred to by their names in the static route configuration.
flock@r01:~$ flockc static
{"ipv4":{"route_count":4},"ipv6":{"route_count":1},"intf_count":6}
Show all interfaces
Static tracks interfaces in the system. These include:
- the interfaces that are enabled and have been assigned an interface ID by the kernel
- the interfaces that are referred to by their names in the static routes but have not been assigned an interface ID by the kernel
Static shows the current interface state as follows:
- the interface
name - the interface
idassigned by the kernel, if one exists - number of
attached_routesreferencing the interface
In the following example, ens4 interface has not been assigned an ID by the kernel
flock@r61:~$ flockc static -i
{"name":"ens1","id":2,"attached_routes":3}
{"name":"ens2","id":3,"attached_routes":2}
{"name":"ens3","id":4,"attached_routes":0}
{"name":"ens4","attached_routes":1}
{"name":"lo","id":1,"attached_routes":0}
Show Static prefix's
Note that this is not the System RIB. The static routes show all the routes from the configuration. For an attached route, it is only programmed in the RIB once its interface has been assigned an ID by the kernel.
Show all prefix's.
flock@r01:~$ flockc static -p --af ipv4
{"ip_net":"20.20.20.0/24"}
{"via":"10.10.10.2"}
{"ip_net":"30.30.30.0/24"}
{"intf":"ens3"}
{"ip_net":"40.40.40.0/24","admin_distance":100}
{"via":"10.10.10.2"}
{"via":"11.11.11.2"}
Show a specific prefix.
flock@r61:~$ flockc static -p 40.40.40.0/24 --af ipv4
{"ip_net":"40.40.40.0/24","nhs":[{"via":"10.10.10.2"},{"via":"20.20.20.2"}],"admin_distance":100}
ipv6 is the default AF type if the af option is not given
Static Operation Commands Reference
Help
flockc static -h
Overview
flockc static
All static interfaces
flockc static -i
Single interface
flockc static -i <interface-name>
Prefixes
flockc static -p [<ip-network>] [--af {ipv4 | ipv6}]
ipv6 is the default AF type if af option is not specified
Linux Kernel Settings
Linux kernel settings can be read using sysctl and written using sysctl -w. They can be made permanent / configured on boot, by adding an entry to /etc/sysctl.conf.
IP forwarding
For a Linux host to operate as an IP Router, IP forwarding must be enabled.
/etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
Bind IPv6 Only
By default a Linux host will operate as an IPv4/IPv6 dual stack node. See RFC 4038: 4.2. IPv6 Applications in a Dual-Stack Node.
This means that when flockd binds to an IPv6 socket, IPv4 requests will also be serviced. To restrict IPv6 sockets to only service IPv6 requests, the IPV6_V6ONLY socket option needs to be set.
/etc/sysctl.conf
net.ipv6.bindv6only = 1
BGP / TCP Termination at scale
Some protocols (most notably BGPv4) and the Operation API rely on a TCP transport. The Linux kernel has two parameters to control how many TCP connections can simultaneously be formed.
tcp_max_syn_backlog: Max TCP connections waiting for final ACK (of the TCP three way handshake)
flock@flocknet:/proc/sys/net/ipv4$ cat tcp_max_syn_backlog
256
somaxconn: Max TCP connections with completed TCP three way handshake waiting for accept() to be called.
flock@flocknet:/proc/sys/net$ sudo cat core/somaxconn
128
If these limits are exceeded the Linux kernel decides it is under a SYN DoS attack and will prevent further connections. Under these conditions this message is logged in /var/log/messages
"TCP: request_sock_TCP: Possible SYN flooding on port 179. Sending cookies."
In a production network it is very unlikely these limits will be reached (unless the router is under a SYN DoS attack). Even with 1000's of BGP neighbors it is unlikely that there will be greater than 128 TCP connections waiting to be accepted. However in the lab using a traffic generator this limit can be hit.
By default the Flock Networks Routing Suite is configured to be able to handle up to 1024 simultaneous BGPv4 TCP connections. To reach this scale the Linux kernel defaults need to be updated to match.
/etc/sysctl.conf
net.ipv4.tcp_max_syn_backlog=1024
net.core.somaxconn=1024
Environment Variables
Some Flock routing suite configuration can be set using environment variables.
flockd environment variables can be set in the systemd service file.
flockc environment variables can be set on the command line
RUST_LOG=debug flockc system
FLOCKD_CONF_DIR
FLOCKD_CONF_DIR sets the directory that flockd will look in for its configuration. If not set /etc/flockd will be used.
FLOCKD_CONF_FILE
FLOCKD_CONF_FILE sets the configuration filename that flockd will look for. If not set flockd.json will be used.
RUST_LOG
The default log level is info. At this level all info and higher priority levels will be logged. Supported log levels in descending priority order are error, warn, info, debug and trace.
Error
Log level error / [ERROR] is used for unexpected events signalled from inside the router. These are never expected to be seen and indicate a bug. Please email a bug report to: support@flocknetworks.com.
Warning
Log level warn / [WARN] is used for unexpected events signalled from outside the router. It is normal to see warnings whilst the network is converging. Warnings should never be seen after the network has converged and remains stable.
[WARN flockd::ospf_neigh] RouterId(10.0.100.2), V4(10.0.3.157) neigh state change Full -> Down
Information
Log level info / [INFO] is used for expected events of note
[INFO flockd] START: PID 385 Compile Mode Release Log Level "debug"
[INFO flockd::sys::sys_intf] Update IntfId(2)] Broadcast Mtu(1500) Up [] event DownToUp
[INFO flockd::ospf_intf] IntfId(2), 10.0.1.168/24 state change Wait -> DrOther
[INFO flockd::ospf_neigh] RouterId(10.0.100.3), V4(10.0.1.152) neigh state change Loading -> Full
Debug
Log level debug / [DEBUG] is used for common expected events.
Trace
Log level trace / [TRACE] is used for very common expected events.
Changing the flockd default log level
The default log level can be changed by setting the RUST_LOG environment variable in the flockd systemd service file.
grep RUST_LOG /lib/systemd/system/flockd.service
Environment=RUST_LOG="info"
When the systemd service file has changed, systemd needs to be told to reload the new flockd configuration.
# systemctl daemon-reload
To enable the new log level, flockd needs to be restarted.
# systemctl restart flockd
Different log levels can be set for different components. This example sets the default log level for all components except the BGP component to info. The BGP component will log at the debug level.
RUST_LOG="info,bgp=debug"
Monitoring via the REST API
flockd supports a REST API to allow monitoring of the Router. The REST API is accessed over HTTP and delivers a JSON payload. The REST API is Read-Only so a connecting client can only query state in the router, never change it. In HTTP terms, by design, only the GET method is supported. The flockc client uses the REST API.
Local Connection within a Router
On router R01 and we use the client to retrieve local state within R01. The JSON is delivered using the REST API via R01's IP loopback address.
flock@R01:~$ flockc rib --af ipv4 --prefix
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.246"}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
Remote Connection from one Router to another
From R01 we can view information on another router using the REST API. The JSON is delivered via HTTP. The command is the same, except we append the host name / IP Address of the target router.
flock@R01:~$ flockc rib --af ipv4 --prefix --host R02
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}
{"ip_net":"10.0.1.0/24","origin":"Kernel","next_hops":[{"intf_id":2}]}
{"ip_net":"10.0.2.0/24","origin":"Kernel","next_hops":[{"intf_id":3}]}
{"ip_net":"10.0.3.0/24","origin":"Kernel","next_hops":[{"intf_id":4}]}
{"ip_net":"10.0.4.0/24","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}
Remote Connection from Host to any Router
Now moving to a host H01. We can install only flockc, and we can monitor all routers, without running flockd. NB: This is the flockc package being installed, which only includes the client, not the flockd package which includes the client and the daemon.
flock@H01# dpkg -i flockc_20.4.x_amd64.deb
flock@H01:~$ flockc rib --af ipv4 --prefix --host R01
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
...
flock@H01:~$ flockc rib --af ipv4 --prefix --host R02
{"ip_net":"0.0.0.0/0","origin":"Ospfv2","next_hops":[{"intf_id":2,"ip_addr":"10.0.1.168"}]}
...
3rd Party Clients
Since the REST API is simply a JSON payload inside HTTP, there are many ways to connect. The Host Operating System can be your choice of Linux, Windows, Mac, Redox, etc. The HTTP client application can be any one that the Operating System supports. For instance, the venerable curl will run on all the above Operating Systems.
To use curl you just need to know the URL to connect to. If you run flockc with the --show-url option, then the URL associated with that command will be displayed rather than connected to.
flock@H01:~$ flockc rib --af ipv4 --prefix --host R01 --show-url
http://R01:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json-lines
flock@H01:~$
you@your-host:~$ curl -s -X GET "http://R01:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json-lines"
{"ip_net":"0.0.0.0/0","origin":"Kernel","next_hops":[{"intf_id":5,"ip_addr":"192.168.122.1"}]}
...
You can then use the language of your choice to consume and manipulate the JSON information. When feeding into a program we want vanilla JSON (rather than the default of JSON with extra line breaks), so we include the --json option to discover the URL.
flock@H01:~$ flockc rib --af ipv4 --prefix --json --host R01 --show-url
http://R01:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json
flock@H01:~$
you@your-host:~$ curl -s -X GET "http://R01:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json" | python -m json.tool
[
{
"ip_net": "0.0.0.0/0",
"next_hops": [
{
"intf_id": 5,
"ip_addr": "192.168.122.1"
}
],
"origin": "Kernel"
},
...
Example: Connecting to the REST API using Python
flock@H01:~$ flockc rib --af ipv4 --prefix --host r02 --json --show-url
http://r02:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json
you@your-host:~$ python3
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('http://r02:8000/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json')
>>> r.json()
[{'ip_net': '0.0.0.0/0', 'origin': 'Ospfv2', 'next_hops': [{'intf_id': 2, 'ip_addr': '10.0.1.168'}, {'intf_id': 4, 'ip_addr': '10.0.3.176'}]}, {'ip_net': '10.0.1.0/24', 'origin': 'Kernel', 'next_hops': [{'intf_id': 2}]}, {'ip_net': '192.168.122.0/24', 'origin': 'Ospfv2', 'next_hops': [{'intf_id': 2, 'ip_addr': '10.0.1.168'}, {'intf_id': 4, 'ip_addr': '10.0.3.176'}]}]
Manually Building REST API URL's
The available REST API URL's can be discovered using the flockc --show-url option. They can also be built by following these rules.
The URL's mirror the flockc long options, for example:
flockc rib --af ipv4 --prefix maps to the URL /rib/af?af_type=ipv4/prefix
Sorting
Append /sort if you want the output sorted.
/rib/af?af_type=ipv4/prefix/sort
Specify the output JSON format
A json format must be specified at the end of the URL.
Append /json if you want vanilla JSON (no extra line breaks).
/rib/af?af_type=ipv4/prefix/sort/json
Append /json-pretty if you want JSON pretty printed.
/rib/af?af_type=ipv4/prefix/sort/json-pretty
Append /json-lines if you want JSON Lines.
/rib/af?af_type=ipv4/prefix/sort/json-lines
Wildcards
In a URL query string a * is treated as wild. prefix?ip_net=* will get all IPv4 Networks.
/rib/af?af_type=ipv4/prefix?ip_net=*/sort/json
Escaping the / character
To escape the / character in a URL use %2F (0x2F is the ASCII value for /). So to specify an IPv4 network use ip_net=10.20.30.0%2F16. If you do not escape the / character this URL will not be found ip_net=10.20.30.0/16. It specifies a URL path ending in /16 which is not what we want.
/rib/af?af_type=ipv4/prefix?ip_net=10.20.30.0%2F24/sort/json
Commonly used REST API URL's
# BGP Overview
/bgp/json
# BGP Autonomous System
/bgp/as?asn=*/json
/bgp/as?asn=65073/json
# BGP Prefixes
/bgp/prefix?ip_net=*/json
/bgp/prefix?ip_net=4.3.0.0%2F16/json
# BGP Neighbors
/bgp/as?asn=65056/neigh?ip_addr=10.20.30.42/json
/bgp/as?asn=65056/neigh?ip_addr=*/json
# OSPFv2 Overview
/ospfv2/json
# OSPFv2 Autonomous System Link State Database (LSA Type 5)
/ospfv2/lsdb/json
# OSPFv2 Area Overview
/ospfv2/area?area_id=0.0.0.0/json
# OSPFv2 Area Link State Database (LSA Type 1 -> Type 4)
/ospfv2/area?area_id=0.0.0.0/lsdb/json
# OSPFv2 Interfaces
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=*/json
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=eth0/json
# OSPFv2 Neighbors
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=*/neigh?id=5.6.7.8/sort/json
/ospfv2/area?area_id=0.0.0.0/intf?intf_name=eth0/neigh?id=5.6.7.8/sort/json
# OSPFv2 Network Prefixes
/ospfv2/prefix-network?ip_net=*/json
/ospfv2/prefix-network?ip_net=20.30.0.0%2F16/json
# RIBv4 Overview
/rib/af?af_type=ipv4/json
# RIBv4 Longest path match lookup
/rib/af?af_type=ipv4/lpm?ip_addr=1.2.3.4/json
# RIBv4 Prefixes
/rib/af?af_type=ipv4/prefix?ip_net=*/json
/rib/af?af_type=ipv4/prefix?ip_net=10.20.30.0%2F24/json
/rib/af?af_type=ipv4/prefix?origin=ospfv2/json
# RIBv6 Overview
/ribv6/json
# RIBv6 Longest path match lookup
/ribv6/lpm?ip_addr=fc00::01/json
# RIBv6 Prefixes
/ribv6/prefix?ip_net=*/json
/ribv6/prefix?ip_net=fc00::01%2F64/json
# System Overview
/system/json
# System Interfaces
/system/intf?intf_name=*/json
/system/intf?intf_name=eth0/json
RFC Compliance
Flock Networks Routing Suite implements the following RFC's.
BGPv4
- RFC 1997 BGP Communities Attribute
- RFC 2545 BGP-4 Multiprotocol Extensions for IPv6 IDR
- RFC 4271 A Border Gateway Protocol 4 (BGP-4)
- RFC 4360 BGP Extended Communities Attribute
- RFC 4456 BGP Route Reflection
- RFC 4760 Multiprotocol Extensions for BGP-4
- RFC 5492 Capabilities Advertisement with BGP-4
- RFC 5668 4-Octet AS Specific Extended Community
- RFC 6793 BGP Support for Four-Octet Autonomous System (AS) Number Space
- RFC 7300 Reservation of Last Autonomous System (AS) Numbers
- RFC 7947 Internet Exchange BGP Route Server
- RFC 8092 BGP Large Communities Attribute
- RFC 8654 Extended Message Support for BGP
OSPFv2
- RFC 2328 OSPF Version 2
Inter-operating with Cisco OSFPv2
Cisco routers enable RFC 5613 "Link Local Signaling" (LLS) by default. Flock Networks OSPFv2 does not yet support RFC5613. To inter-operate LLS needs to be disabled on the Cisco device.
router ospf <n>
no capability lls
If you are hitting this problem, the Flock Networks router will be logging a warning.
[WARN ospf::ospf_intf] Rx OSPF Header sanity failed: "Rx Pak Hello ignore, Options mismatch, expected [E] got [E | L] from RouterId(192.168.0.4)
Manual Install
Flock Networks Routing Suite can be manually installed on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc).
Extract the contents of the Debian package
You will need the ar archive utility from binutils.
For example on Fedora;
# yum install binutils
Then extract the contents of the debian package
$ ar x flockd_20.4.x_amd64.deb
$ ls
control.tar.gz data.tar.xz debian-binary flockd_20.4.x_amd64.deb
$ tar -xf data.tar.xz
$ ls
control.tar.gz debian-binary flockd_20.4.x_amd64.deb usr data.tar.xz etc lib
Install Files
The copyright / software licence file is here;
less usr/share/doc/flockd/copyright
Copy the flockd daemon config files to /etc/flockd
# mkdir /etc/flockd
# cp etc/flockd/* /etc/flockd
Copy the flockd daemon to /usr/sbin
# cp usr/sbin/flockd /usr/sbin
Copy the flockc client to /usr/bin
# cp usr/bin/flockc /usr/bin
Start the daemon
# RUST_LOG=info /usr/sbin/flockd &
Check the daemon is running
$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "20.4.0"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
$
SONiC Support
By default, Flock uses the Linux kernal as control plane. However, the Flock Networks Routing Suite can act as the IP Routing control plane for the SONiC Network Operating System. SONiC uses the forwarding plane manager (FPM) to program fpmsyncd. fpmsyncd then sends the routing updates into the SONiC APPL_DB database.
To program fpmsyncd with IP routes the rib object in the /etc/flockd/flockd.json configuration file requires the fpm object to be defined.
{
"rib": {
"dataplane": {
"fpm": {
"tcp_port": 2620
}
}
}
}
Then check FPM is enabled by using the REST API.
$ flockc rib -d -j | grep fpm_state
"fpm_state": "enabled"
$
NB: The default IP Routing suite shipped with SONiC will need to be disabled, or it will conflict with the Flock Networks IP Routing Suite.
Debugging SONiC
An overview of the SONiC routing-state interactions is here.
Check the fpmsyncd logs for a successful connection from the Flock Networks IP Routing Suite.
# grep "fpmsyncd Connected" /var/log/messages
2020-08-25T08:44:49.707069+00:00 ad2e7e7845ad supervisord: fpmsyncd Connected!
Check the ASIC's have been programmed
The entries below match this SONiC P4 example.
host1 (Ubuntu, 192.168.1.2/24) <--> switch1 (SONiC) <--> switch2 (SONiC) <--> host2 (Ubuntu, 192.168.2.2/24)
Use the SONiC saidump utility to show the programmed ASIC entries. Below are shown the entries used to forward to the remote BGP subnet 192.168.2.0/24.
switch1$ saidump
# Remote BGP subnet
SAI_OBJECT_TYPE_ROUTE_ENTRY {"dest":"192.168.2.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000066"}
SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID : oid:0x400000000009c
# Which uses this nexthop
SAI_OBJECT_TYPE_NEXT_HOP oid:0x400000000009c
SAI_NEXT_HOP_ATTR_IP : 10.0.0.1
SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID : oid:0x6000000000095
SAI_NEXT_HOP_ATTR_TYPE : SAI_NEXT_HOP_TYPE_IP
# Which is out of this interface
SAI_OBJECT_TYPE_ROUTER_INTERFACE oid:0x6000000000095
SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS : 00:01:04:4C:49:F5
SAI_ROUTER_INTERFACE_ATTR_TYPE : SAI_ROUTER_INTERFACE_TYPE_VLAN
SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID : oid:0x3000000000066
SAI_ROUTER_INTERFACE_ATTR_VLAN_ID : oid:0x2600000000008f