Introduction

Welcome to the User Guide for Flock Networks® IP Routing Suite.

The Flock Networks IP Routing Suite will run on any version of Linux, including the SONiC NOS.

At Flock Networks we believe networks should be fast, secure and uniform.

Performance is achieved by:
- Running in a single operating system process, with each component having its own thread. This makes the communication between components dramatically faster.
- Using a thread pool in each component to achieve linear scale vs available CPU cores.
- Keeping the entire implementation lock free. This means each work flow can progress unhindered by any external interruptions.
- Handling network protocol updates in batches. This means route calculation can be performed less frequently, and keeps the CPU's instruction cache hot.
- Only using data structures that are CPU data cache friendly.
Security is achieved by the routing suite being exclusively written in the Rust programming language. Rust has a similar performance to C / C++, but is memory safe at compile time.
Uniformity is achieved by having a similar configuration across all routers in the network. Device specific and in particular interface specific configuration is to be avoided.

If you have any questions, suggestions or issues, please email support@flocknetworks.com.

Installation

The Flock Networks Routing Suite is shipped as a Debian package. However the Routing Suite can run on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc). The following instructions are for installing on a Debian based Linux distribution (Debian, Ubuntu, Mint, etc). To install on other systems, see Manual Install. The Flock Networks Routing Suite can also be used as a drop in replacement on the SONiC Network Operating System, see SONiC.

Download the latest Debian package.

Install the application from the Debian package (as root or using sudo);

# dpkg -i flockd_<version>_amd64.deb

The Flock Routing Suite Client flockc is copied to;

/usr/bin/flockc

The Flock Routing Suite Daemon flockd is copied to;

/usr/sbin/flockd

All Flock Routing Suite flockd configuration is stored in a single file;

/etc/flockd/flockd.json

flockd is controlled by systemd;

$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd

logging can be viewed via journalctl;

# journalctl -u flockd --boot

Example successful installation

# dpkg -i flockd_<version>_amd64.deb
Selecting previously unselected package flockd.
(Reading database ... 27695 files and directories currently installed.)
Preparing to unpack flockd_<version>_amd64.deb ...
Unpacking flockd ...
Setting up flockd ...

# systemctl start flockd

# systemctl status flockd
● flockd.service - Flock Networks Routing Suite Daemon
Loaded: loaded (/lib/systemd/system/flockd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-01-24 12:39:24 GMT; 6s ago
Main PID: 368 (flockd)
    Tasks: 1 (limit: 1150)
Memory: 3.2M
CGroup: /system.slice/flockd.service
        └─785 /usr/sbin/flockd
Nov 17 15:39:15 debian10-clean systemd[1]: Started Flock Networks Routing Suite Daemon.
Nov 17 15:39:15 debian10-clean flockd[368]: [INFO  flockd::sys::sys_inst] System Up: version=<version> model="Multi-threaded" pid=3
Nov 17 15:39:15 debian10-clean flockd[368]: [INFO  flockd::sys::sys_inst] REST API TcpListener { addr: 0.0.0.0:8000, fd: 4 }

$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "<version>"
"model": "Large"
"base_os": "Linux"
"pid": 368
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 14 }
"enabled_protocols": []

Troubleshooting Installation

`flockd` fails to start

systemd PolicyKit / polkit failure

PolicyKit / polkit is the "sudo of systemd". If you have PolicyKit / polkit active on the install host, and you want to start flockd as an unprivileged user, you will need to add PolicyKit / polkit configuration for flockd. Alternatively you can just run systemctl start flockd as root or under sudo.

$ systemctl start flockd
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'flockd.service'.
Authenticating as: www-data
Password:
Failed to start flockd.service: Connection timed out
See system logs and 'systemctl status flockd.service' for details.
polkit-agent-helper-1: pam_authenticate failed: Authentication failure

Flock Routing Suite Daemon `flockd`

Overview

The flockd process is controlled by systemd.

$ systemctl status flockd
# systemctl start flockd
# systemctl stop flockd

flockd is a single process that is split into separate components.

System Component

The system component manages the other components and presents a unified API to the outside world.

The system component monitors external signals and updates the other components as required. Example external signals would be the addition of an IP address to an interface, or the addition of a route to the Linux Kernel.

The system component provides the configuration and operations API.

Routing Information Base (RIB) Component

The RIB component receives route updates from other components and from the Linux Kernel. The RIB component selects the best update for each route and redistributes it to other components and to the Linux Kernel. This redistribution can be controlled by configuration. By default the RIB will send the best route to the Linux Kernel and will accept any routes from the Linux Kernel that are not sourced by flockd.

The RIB component also includes the Label RIB (LRIB) which manages MPLS label bindings programmed by protocols such as BGP and OSPF Segment Routing.

OSPF Components

The OSPF components run the OSPFv2 and OSPFv3 protocols. These components get interface state information and connected IP subnet information from the System Component. These components get external state information by connecting to OSPF neighbors via Linux Kernel raw IP sockets. This information is combined and each best route is derived and sent to the RIB component.

BGPv4 Component

The BGPv4 component runs the BGPv4 protocol. This component gets IP route information from the RIB component and external state information by connecting to BGP neighbors via Linux Kernel TCP sockets. This information is combined and each best route is derived and sent to the RIB component.

BFD Component

The BFD (Bidirectional Forwarding Detection) component provides sub-second failure detection for routing protocols. BFD sessions are created by protocols (BGP, OSPF, static routes) rather than configured directly. When a BFD session detects that a neighbor is unreachable, the associated protocol is notified immediately, enabling faster convergence than relying on protocol-level hello timers alone.

Static Component

The Static component manages the configuration of IPv4/IPv6 static routes. Static routes have a default administrative distance of 1 and therefore are preferred over dynamic routes by default. However, Static component allows you to specify a non-default administrative distance, thereby allowing a static route to be used as a backup route to a dynamic route. The configured static routes can be either recursive or non-recursive by specifying an explicit outgoing interface.

Multithreading

The routing suite runs in a single process and each component runs in its own thread. One way to view the active threads is via the Linux /proc virtual file system. The System, RIBv4, RIBv6, OSPFv2 and BGPv4 components are allocated a thread each. If a component is not enabled, it will not have any threads allocated.

The BGPv4 main thread travels with a BGP thread pool. The BGP thread pool will have a thread for each logical CPU core on the router.

flock@flocknet:~$ grep Name /proc/`pidof flockd`/task/*/status
/proc/409/task/409/status:Name: flockd
/proc/409/task/432/status:Name: OSPFv2 Main
/proc/409/task/433/status:Name: BGPv4
/proc/409/task/434/status:Name: RIBv4 Main
/proc/409/task/435/status:Name: RIBv6 Main
/proc/409/task/440/status:Name: BGPv4
/proc/409/task/441/status:Name: BGPv4
/proc/409/task/442/status:Name: BGPv4
/proc/409/task/443/status:Name: BGPv4
flock@flocknet:~$

In the example above:

The entire routing suite is running in a single process (process id 409).
The thread with thread id 409 is the flockd system thread.
The thread with thread id 433 is the BGP Main thread. The threads with id's 440, 441, 442 and 443 belong to the BGP thread pool. There are 4 threads in the BGP thread pool as the router has 4 logical CPU's.
```
  flock@flocknet:~$ cat /proc/cpuinfo | grep -c processor
  4
  flock@flocknet:~$
```

The current CPU and memory ulitization of each thread can be viewed using

  flock@flocknet:~$ top -H -p `pidof flockd`
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  525 root      20   0  280708   4460   4008 S   0.3   0.4   0:00.96 Static Main
  518 root      20   0  280708   4460   4008 S   0.0   0.4   0:01.38 flockd
  523 root      20   0  280708   4460   4008 S   0.0   0.4   0:02.06 BGPv4
  526 root      20   0  280708   4460   4008 S   0.0   0.4   0:01.29 RIB Main
  535 root      20   0  280708   4460   4008 S   0.0   0.4   0:00.96 BGPv4
  flock@flocknet:~$

Logging

Logging can be viewed using journalctl.

View flockd logs since router was booted.

# journalctl -u flockd --boot
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:38:44 BST. --
Jun 29 08:49:36 r61 flockd[455]: [INFO  flockd] START: PID=455, Compile Mode=Release, Log Level="info"
...

View flockd logs since a certain time.

# journalctl -u flockd --since "2020-06-26 17:19:20"
-- Logs begin at Fri 2020-01-10 14:01:47 GMT, end at Mon 2020-06-29 09:40:55 BST. --
Jun 26 17:19:20 r61 flockd[431]: [INFO  flockd::bgp::bgp_neigh] 90.0.93.70/Some(BgpId(70.0.100.70)) BgpAsn2(70) Outgoing FSM state change OpenConfirm -> Established
...

Monitor for the latest flockd logs.

# journalctl -u flockd --follow
-- Logs begin at Fri 2020-01-10 14:01:47 GMT. --
Jun 29 09:44:34 r61 flockd[455]: [INFO  flockd::bgp::bgp_neigh] 60.0.60.60/Some(BgpId(60.0.100.60)) BgpAsn2(60) Incoming FSM state change OpenConfirm -> Established
...

Log Levels

Log levels can be set using the RUST_LOG environment variable.

Flock Routing Suite Client `flockc`

flockc is the client provided to query the current state of flockd. By design flockc can be run by a user with no special privileges. flockc connects to a Read-Only connection on flockd. flockd state can be viewed but cannot be changed. All output is in JSON format, making it suitable for consumption by humans or machines.

flockc connects to a REST API on flockd. This REST API will talk to any HTTP based client. This means any HTTP based client can be used to monitor flockd.

See here for more information on using the REST API.

Local Connectivity

flockc uses the REST API via the local IP Loopback address to retrieve state information from the local flockd.

This command shows all the IPv4 Prefixes in the RIB of the local flocknet1 router.

flock@flocknet1:~$ flockc rib -v default --af ipv4 -p
[["50.0.0.0/8",{"protos":[{"admin_dist":20,"origin":"Bgp", ...,"ip_nhs":[{"RecursiveNh":{"table_id":{"vrf_id":254,"afi":"ipv4-unicast"},"ip_addr":"90.0.93.61", ...}}], ...}],"fib_ip_nhs":[...], ...}], ...]

General Command Flags

General Flags can be included in any flockc command.

-d, --detail
-h, --help
-j, --json-pretty    Output in Pretty Print JSON
-y, --yaml           Output in YAML

Changing the output format

By default the output is in JSON Lines format, which is JSON with newlines added to aid reading by a human. The output can also be changed to JSON Pretty Print or YAML.

JSON Lines format

flock@r70:~$ flockc ospfv2 -n
{"r70-n75":[{"ip_addr":"70.0.75.71","router_id":"70.0.100.71", ...,"state":"Full", ...}], ...}

JSON Pretty format

Using the -j, --json-pretty flag the output is pretty printed JSON.

flock@r70:~$ flockc bgp -j
{
  "overview": {
    "id": "70.0.100.70",
    "asn": 70,
    "cluster_id": null,
    "route_server": false,
    ...
  },
  ...
}

YAML format

Using the -y, --yaml flag the output is in YAML format.

flock@r70:~$ flockc bgp -y

System Component

Configuration Overview

The system configuration is held under the top level system object in /etc/flockd/flockd.json. The system object must exist and it must contain an api object. The api object can be left empty.

flock@flocknet:~$ cat /etc/flockd/flockd.json
{
    "system": {
        "api": {}
    }
}
flock@flocknet:~$

Configuration in detail

Operations REST API Configuration

The Operations REST API is used to view flockd internal state. It is a 'read only' API. By design state can be viewed but cannot be changed using this API.

The Operations REST API is configured using the rest_api JSON object. The rest_api object is optional. If the object is not specified the Operations REST API will not be started.

This configuration will bind to all local IPv4 and IPv6 addresses on the default port (3000).

"system": {
    "api": {
        "rest": {
            "bind_ip_addr": "::"
        }
    }
},

This configuration will bind to all local IPv4 and IPv6 addresses on the port (4000).

"system": {
    "api": {
        "rest": {
            "bind_ip_addr": "::",
            "bind_port": 4000
        }
    }
},

The bind_port field is optional and defaults to 3000. When running multiple flockd instances on the same host, each instance should use a different port. The flockc CLI can connect to a non-default port using the --port flag:

flockc --port 4000 system

This configuration will bind to only the loopback IPv6 address. The Operations REST API will not be available from outside of the router.

"system": {
    "api": {
        "rest": {
            "bind_ip_addr": "::1"
        }
    }
},

By default a Linux host will operate as an IPv4/IPv6 dual stack node, meaning IPv4 requests will be serviced. If you only want to service IPv6 requests please see Bind IPv6 Only

This configuration will bind to all local IPv4 addresses.

"system": {
    "api": {
        "rest": {
            "bind_ip_addr": "0.0.0.0"
        }
    }
},

This configuration will bind to only the loopback IPv4 address. The Operations REST API will not be available from outside of the router.

"system": {
    "api": {
        "rest": {
            "bind_ip_addr": "127.0.0.1"
        }
    }
},

Filtering interfaces in system component

System component receives interfaces from the Linux kernel. Interfaces not used by any routing component can be filtered out at the system component level by using intf_denylist JSON object.

This configuration will ignore interface eth0 and all interfaces starting with en:

"system": {
    ...
    "intf_denylist": [
        # Specify the entire interface name i.e. "eth0"
        # or use ^ to match the start of interface names
        #   e.g. "^en" will match all interfaces that start with "en"
        # Multiple interface names and patterns can be used
        "eth0",
        "^en"
    ]
},

The intf_denylist object is optional. If it is not specified, all interfaces present in kernel will also be present in system component. The filtered out interfaces will not be present in the flockc system -i command output.

Operational State Overview

Check status of `flockd`

flock@r70:~$ flockc system
{"host_info":{"hostname":"r70", ...},"system_info":{"name":"flockd","description":"The Flock Networks Ltd Routing Suite Daemon","profile":"Release", ...},"pid":1234,"log_level":"info","uptime":"days: 0, hours: 0, mins: 0, secs: 19","enabled_protocols":["BGPv4","OSPFv2","Static"],"software_errors":0, ...}

Show all system interfaces

flock@r70:~$ flockc system -i
[{"base":{"intf_name":"lo","intf_id":1, ...,"intf_type":"loopback","intf_mtu":65536,"intf_state":"up", ...},"ip_prefixes":["127.0.0.1/8","70.0.100.70/32","::1/128"]},{"base":{"intf_name":"r70-n70","intf_id":125675, ...,"intf_type":"broadcast","intf_mtu":1500,"intf_state":"up", ...},"ip_prefixes":["70.0.70.70/24","fc00:46:46::46/64", ...]}, ...]

Show single system interface

flock@r70:~$ flockc system -i r70-n70
[{"base":{"intf_name":"r70-n70","intf_id":125675, ...,"intf_state":"up", ...},"ip_prefixes":["70.0.70.70/24", ...]}]

Show all vrfs

flock@r70:~$ flockc system -v
[{"vrf_id":254,"vrf_name":"default","underlay_vrf_name":null,"kernel_intf_name":null, ...}]

Show single vrf

flock@r70:~$ flockc system -v default
[{"vrf_id":254,"vrf_name":"default","underlay_vrf_name":null,"kernel_intf_name":null, ...}]

System Operation

Help

flockc system -h

Overview

flockc system

All system interfaces

flockc system -i

Single interface

flockc system -i <interface-name>

All vrfs

flockc system -v

Single vrf

flockc system -v <vrf-name>

RIB Component

The RIB component receives routes from each routing protocol and stores them in the RIB. For each network prefix the 'best' route is sent to the dataplane for use when forwarding packets. By default, flock uses the Linux kernel as dataplane, but it can be configured to use other dataplanes.

Operational State

Overview

flock@r70:~$ flockc rib --af ipv4
{"total":15,"total_v4":12,"total_v6":3}

flock@r70:~$ flockc rib --af ipv6
{"total":15,"total_v4":12,"total_v6":3}

In all RIB commands IPv6 is the default address family

flock@r70:~$ flockc rib
{"total":15,"total_v4":12,"total_v6":3}

Longest Prefix Match (LPM) for destination

fib_ip_nhs are the resolved next hops taken from the 'best' protocol

  flock@r70:~$ flockc rib -v default --af ipv4 -l 70.0.70.1
  {"prefix":"70.0.70.0/24","route":{"fib_ip_nhs":[{"ConnectedNh":{"intf_id":125675, ...,"table_id":{"afi":"ipv4-unicast","vrf_id":254}}}], ...,"protos":[{"admin_dist":0, ...,"origin":"Connected", ...,"ip_nhs":[{"ConnectedNh":{...}}], ...}]}}

Single RIB prefix entry

flock@r70:~$ flockc rib -v default --af ipv4 -p 70.0.0.0/8 -j
{
  "protos": [
    {
      "admin_dist": 1,
      "origin": {
        "Static": "Conf"
      },
      ...
      "ip_nhs": [
        {
          "SpecialNh": {
            "table_id": {
              "vrf_id": 254,
              "afi": "ipv4-unicast"
            },
            "special_nh_type": "Discard",
            ...
          }
        }
      ],
      ...
    }
  ],
  "fib_ip_nhs": [
    {
      "SpecialNh": {
        ...
        "special_nh_type": "Discard",
        ...
      }
    }
  ],
  ...
}

All prefix entries in the RIB

Only the 'best' origin protocol is shown.

  flock@r70:~$ flockc rib -v default --af ipv4 -p
  [["50.0.0.0/8",{"protos":[{"admin_dist":20,"origin":"Bgp", ...,"ip_nhs":[{"RecursiveNh":{ ...,"ip_addr":"90.0.93.61", ...}}], ...}],"fib_ip_nhs":[{"AttachedNh":{ ...,"ip_addr":"90.0.93.61", ...}}], ...}],["70.0.0.0/8",{"protos":[{"admin_dist":1,"origin":{"Static":"Conf"}, ...}], ...}],["70.0.70.0/24",{"protos":[{"admin_dist":0,"origin":"Connected", ...}], ...}], ...]

RIB Operation

Help

flockc rib -h

RIB Overview

flockc rib --af [ipv4 | ipv6 ]

RIB Prefixes

flockc rib -v <vrf-name> --af [ ipv4 | ipv6 ] -p [<ip-network>]

Connected routes

flockc rib -v <vrf-name> --af [ ipv4 | ipv6 ] -c

Prefixes by route origin.

Origin can be bgp, connected, dhcpv4, dhcpv6, ipv6ra, ipsecctrl, ipsecike, kernel, ospfv2, ospfv3, or static.

flockc rib -v <vrf-name> --af [ipv4 | ipv6 ] -p -o ospfv2

Longest Prefix Match (LPM)

flockc rib -v <vrf-name> --af [ipv4 | ipv6 ] -l <ip-address>

Label RIB (LRIB)

The Label RIB manages MPLS label bindings programmed by protocols such as BGP and OSPF Segment Routing.

LRIB Overview

flockc lrib

LRIB label lookup

flockc lrib -l [<label>]

OSPFv2 Component

Enabling the OSPFv2 Component

The OSPFv2 configuration is held under the top level ospfv2 object in /etc/flockd/flockd.json. If the ospfv2 object exists OSPFv2 will be enabled and the OSPFv2 master thread will be started.

The Flock Networks Routing Suite is designed for massive scale so placing all routers in a single OSPF area is recommended. (If you are adding a device to an existing multi-area OSPF Autonomous System, multiple areas are fully supported).

With this configuration file:

The OSPFv2 master thread will be started
The OSPFv2 router will advertise a router id of 10.0.100.2

All interfaces with names starting with en will be placed in OSPF area 0.

  "ospfv2": {
      "vrfs": {
          "default": {
              "router_id": "10.0.100.2",
              "areas": {
                  "0.0.0.0": {
                      "intfs": {
                          "^en": {}
                      }
                  }
              }
          }
      }
  }

This is all the OSPFv2 configuration you need, to create an OSPF network as large as you like. Each device has an identical configuration which simplifies the operation of the network. A management station can easily determine all the Router Id's in the network by querying a single device for all of its Router LSA's.

Redistribution of routes into OSPF

You may wish to redistribute routes from the RIB into OSPFv2. Use the redistribute json object. The origin field specifies the protocol that programmed the routes into the RIB.

flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
    "vrfs": {
        "default": {
            "redistribute": [
                {
                    "metric": 200,
                    "metric_type": 2,
                    "origin": "static"
                }
            ]
        }
    }
}

As a minimum we may want a default route added to the kernel of each ASBR router. This route will appear in the RIB and then be redistributed into OSPFv2. OSPFv2 will advertise this route across the AS, so all nodes learn the route to exit the network. Static routes are added using the flockd static component.

flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"static": {
    "vrfs": {
        "default": {
            "routes": {
                "0.0.0.0/0": {
                    "next_hops": [
                        {
                            "ip_addr": "192.168.122.171",
                            "intf_name": "enp8s0"
                        }
                    ]
                }
            }
        }
    }
}

Stub Router

Configure the router to advertise maximum metric in its Router LSA, causing other routers to avoid using it as a transit node. This is useful during maintenance windows or router startup.

"ospfv2": {
    "vrfs": {
        "default": {
            "stub_router": {
                "mode": "MaxMetric",
                "always": true
            }
        }
    }
}

Default Route Origination

Advertise a default route into the OSPF domain.

"ospfv2": {
    "vrfs": {
        "default": {
            "default_originate": {
                "always": true
            }
        }
    }
}

When always is true, the default route is originated regardless of whether one exists in the RIB.

Administrative Distance

Override the default OSPF administrative distance (default: 110).

"ospfv2": {
    "vrfs": {
        "default": {
            "admin_distance": 115
        }
    }
}

LSA Arrival Timer

The timer_lsa_arrival sets the minimum interval (in milliseconds) between accepting the same LSA from a neighbor.

"ospfv2": {
    "vrfs": {
        "default": {
            "timer_lsa_arrival": 1000
        }
    }
}

Area Summary Ranges

Area summary ranges allow aggregation of prefixes at area boundaries.

"ospfv2": {
    "vrfs": {
        "default": {
            "areas": {
                "0.0.0.1": {
                    "summary": {
                        "10.0.0.0/8": {
                            "advertise": true
                        }
                    },
                    "intfs": {
                        "^eth": {}
                    }
                }
            }
        }
    }
}

Implicit Router Id

With no explicit configuration the highest IPv4 Address is used as the Router Id. IPv4 Addresses on loopback interfaces are always preferred over IPv4 Addresses on physical interfaces.

Explicit Router Id

To explicitly set the Router Id to 10.0.100.1.

flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
    "vrfs": {
        "default": {
            "router_id": "10.0.100.1",
            ...
        }
    }
}

Advertising the IPv4 Address that is being used as the Router Id

When operating a network it can be useful to have the Router Id's advertised as an IPv4 host route in OSPF. This means the Router Id will respond to network operation tools such as ping and traceroute. To do this create a loopback interface and assign the IPv4 host route to it. Then enable OSPF on the interface that is providing the Router Id IPv4 address.

Create a loopback interface and assign the IPv4 host route to it.

The method for permanently adding IP addresses to loopback interfaces is Linux distribution specific. For example Debian uses the /etc/network/interfaces file.

flock@flocknet:~$ cat /etc/network/interfaces
...
# The loopback network interface
auto lo
iface lo inet loopback

# Add IPv4 Address to be used as RouterId
auto lo:20
iface lo:20 inet static
address 70.0.100.71/32
...

flock@flocknet:~$ sudo systemctl restart networking
flock@flocknet:~$

Enable OSPF on the loopback interface

flock@flocknet:~$ cat /etc/flockd/flockd.json
...
"ospfv2": {
    "vrfs": {
        "default": {
            "areas": {
                "0.0.0.0": {
                    "intfs": {
                        "lo": {}
                    }
                }
            }
        }
    }
},

Check the Router Id is as expected

flock@r71:~$ flockc ospfv2 -j
...
"id": "70.0.100.71",
...

Operational State Overview

Check OSPFv2 is enabled

Check OSPFv2 is listed in the enabled_protocols field.

flock@r70:~$ flockc system
{"host_info":{"hostname":"r70", ...},"system_info":{"name":"flockd", ...},"pid":1234,"log_level":"info","uptime":"days: 0, hours: 0, mins: 0, secs: 19","enabled_protocols":["OSPFv2"],"software_errors":0, ...}

Show OSPFv2 Overview

flock@r70:~$ flockc ospfv2
[{"my_router":{"id":"70.0.100.70","class":""},"vrf_name":"default", ...,"stats":{"ls_db_counters":{...},"route_count":8,"neigh_state_count":{"down":0,"attempt":0,"init":0,"two_way":0,"ex_start":0,"exchange":0,"loading":0,"full":3}}}]

Show all neighbors (out of all interfaces, in all areas)

flock@r70:~$ flockc ospfv2 -n
{"r70-n75":[{"ip_addr":"70.0.75.71","router_id":"70.0.100.71", ...,"state":"Full","dr":...,"bdr":..., ...}],"r70-n70":[{"ip_addr":"70.0.70.73","router_id":"70.0.100.73", ...,"state":"Full", ...},{"ip_addr":"70.0.70.72","router_id":"70.0.100.72", ...,"state":"Full", ...}]}

Show Area 0 Link State Database

flock@r70:~$ flockc ospfv2 -a 0 -l
[{"V2":{"hdr":{"lsa_age":56,"lsa_opts":"E","lsa_type":"Router","lsa_id":"70.0.100.70","lsa_router_id":"70.0.100.70","lsa_seq":-2147483645,"lsa_checksum":14522,"lsa_len":60},"body":{"Router":{...}}}}, ...]

Annotated OSPFv2 Configuration

"ospfv2": {
    "vrfs": {
        "<vrf-name>": {
            # RFC2328 1.2 Router ID
            # Optional: If not specified highest IPv4 Address is used.
            "router_id": "String in dotted decimal format",
            # Array of 'redistribute' objects
            "redistribute": [
                {
                # Origin of the Routes in the RIB to be redistributed
                "origin": ["static" | "connected"],
                # RFC2328 2.3 Type 1 / Type 2 external metrics
                "metric_type": [ 1 | 2 ],
                # OSPF metric to reach redistributed routes, from this router.
                # RFC2328 B. LSInfinity => 16777215
                "metric": ( 0..16777215 )
                },
            ],
            # OSPFv2 Area level configuration
            # -------------------------------
            "areas": {
                # RFC2328 C.2 Area ID (dotted decimal)
                "<area-id>": {
                    # OSPFv2 Interface level configuration
                    # ------------------------------------
                    "intfs": {
                        # Key is the interface name
                        # Specify the entire interface name i.e. "eno1"
                        # or use ^ to match the start of interface names
                        #   e.g. "^en" will match all interfaces that start with "en"
                        "<interface-name>": {
                        # OSPF interface type
                        # Optional: Default is "broadcast"
                        "ospf_intf_type": ["broadcast" | "point-to-point" | "nbma" | "point-to-multipoint"],
                        # RFC2328 C.3 Interface output cost
                        # Optional: Default is 10
                        "cost": ( 1..65,535 ),
                        # RFC2328 C.3 Router Priority
                        # Optional: Default is 1
                        "priority": ( 0..255 ),
                        # RFC2328 C.3 HelloInterval
                        # Optional: Default is 10s
                        "hello_interval": ( 1..65,535 seconds),
                        # RFC2328 C.3 RouterDeadInterval
                        # Optional: Default is 40s
                        "dead_interval": ( 1..65,535 seconds),
                        # Retransmit interval in seconds
                        "rxmt_interval": <seconds>,
                        # Ignore MTU mismatch with neighbor
                        # Optional: Default is false
                        "mtu_ignore": [ true | false ],
                        # MD5 authentication
                        "auth": {
                            "auth_algo": "md5",
                            "auth_key_id": ( 0..255 ),
                            "auth_key": "<key-string>"
                        },
                        # BFD failure detection
                        "bfd": {
                            "multiplier": ( 1..255 ),
                            "min_tx": "<duration>"
                        }
                        }
                    }
                }
            }
        }
    }
}

Example Exhaustive OSPFv2 Configuration

"ospfv2": {
    "vrfs": {
        "default": {
            "router_id": "10.0.1.100",
            "redistribute": [
                {
                    "metric": 100,
                    "metric_type": 1,
                    "origin": "static"
                },
                {
                    "metric": 1000,
                    "metric_type": 2,
                    "origin": "connected"
                }
            ],
            "areas": {
                "0.0.0.0": {
                    "intfs": {
                        "enp0s0": {
                            "cost": 20,
                            "dead_interval": 4,
                            "hello_interval": 1,
                            "priority": 10
                        },
                        "^eth": {
                            "cost": 40,
                            "dead_interval": 8,
                            "hello_interval": 2,
                            "priority": 20
                        }
                    }
                },
                "0.0.0.1": {
                    "intfs": {
                        "enp1s0": {
                            "cost": 30,
                            "dead_interval": 80,
                            "hello_interval": 20,
                            "priority": 30
                        }
                    }
                }
            }
        }
    }
}

OSPFv2 Operation

Help

flockc ospfv2 -h

Overview

flockc ospfv2

Overview of areas

flockc ospfv2 -a [<area-id>]

All interfaces in Area 20

flockc ospfv2 -i -a 20

All neighbors on interface enp1s0 in Area 0

flockc ospfv2 -n -i enp1s0 -a 0

All neighbors on all interfaces in all areas

flockc ospfv2 -n

Autonomous System Link State Database

flockc ospfv2 -l

Area 0.0.0.0 Link State Database

flockc ospfv2 -a 0 -l

Network route table prefixes

flockc ospfv2 -p [<ipv4-network>]

Router route table prefixes

flockc ospfv2 -P [<router-id>]

LSDB filtered by LSA type

flockc ospfv2 -a <area-id> -l --lsa-type <type>

LSDB filtered by router ID

flockc ospfv2 -a <area-id> -l --lsa-router-id <router-id>

Redistributed RIB

flockc ospfv2 -r

Event buffer

flockc ospfv2 --events

OSPFv3 Component

Enabling the OSPFv3 Component

The OSPFv3 configuration is held under the top level ospfv3 object in /etc/flockd/flockd.json. If the ospfv3 object exists OSPFv3 will be enabled and the OSPFv3 master thread will be started.

OSPFv3 (RFC 5340) is the IPv6 version of OSPF. The configuration structure mirrors OSPFv2.

With this configuration file:

The OSPFv3 master thread will be started
The OSPFv3 router will advertise a router id of 10.0.100.1
All interfaces with names starting with r301 will be placed in OSPF area 0.

The loopback interface will also be placed in area 0.

  "ospfv3": {
      "vrfs": {
          "default": {
              "router_id": "10.0.100.1",
              "areas": {
                  "0.0.0.0": {
                      "intfs": {
                          "lo": {},
                          "^r301": {
                              "hello_interval": 1
                          }
                      }
                  }
              }
          }
      }
  }

Redistribution of routes into OSPFv3

You may wish to redistribute routes from the RIB into OSPFv3. Use the redistribute JSON array. The origin field specifies the protocol that programmed the routes into the RIB.

"ospfv3": {
    "vrfs": {
        "default": {
            "router_id": "10.0.100.1",
            "redistribute": [
                {
                    "origin": "static",
                    "metric_type": 2,
                    "metric": 1000
                }
            ],
            "areas": {
                "0.0.0.0": {
                    "intfs": {
                        "lo": {},
                        "^r301": {}
                    }
                }
            }
        }
    }
}

Implicit and Explicit Router Id

As with OSPFv2, the router id can be set explicitly via the router_id field, or left unset to use the highest IPv4 address.

Interface Configuration

OSPFv3 interfaces support the same configuration options as OSPFv2:

"ospfv3": {
    "vrfs": {
        "<vrf-name>": {
            "areas": {
                "<area-id>": {
                    "intfs": {
                        "<interface-name>": {
                            # Interface type (default: broadcast)
                            "ospf_intf_type": ["broadcast" | "point-to-point" | "nbma" | "point-to-multipoint"],
                            # Interface output cost (default: 10)
                            "cost": ( 1..65535 ),
                            # Router Priority (default: 1)
                            "priority": ( 0..255 ),
                            # HelloInterval in seconds (default: 10)
                            "hello_interval": ( 1..65535 ),
                            # RouterDeadInterval in seconds (default: 40)
                            "dead_interval": ( 1..65535 ),
                            # Retransmit interval in seconds
                            "rxmt_interval": <seconds>,
                            # Ignore MTU mismatch with neighbor
                            "mtu_ignore": [ true | false ],
                            # BFD failure detection
                            "bfd": {
                                "multiplier": 3,
                                "min_tx": "1s"
                            }
                        }
                    }
                }
            }
        }
    }
}

Area Configuration

Area summary ranges

Area summary ranges allow aggregation of prefixes at area boundaries. When a summary range is configured, individual prefixes within that range are suppressed and replaced with a single summary prefix.

"ospfv3": {
    "vrfs": {
        "default": {
            "areas": {
                "0.0.0.1": {
                    "summary": {
                        "fc00:1::/32": {
                            "advertise": true
                        }
                    },
                    "intfs": {
                        "^eth": {}
                    }
                }
            }
        }
    }
}

Global Configuration Options

Stub router

Configure the router to advertise maximum metric in its Router LSA, causing other routers to avoid using it as a transit node.

"ospfv3": {
    "vrfs": {
        "default": {
            "stub_router": {
                "mode": "MaxMetric",
                "always": true
            }
        }
    }
}

Default route origination

"ospfv3": {
    "vrfs": {
        "default": {
            "default_originate": {
                "always": true
            }
        }
    }
}

Administrative distance

"ospfv3": {
    "vrfs": {
        "default": {
            "admin_distance": 115
        }
    }
}

LSA arrival timer

The timer_lsa_arrival sets the minimum interval (in milliseconds) between accepting the same LSA from a neighbor. This helps dampen rapid LSA flooding.

"ospfv3": {
    "vrfs": {
        "default": {
            "timer_lsa_arrival": 1000
        }
    }
}

Segment Routing and Flexible Algorithm

OSPFv3 supports Segment Routing with Prefix SIDs and Flexible Algorithms (RFC 9350).

Prefix SID Map

Prefix SIDs assign MPLS labels to IPv6 prefixes, enabling Segment Routing forwarding.

"ospfv3": {
    "vrfs": {
        "default": {
            "router_id": "10.0.100.53",
            "prefix_sid_map": {
                "fc00::0161/128": {
                    "prefix_sid": 353,
                    "flex_algo_sids": {
                        "201": {
                            "sid": 90569
                        }
                    }
                }
            }
        }
    }
}

Flexible Algorithm

Flexible Algorithms allow defining custom constraint-based paths through the network.

"ospfv3": {
    "vrfs": {
        "default": {
            "flex_algos": {
                "201": {
                    "advertise_definition": true
                },
                "202": {
                    "advertise_definition": true
                }
            }
        }
    }
}

Operational State Overview

Check OSPFv3 is enabled

Check OSPFv3 is listed in the enabled_protocols field.

flock@r301:~$ flockc system
{...,"enabled_protocols":["OSPFv3"],...}

Show OSPFv3 Overview

flock@r301:~$ flockc ospfv3

Show all neighbors

flock@r301:~$ flockc ospfv3 -n

Show Area Link State Database

flock@r301:~$ flockc ospfv3 -a 0 -l

Filter LSDB by LSA type

flock@r301:~$ flockc ospfv3 -a 0 -l --lsa-type <type>

Filter LSDB by router ID

flock@r301:~$ flockc ospfv3 -a 0 -l --lsa-router-id <router-id>

Show network route table prefixes

flock@r301:~$ flockc ospfv3 -p [<ipv6-network>]

Show router route table prefixes

flock@r301:~$ flockc ospfv3 -P [<router-id>]

Show event buffer

flock@r301:~$ flockc ospfv3 --events

Show redistributed RIB

flock@r301:~$ flockc ospfv3 -r

Show Flexible Algorithm routes

flock@r301:~$ flockc ospfv3 --flex-algo <algo-id>

OSPFv3 Operation Commands Reference

Help

flockc ospfv3 -h

Overview

flockc ospfv3

Overview of areas

flockc ospfv3 -a [<area-id>]

All interfaces in an area

flockc ospfv3 -i -a <area-id>

All neighbors

flockc ospfv3 -n

Link State Database

flockc ospfv3 -l

Area Link State Database

flockc ospfv3 -a <area-id> -l

LSDB filtered by LSA type

flockc ospfv3 -a <area-id> -l --lsa-type <type>

LSDB filtered by router ID

flockc ospfv3 -a <area-id> -l --lsa-router-id <router-id>

Network route table prefixes

flockc ospfv3 -p [<ipv6-network>]

Router route table prefixes

flockc ospfv3 -P [<router-id>]

Redistributed RIB

flockc ospfv3 -r

Event buffer

flockc ospfv3 --events

Flexible Algorithm routes

flockc ospfv3 --flex-algo <algo-id>

BGP Component

Overview

Enabling the BGP Component

The BGP configuration is held under the top level bgp object in /etc/flockd/flockd.json. If the bgp object exists BGP will be enabled and the BGP master thread will be started.

With this configuration file:

The BGP master thread will be started.
The router is in an Autonomous System identified by the Autonomous System Number 65016

The router has a BGP router identifier assigned as 172.16.10.1

  "bgp": {
      "asn": 65016,
      "id": "172.16.10.1"
  }

Show the status of the BGP component

Check BGP is listed in the enabled_protocols field.

flock@r70:~$ flockc system
{"host_info":{"hostname":"r70", ...},"system_info":{"name":"flockd","description":"The Flock Networks Ltd Routing Suite Daemon", ...},"pid":1234,"log_level":"info","uptime":"days: 0, hours: 0, mins: 0, secs: 19","enabled_protocols":["BGPv4"],"software_errors":0, ...}

Show BGP Overview

flock@r70:~$ flockc bgp
{"overview":{"id":"70.0.100.70","asn":70,"cluster_id":null,"route_server":false,"route_reflector":false, ...,"routes":{"unicast_routes":{"default":{"ipv4_unicast":{"route_count":3, ...},"ipv6_unicast":{"route_count":0, ...}}},"vpn_routes":{}},"neighbor_summary":{"default":{"count":4,"established":4,"send_converged":4,"recv_converged":4}}}, ...}

send_converged means all updates have been sent to this neighbor. The neighbor send update queue is empty. The neighbor may not have received all the updates yet, they may still be in the local TCP send buffer (or the neighbors TCP receive buffer)

recv_converged means all available updates from this neighbor have been processed. The neighbor receive TCP buffer is empty. However the neighbor may not have managed to send all updates yet.

There are 3 IPv4 Unicast routes in the BGP RIB
There are 4 neighbors, all of which have reached established state
All 4 neighbors are send_converged and recv_converged

Originating Networks

With this configuration file:

The router originates the 172.16.0.0/16 and fc00:46::/32 networks

  "bgp": {
      "asn": 65016,
      "id": "172.16.10.1",
      "vrfs": {
          "default": {
              "networks": {
                  "172.16.0.0/16": {},
                  "fc00:46::/32": {}
              }
          }
      }
  }

By default, a network is only originated when a matching prefix exists in the RIB. To originate a network unconditionally, set originate_always to true:

    "networks": {
        "172.16.0.0/16": {
            "originate_always": true
        }
    }

    flock@r01:~$ flockc bgp -v default -p 172.16.0.0/16 --af ipv4
    {"best":[{"neigh_api_key":null, ...,"path_type":"ViaOriginate", ...,"reason":"SelfOriginated", ...}], ...}

If not specified the address family defaults to --af=ipv6

Address Families

The following address families can be configured per-neighbor via the afis object:

ipv4-unicast — IPv4 unicast routes
ipv6-unicast — IPv6 unicast routes
ipv4-mplsvpn — IPv4 MPLS VPN routes (L3VPN)
ipv6-mplsvpn — IPv6 MPLS VPN routes (L3VPN)
l2vpn-evpn — L2VPN EVPN routes
ipv4-mpls — IPv4 labeled unicast routes
ipv6-mpls — IPv6 labeled unicast routes

Configuring Neighbors

With this configuration file:

The router has a single iBGP neighbor 172.16.20.2
- The iBGP connection source is 172.16.20.1 (specified via the neighbor key "172.16.20.2 172.16.20.1")
- The iBGP connection will advertise IPv4 unicast routes
- Routes are advertised over iBGP with a next hop of 172.16.20.1 (next_hop_self)
- BGP add-path [RFC7911] is enabled for both send and receive

The router has a single eBGP neighbor 172.17.20.1 in remote AS 65017

  "bgp": {
      "asn": 65016,
      "id": "172.16.10.1",
      "vrfs": {
          "default": {
              "neighs": {
                  "172.16.20.2 172.16.20.1": {
                      "asn": 65016,
                      "next_hop_self": true,
                      "add_path": {
                          "receive": true,
                          "transmit": true
                      },
                      "afis": {
                          "ipv4-unicast": {}
                      }
                  },
                  "172.17.20.1": {
                      "asn": 65017,
                      "afis": {
                          "ipv4-unicast": {}
                      }
                  }
              }
          }
      }
  }

The neighs object is keyed by the neighbor IP address. When a local source IP is needed, the key takes the form "<remote_ip> <local_ip>".

Enable Dynamic BGP Neighbors

When the "Dynamic BGP Neighbor" feature is enabled, BGP neighbors are allowed to connect without any explicit configuration.

With this configuration file:

Only neighbors with a source address in the 192.168.0.0/16 will be accepted as dynamic neighbors

Connecting neighbors will be expected to be part of autonomous system number (ASN) 65073. If the BGP Open message from the neighbor is not from ASN 65073 the connection will be terminated.

  "bgp": {
      "asn": 65016,
      "id": "172.16.10.1",
      "vrfs": {
          "default": {
              "dynamic_neighs": {
                  "192.168.0.0/16": {
                      "asn": 65073,
                      "connect_mode": "Passive",
                      "afis": {
                          "ipv4-unicast": {}
                      }
                  }
              }
          }
      }
  }

The dynamic_neighs object is keyed by the listen range prefix. Each entry specifies the expected ASN, connect mode, and address families.

Show all neighbors

BGP runs two Finite State Machines (FSM's) per neighbor. One FSM handles the Outgoing TCP connection and the other handles the Incoming TCP connection. The Flock Routing Suite does not hide this from the operator. In the final working state each neighbor should have one FSM in the Established state, and one FSM in the Idle state.

The last error to cause a BGP Notify Message is held in each FSM's last_notify_send / last_notify_recv fields. These fields are never cleared, they are only overwritten with the last error. So a value of null means there have been no errors that have caused a notify message since flockd was started.

flock@r70:~$ flockc bgp -v default -n
[{"common":{"neigh_key":"70.0.100.72 70.0.100.70","hostname":"...","asn":70,"bgp_id":"70.0.100.72","neigh_type":"Internal","connect_mode":"Both", ...},"outgoing":{"state":"Idle","last_notify_send":"Cease","last_notify_recv":null, ...},"incoming":{"state":"Established","last_notify_send":null,"last_notify_recv":null, ...},"stats":{...}}, ...]

Show BGP RIB prefixes

Note that this is not the RIB held in the RIB component, this is the BGP RIB. The BGP RIB records routes from all neighbors and sends the 'best entry' route to the RIB component. By default, BGP will show the ipv6 routes if the af parameter is not specified.

Show all prefixes. Only the 'best entry' for each prefix is shown, along with the reason why it was the best.

flock@r70:~$ flockc bgp -v default --af ipv4 -p
[{"prefix":"50.0.0.0/8","entry":{"best":[{...,"attrs":{"origin":"Igp","as_path":{"segments":[{"segment_type":"AsSequence","segment_value":[70,60,60,60,50,50]}]},"next_hop":"90.0.93.61", ...},"reason":"OnlyValidPeer", ...}], ...}},{"prefix":"60.0.0.0/8","entry":{...}},{"prefix":"70.0.0.0/8","entry":{"best":[{...,"reason":"SelfOriginated", ...}], ...}}]

Show a specific prefix. The 'best entry' and all the candidate paths are shown.

flock@r70:~$ flockc bgp -v default -p 70.0.0.0/8 --af ipv4
{"best":[{...,"path_type":"ViaOriginate", ...,"reason":"SelfOriginated", ...}],"paths":[{...,"route_type":"Originated","attrs":{"origin":"Igp","as_path":{"segments":[]}, ...}, ...}], ...}

Configuring BGP Active / Passive Neighbors

By default BGP will try to create two TCP transport connections to each neighbor. One outgoing to the neighbors remote BGP TCP port 179, and one allowing incoming connections from the neighbor to the local BGP TCP port 179. A tie break is used to ensure only one connection remains when the BGP neighbor moves to the 'Established' state.

The router can be configured to form a single TCP transport connection to each neighbor using the connect_mode neighbor configuration parameter.

"bgp": {
    "vrfs": {
        "default": {
            "neighs": {
                "172.17.20.1": {
                    "asn": 65017,
                    # Only create the outgoing connection to this neighbor.
                    # Refuse any incoming connection.
                    "connect_mode": "Active",
                    "afis": {
                        "ipv4-unicast": {}
                    }
                }
            }
        }
    }
}

            # Only allow the incoming connection from this neighbor.
            # Do not create any outgoing connection.
            "connect_mode": "Passive"

Configuring BGP Route Reflectors

To configure a router as a BGP Route Reflector, specify which neighbors are Route Reflector clients using the route_reflector_client configuration boolean.

"bgp": {
    "vrfs": {
        "default": {
            "neighs": {
                "172.16.20.2 172.16.20.1": {
                    "asn": 65016,
                    # Reflect iBGP routes to and from this neighbor
                    "route_reflector_client": true,
                    "afis": {
                        "ipv4-unicast": {}
                    }
                }
            }
        }
    }
}

To deploy redundant Route Reflectors a Route Reflector Cluster Id can optionally be configured.

"bgp": {
    "cluster_id": "1.2.3.4"
}

Configuring BGP to act as a Route Server

BGP Route Server functionality is defined in RFC7947. To configure a router as a BGP Route Server use the route_server configuration boolean.

"bgp": {
    "asn": 65056,
    "id": "192.168.0.14",
    "route_server": true,
    "vrfs": {}
}

To check BGP is running as a route server.

flock@r01:~$ flockc bgp -j
  ...
  "route_server": true,
  ...

Configuring Multihop BGP

Multihop BGP is configured by changing the Time to Live (TTL) of the BGP packets that are sent.

The default BGP packet TTL's are iBGP = 64 and eBGP = 1.

Use the neighbor ttl configuration keyword to override the defaults.

"bgp": {
    "vrfs": {
        "default": {
            "neighs": {
                "60.0.20.61": {
                    "asn": 60,
                    "ttl": {
                        "send": 2
                    },
                    "afis": {
                        "ipv4-unicast": {}
                    }
                }
            }
        }
    }
}

Additional Neighbor Options

Local AS

Override the AS number used in the BGP OPEN message for a specific neighbor. This is useful during AS migration.

"neighs": {
    "172.16.20.2 172.16.20.1": {
        "asn": 65016,
        "local_as": 65099,
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

MD5 Authentication

Enable TCP MD5 authentication for a BGP neighbor.

"neighs": {
    "172.16.20.2": {
        "asn": 65017,
        "auth_password": "secret123",
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

Disabled Neighbor

A neighbor can be administratively disabled. The neighbor configuration is retained but no TCP connections will be established.

"neighs": {
    "172.16.20.2": {
        "asn": 65017,
        "disabled": true,
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

Allow AS In

Allow the local AS number to appear in the AS path received from a neighbor. The value specifies the maximum number of times the local AS is allowed. Default is 0 (not allowed).

"neighs": {
    "172.16.20.2": {
        "asn": 65017,
        "allow_as_in": 2,
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

Remove Private AS

Remove private AS numbers from the AS path when advertising to a neighbor. Options are RemoveAll (remove all private AS numbers) or ReplaceAll (replace private AS numbers with the local AS).

"neighs": {
    "172.16.20.2": {
        "asn": 65017,
        "remove_private_as": "RemoveAll",
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

AS Override on Export

Replace the neighbor's AS number in the AS path with the local AS when advertising routes.

"neighs": {
    "172.16.20.2": {
        "asn": 65017,
        "as_override_export": true,
        "afis": {
            "ipv4-unicast": {}
        }
    }
}

Configuring L3VPN (BGP/MPLS VPN)

L3VPN configuration uses the l3vpn object within a VRF to define the Route Distinguisher (RD) and Route Targets (RTs). BGP neighbors in the default VRF carry VPN routes using the ipv4-mplsvpn or ipv6-mplsvpn address families.

"bgp": {
    "asn": 200,
    "id": "200.0.100.200",
    "vrfs": {
        "default": {
            "neighs": {
                "200.0.100.251 200.0.100.200": {
                    "asn": 200,
                    "local_as": 200,
                    "afis": {
                        "ipv4-mplsvpn": {}
                    }
                }
            }
        },
        "green": {
            "l3vpn": {
                "rd": "0:100:100",
                "rts_v4": {
                    "import": [
                        "route-target:100:100"
                    ],
                    "export": [
                        "route-target:100:100"
                    ]
                }
            },
            "neighs": {
                "100.200.1.100": {
                    "asn": 100,
                    "local_as": 200,
                    "afis": {
                        "ipv4-unicast": {}
                    }
                }
            }
        }
    }
}

The l3vpn object contains:

rd — Route Distinguisher in type:value:value format
rts_v4 — IPv4 route targets with import and export lists
rts_v6 — IPv6 route targets with import and export lists

Configuring EVPN

EVPN (Ethernet VPN) uses the l2vpn-evpn address family. Ethernet Segment Identifiers (ESIs) for multihoming are configured under the system intfs object.

"system": {
    "intfs": {
        "bond-d80": { "esi": "03:44:38:39:ff:ff:01:00:00:01" }
    }
},
"bgp": {
    "asn": 80,
    "id": "10.80.100.80",
    "vrfs": {
        "default": {
            "neighs": {
                "10.80.100.88 10.80.100.80": {
                    "asn": 80,
                    "afis": {
                        "ipv4-unicast": {},
                        "l2vpn-evpn": {}
                    }
                }
            }
        }
    }
}

Route Aggregation

Route aggregation allows summarizing multiple BGP prefixes into a single aggregate prefix. Configure aggregates under the aggregate_addresses object within a VRF.

"bgp": {
    "asn": 65016,
    "id": "172.16.10.1",
    "vrfs": {
        "default": {
            "aggregate_addresses": {
                "10.0.0.0/8": {
                    "summary_only": true,
                    "as_set": false
                }
            }
        }
    }
}

The aggregate address options are:

summary_only — when true, suppress the more-specific prefixes and only advertise the aggregate
as_set — when true, include an AS_SET in the aggregate to preserve path information from the contributing routes
export_to_vpn — when true, export the aggregate to VPN address families

Configuring prefix-limit

The prefix-limit feature can be configured per-AFI at the neighbor level or at the VRF level. By default prefix-limit is disabled.

Inheritance principle is used to determine which configuration takes effect for a neighbor: neighbor level configuration takes precedence if present, if not VRF level configuration if present.

Configuration consists of two items:

The maximum number of prefixes allowed ('max-prefixes')
The action to take when max-prefixes has been reached. The default action is 'discard', 'reset' is also supported but not recommended.

When the action is 'discard', any extra prefixes are discarded. When the number of prefixes eventually goes below 'max-prefixes', a Route-Refresh msg is sent to the peer to request the peer to resend Update msgs. The Route-Refresh msg is sent at most once every minute (hard-coded).

When the action is 'reset', the BGP connection to the neighbor is reset when we exceed the allowed 'max-prefixes'. This can lead to network convergence issues and is therefore not recommended.

There is also an alarm raised when we reach the soft-limit, which is set to 75% of max-prefixes. Another alarm is raised if we exceed 'max-prefixes'.

BGP Operation Commands Reference

Help

flockc bgp -h

Overview

flockc bgp

Memory usage

flockc bgp -m

Neighbors

flockc bgp -v <vrf-name> -n [<ip-addr>]

Neighbor adjacency RIB

flockc bgp -v <vrf-name> -n <ip-addr> --adj-rib {in | out}

Prefixes

flockc bgp -v <vrf-name> --af {ipv4 | ipv6 | l2vpn} -p [<ip-network>]

VPN route lookup by RD

flockc bgp -v <vrf-name> --af ipv4 -p --rd <rd>

Event buffer

flockc bgp --events

Source deaggregation labels

flockc bgp --source-deagg-labels

EVPN L2 VNI database

flockc bgp --vni [<vni>]

EVPN Ethernet Segment database

flockc bgp --es [<esi>]

BFD Component

Overview

BFD (Bidirectional Forwarding Detection) provides sub-second failure detection for routing protocols. BFD sessions are created by protocols (BGP, OSPF, static routes) rather than configured directly. When a BFD session detects that a neighbor is unreachable, the associated protocol is notified immediately, enabling faster convergence than relying on protocol-level hello timers alone.

Configuration

BFD sessions are not configured independently. Instead, BFD is enabled per-protocol:

BFD for OSPFv2 / OSPFv3 interfaces

Enable BFD on an OSPF interface by adding a bfd object with multiplier and min_tx parameters.

"ospfv2": {
    "vrfs": {
        "default": {
            "router_id": "10.0.100.1",
            "areas": {
                "0.0.0.0": {
                    "intfs": {
                        "^eth": {
                            "bfd": {
                                "multiplier": 3,
                                "min_tx": "1s"
                            }
                        }
                    }
                }
            }
        }
    }
}

The same configuration applies to OSPFv3 interfaces under the ospfv3 object.

BFD for BGP neighbors

Enable BFD on a BGP neighbor by adding a bfd object with multiplier and min_tx parameters.

"bgp": {
    "asn": 65016,
    "id": "172.16.10.1",
    "vrfs": {
        "default": {
            "neighs": {
                "172.16.20.2 172.16.20.1": {
                    "asn": 65016,
                    "bfd": {
                        "multiplier": 3,
                        "min_tx": "1s"
                    },
                    "afis": {
                        "ipv4-unicast": {}
                    }
                }
            }
        }
    }
}

BFD parameters

multiplier — the number of missed BFD packets before declaring the session down. Default is 3.
min_tx — the minimum interval between transmitted BFD packets. Specified as a duration string (e.g., "1s", "300ms").

The detection time is multiplier × min_tx. For example, with multiplier: 3 and min_tx: "1s", failure is detected within 3 seconds.

Operational State

BFD overview

flockc bfd

Show all BFD sessions

flockc bfd -s

Filter sessions by destination IP

flockc bfd -s --dest_ip <ip-address>

Filter sessions by source IP

flockc bfd -s --src_ip <ip-address>

Filter sessions by interface name

flockc bfd -s --intf_name <interface-name>

Static Component

Configuration Overview

The static route configuration is held under the top level static object in /etc/flockd/flockd.json. If the static object exists static routes will be enabled and the static master thread will be spawned.

In the configuration format:

The vrfs object is a map of VRF names to VRF configurations.
The routes object is a map of route prefixes to route configurations. The prefix can be either IPv4 or IPv6.
The next_hops array specifies a set of next-hop(s) for the static route.
Each next-hop can be a destination ip_addr, an outgoing intf_name, both, or a special next-hop string ("Discard" or "Reject").

In this example there is a single VRF containing a single static route to the 192.0.2.0/25 network. Traffic for that network will be forwarded to the neighboring device with the address 10.0.1.2.

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "192.0.2.0/25": {
                    "next_hops": [
                        { "ip_addr": "10.0.1.2" }
                    ]
                }
            }
        }
    }
}

Configuration in detail

If a next_hop only contains an ip_addr the static route is said to be "recursive" as a further lookup is needed to find the outgoing interface. If a next_hop has an outgoing intf_name defined the static route is said to be "attached" (non-recursive).

"Floating static" routes can also be defined using the distance object. A "floating static" route is given a high distance (Admin Distance) which means during normal operation the RIB will choose to use another protocol that has a lower distance. If there is a network outage and the preferred route is withdrawn from the RIB, the "floating static" route will take over.

Recursive route configuration

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "192.0.2.0/25": {
                    "next_hops": [
                        { "ip_addr": "10.0.1.2" }
                    ]
                }
            }
        }
    }
}

Attached route configuration

An attached route specifies both an ip_addr and an intf_name, pinning the route to a specific outgoing interface.

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "192.0.2.128/25": {
                    "next_hops": [
                        { "ip_addr": "10.0.1.2", "intf_name": "r1-n1" }
                    ]
                }
            }
        }
    }
}

Multi-path recursive route configuration

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "20.20.20.0/24": {
                    "next_hops": [
                        { "ip_addr": "10.10.10.2" },
                        { "ip_addr": "11.11.11.2" }
                    ]
                }
            }
        }
    }
}

Discard route configuration

A discard route silently drops matching traffic (similar to a blackhole route).

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "198.51.100.0/24": {
                    "next_hops": [
                        "Discard"
                    ]
                }
            }
        }
    }
}

Reject route configuration

A reject route drops matching traffic and sends an ICMP unreachable response to the sender.

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "203.0.113.0/24": {
                    "next_hops": [
                        "Reject"
                    ]
                }
            }
        }
    }
}

Floating static route configuration

"static": {
    "vrfs": {
        "default": {
            "routes": {
                "20.20.20.0/24": {
                    "next_hops": [
                        { "ip_addr": "10.10.10.2" }
                    ],
                    "distance": 120
                }
            }
        }
    }
}

Operational State Overview

Check Static is enabled

Check static is listed in the enabled_protocols field.

flock@r70:~$ flockc system
{"host_info":{"hostname":"r70", ...},"system_info":{"name":"flockd", ...},"pid":1234,"log_level":"info","uptime":"days: 0, hours: 0, mins: 0, secs: 19","enabled_protocols":["BGPv4","OSPFv2","Static"],"software_errors":0, ...}

Show Static Overview

This shows the count and state of VRF's learned from the system component and the active number of IPv4 and IPv6 static routes across all VRF's. It also shows the number of interfaces that the static component is aware of, these include both the system interfaces that are currently enabled as well as the interfaces that are not enabled in the system but are referred to by their names in the static route configuration.

flock@r70:~$ flockc static
{"sys_vrfs":{"vrf_count_active":1,"vrf_count_total":1},"ipv4":{"route_count":5},"ipv6":{"route_count":3},"intf_count":4}

Show Static prefixes

Note that this is not the System RIB. The static routes show all the routes from the configuration. For an attached route, it is only programmed in the RIB once its interface has been assigned an ID by the kernel.

Show all prefixes.

flock@r70:~$ flockc static -v default --af ipv4 -p
{"70.0.0.0/8":[{"admin_dist":null,"origin":{"Static":"Conf"},"nhs":[{"SpecialNextHop":{"vrf_id":254,"special_next_hop_type":"Discard"}}]}],"70.0.75.0/24":[{"admin_dist":0,"origin":"Connected","nhs":[{"Connected":{"name":"r70-n75", ...}}]}], ...}

Show a specific prefix.

flock@r01:~$ flockc static -v default --af ipv4 -p 40.40.40.0/24
{"40.40.40.0/24":[{"admin_dist":null,"origin":{"Static":"Conf"},"nhs":[...]}]}

default is the default vrf-name if the -v option is not specified
ipv6 is the default AF type if the af option is not given

Show Static prefixes in a VRF

flock@r01:~$ flockc static -v pink --af ipv4 -p
{...}

default is the default vrf-name if the -v option is not specified
ipv6 is the default AF type if the af option is not given

Show Static VRF's Overview

flock@r50:~$ flockc static -v
[{"vrf_id":1000,"route_count_ipv4":{"route_count":1024},"route_count_ipv6":{"route_count":42}},{"vrf_id":1001,"route_count_ipv4":{"route_count":73},"route_count_ipv6":{"route_count":56}}]

Show all interfaces

Static tracks interfaces in the system. These include:

the interfaces that are enabled and have been assigned an interface ID by the kernel
the interfaces that are referred to by their names in the static routes but have not been assigned an interface ID by the kernel

Static shows the current interface state as follows:

the interface name
the interface id assigned by the kernel, if one exists
number of attached_routes referencing the interface

In the following example, ens4 interface has not been assigned an ID by the kernel

flock@r61:~$ flockc static -i
{"name":"ens1","id":2,"attached_routes":3}
{"name":"ens2","id":3,"attached_routes":2}
{"name":"ens3","id":4,"attached_routes":0}
{"name":"ens4","attached_routes":1}
{"name":"lo","id":1,"attached_routes":0}

Static Operation Commands Reference

Help

flockc static -h

Overview

flockc static

All static interfaces

flockc static -i

Single interface

flockc static -i <interface-name>

All VRF 's

flockc static -v

Single VRF

flockc static -v <vrf-name>

Prefixes

flockc static [-v <vrf-name>] -p [<ip-network>] [--af {ipv4 | ipv6}]

default is the default vrf-name if the -v option is not specified
ipv6 is the default AF type if af option is not specified

Linux Kernel Settings

Linux kernel settings can be read using sysctl and written using sysctl -w. They can be made permanent / configured on boot, by adding an entry to /etc/sysctl.conf.

IP forwarding

For a Linux host to operate as an IP Router, IP forwarding must be enabled. flockd enables IP forwarding on startup.

/proc/sys/net/ipv4/ip_forward = 1
/proc/sys/net/ipv6/conf/all/forwarding = 1

Bind IPv6 Only

By default a Linux host will operate as an IPv4/IPv6 dual stack node. See RFC 4038: 4.2. IPv6 Applications in a Dual-Stack Node.

This means that when flockd binds to an IPv6 socket, IPv4 requests will also be serviced. To restrict IPv6 sockets to only service IPv6 requests, the IPV6_V6ONLY socket option needs to be set.

/etc/sysctl.conf
  net.ipv6.bindv6only = 1

BGP / TCP Termination at scale

Some protocols (most notably BGPv4) and the Operation API rely on a TCP transport. The Linux kernel has two parameters to control how many TCP connections can simultaneously be formed.

tcp_max_syn_backlog: Max TCP connections waiting for final ACK (of the TCP three way handshake)

flock@flocknet:/proc/sys/net/ipv4$ cat tcp_max_syn_backlog
256

somaxconn: Max TCP connections with completed TCP three way handshake waiting for accept() to be called.

flock@flocknet:/proc/sys/net$ sudo cat core/somaxconn
128

If these limits are exceeded the Linux kernel decides it is under a SYN DoS attack and will prevent further connections. Under these conditions this message is logged in /var/log/messages

"TCP: request_sock_TCP: Possible SYN flooding on port 179. Sending cookies."

In a production network it is very unlikely these limits will be reached (unless the router is under a SYN DoS attack). Even with 1000's of BGP neighbors it is unlikely that there will be greater than 128 TCP connections waiting to be accepted. However in the lab using a traffic generator this limit can be hit.

By default the Flock Networks Routing Suite is configured to be able to handle up to 1024 simultaneous BGPv4 TCP connections. To reach this scale the Linux kernel defaults need to be updated to match.

/etc/sysctl.conf
  net.ipv4.tcp_max_syn_backlog=1024
  net.core.somaxconn=1024

Environment Variables

Some Flock routing suite configuration can be set using environment variables.

flockd environment variables can be set in the systemd service file.

flockc environment variables can be set on the command line

RUST_LOG=debug flockc system

FLOCKD_CONF_DIR

FLOCKD_CONF_DIR sets the directory that flockd will look in for its configuration. If not set /etc/flockd will be used.

FLOCKD_CONF_FILE

FLOCKD_CONF_FILE sets the configuration filename that flockd will look for. If not set flockd.json will be used.

FLOCKD_PID_FILE_PATH

FLOCKD_PID_FILE_PATH sets the directory and filename that will hold the flockd process id file. If not set /var/run/flockd.pid will be used.

RUST_LOG

The default log level is info. At this level all info and higher priority levels will be logged. Supported log levels in descending priority order are error, warn, info, debug and trace.

Error

Log level error / [ERROR] is used for unexpected events signalled from inside the router. These are never expected to be seen and indicate a bug. Please email a bug report to: support@flocknetworks.com.

Warning

Log level warn / [WARN] is used for unexpected events signalled from outside the router. It is normal to see warnings whilst the network is converging. Warnings should never be seen after the network has converged and remains stable.

[WARN  flockd::ospf_neigh] RouterId(10.0.100.2), V4(10.0.3.157) neigh state change Full -> Down

Information

Log level info / [INFO] is used for expected events of note

[INFO  flockd] START: PID 385 Compile Mode Release Log Level "debug"
[INFO  flockd::sys::sys_intf] Update IntfId(2)] Broadcast Mtu(1500) Up [] event DownToUp
[INFO  flockd::ospf_intf] IntfId(2), 10.0.1.168/24 state change Wait -> DrOther
[INFO  flockd::ospf_neigh] RouterId(10.0.100.3), V4(10.0.1.152) neigh state change Loading -> Full

Debug

Log level debug / [DEBUG] is used for common expected events.

Trace

Log level trace / [TRACE] is used for very common expected events.

Changing the flockd default log level

The default log level can be changed by setting the RUST_LOG environment variable in the flockd systemd service file.

grep RUST_LOG /lib/systemd/system/flockd.service
    Environment=RUST_LOG="info"

When the systemd service file has changed, systemd needs to be told to reload the new flockd configuration.

# systemctl daemon-reload

To enable the new log level, flockd needs to be restarted.

# systemctl restart flockd

Different log levels can be set for different components. This example sets the default log level for all components except the BGP component to info. The BGP component will log at the debug level.

RUST_LOG="info,fire::bgp=debug"

RFC Compliance

Flock Networks Routing Suite implements the following RFC's.

BFD

RFC 5880 Bidirectional Forwarding Detection (BFD)

BGPv4

RFC 1997 BGP Communities Attribute
RFC 2545 BGP-4 Multiprotocol Extensions for IPv6 IDR
RFC 2918 Route Refresh Capability for BGP-4
RFC 3107 Carrying Label Information in BGP-4
RFC 4271 A Border Gateway Protocol 4 (BGP-4)
RFC 4360 BGP Extended Communities Attribute
RFC 4364 BGP/MPLS IP Virtual Private Networks (VPNs)
RFC 4456 BGP Route Reflection
RFC 4486 Subcodes for BGP Cease Notification Message
RFC 4684 Constrained Route Distribution for Border Gateway Protocol/MultiProtocol Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs)
RFC 4724 Graceful Restart Mechanism for BGP
RFC 4760 Multiprotocol Extensions for BGP-4
RFC 5492 Capabilities Advertisement with BGP-4
RFC 5668 4-Octet AS Specific Extended Community
RFC 6286 Autonomous-System-Wide Unique BGP Identifier for BGP-4
RFC 6513 Multicast in MPLS/BGP IP VPNs
RFC 6514 BGP Encodings and Procedures for Multicast in MPLS/BGP IP VPNs
RFC 6608 Subcodes for BGP Finite State Machine Error
RFC 6793 BGP Support for Four-Octet Autonomous System (AS) Number Space
RFC 7300 Reservation of Last Autonomous System (AS) Numbers
RFC 7606 Revised Error Handling for BGP UPDATE Messages
RFC 7911 Advertisement of Multiple Paths in BGP
RFC 7947 Internet Exchange BGP Route Server
RFC 7988 Ingress Replication Tunnels in Multicast VPN
RFC 8092 BGP Large Communities Attribute
RFC 8203 BGP Administrative Shutdown Communication
RFC 8654 Extended Message Support for BGP
RFC 8950 Advertising IPv4 Network Layer Reachability Information (NLRI) with an IPv6 Next Hop
RFC 9012 The BGP Tunnel Encapsulation Attribute

IP

RFC 2460 Internet Protocol, Version 6 (IPv6) Specification
RFC 4007 IPv6 Scoped Address Architecture
RFC 4038 Application Aspects of IPv6 Transition

MPLS

RFC 3032 MPLS Label Stack Encoding
RFC 5462 Multiprotocol Label Switching (MPLS) Label Stack Entry

OSPF

RFC 2328 OSPF Version 2
RFC 4576 Using a Link State Advertisement (LSA) Options Bit to Prevent Looping in BGP/MPLS IP Virtual Private Networks (VPNs)
RFC 5309 Point-to-Point Operation over LAN in Link State Routing Protocols
RFC 5340 OSPF for IPv6
RFC 6987 OSPF Stub Router Advertisement
RFC 8362 OSPFv3 Link State Advertisement (LSA) Extensibility
RFC 9350 IGP Flexible Algorithm

Manual Install

Flock Networks Routing Suite can be manually installed on any Linux system. The binaries have no dependencies other than the Linux Kernel API (Sockets, Netlink, etc).

Extract the contents of the Debian package

You will need the ar archive utility from binutils.

For example on Fedora;

# yum install binutils

Then extract the contents of the debian package

$ ar x flockd_<version>_amd64.deb
$ ls
control.tar.gz  data.tar.xz  debian-binary  flockd_<version>_amd64.deb
$ tar -xf data.tar.xz
$ ls
control.tar.gz  debian-binary  flockd_20.4.x_amd64.deb  usr data.tar.xz etc lib

Install Files

Copy the flockd daemon config files to /etc/flockd

# mkdir /etc/flockd
# cp etc/flockd/* /etc/flockd

Copy the flockd daemon to /usr/sbin

# cp usr/sbin/flockd /usr/sbin

Copy the flockc client to /usr/bin

# cp usr/bin/flockc /usr/bin

Start the daemon

# RUST_LOG=info /usr/sbin/flockd &

Check the daemon is running

$ flockc system
"hostname": "flocknet"
"software": "Flock Networks Routing Suite"
"version": "<version>"
"model": "Large"
"base_os": "Linux"
"pid": 2423
"compile_mode": "Release"
"log_level": "info"
"uptime": Uptime { days: 0, hours: 0, mins: 0, secs: 19 }
"enabled_protocols": ["OSPFv2"]
$

SONiC Support

By default, Flock uses the Linux kernal as control plane. However, the Flock Networks Routing Suite can act as the IP Routing control plane for the SONiC Network Operating System. SONiC uses the forwarding plane manager (FPM) to program fpmsyncd. fpmsyncd then sends the routing updates into the SONiC APPL_DB database.

To program fpmsyncd with IP routes the rib object in the /etc/flockd/flockd.json configuration file requires the fpm object to be defined.

{
    "rib": {
        "dataplane": {
            "fpm":  {
                "tcp_port": 2620
            }
        }
    }
}

Then check FPM is enabled by using the REST API.

$ flockc rib -d -j | grep fpm_state
"fpm_state": "enabled"
$

NB: The default IP Routing suite shipped with SONiC will need to be disabled, or it will conflict with the Flock Networks IP Routing Suite.

Debugging SONiC

An overview of the SONiC routing-state interactions is here.

Check the `fpmsyncd` logs for a successful connection from the Flock Networks IP Routing Suite.

# grep "fpmsyncd Connected" /var/log/messages
2020-08-25T08:44:49.707069+00:00 ad2e7e7845ad supervisord: fpmsyncd Connected!

Check the ASIC's have been programmed

The entries below match this SONiC P4 example.

host1 (Ubuntu, 192.168.1.2/24) <--> switch1 (SONiC) <--> switch2 (SONiC) <--> host2 (Ubuntu, 192.168.2.2/24)

Use the SONiC saidump utility to show the programmed ASIC entries. Below are shown the entries used to forward to the remote BGP subnet 192.168.2.0/24.

switch1$ saidump

# Remote BGP subnet
SAI_OBJECT_TYPE_ROUTE_ENTRY {"dest":"192.168.2.0/24","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000066"}
    SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID : oid:0x400000000009c

# Which uses this nexthop
SAI_OBJECT_TYPE_NEXT_HOP oid:0x400000000009c
    SAI_NEXT_HOP_ATTR_IP                  : 10.0.0.1
    SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID : oid:0x6000000000095
    SAI_NEXT_HOP_ATTR_TYPE                : SAI_NEXT_HOP_TYPE_IP

# Which is out of this interface
SAI_OBJECT_TYPE_ROUTER_INTERFACE oid:0x6000000000095
    SAI_ROUTER_INTERFACE_ATTR_SRC_MAC_ADDRESS   : 00:01:04:4C:49:F5
    SAI_ROUTER_INTERFACE_ATTR_TYPE              : SAI_ROUTER_INTERFACE_TYPE_VLAN
    SAI_ROUTER_INTERFACE_ATTR_VIRTUAL_ROUTER_ID : oid:0x3000000000066
    SAI_ROUTER_INTERFACE_ATTR_VLAN_ID           : oid:0x2600000000008f

Monitoring and Configuration via the gRPC API

flockd supports a gRPC API for monitoring and configuring the router. The gRPC API uses Protocol Buffers for message serialization and HTTP/2 for transport. The flock_cli client uses the gRPC API.

Unlike the REST API which is read-only, the gRPC API also supports configuration updates (e.g. BGP neighbor and VRF configuration).

The `flock_cli` Client

flock_cli is the gRPC-based CLI client for flockd. It connects to the gRPC endpoint (default http://[::1]:50051) and supports multiple output formats.

Global Options

flock_cli [OPTIONS] <COMMAND>

Options:
  -e, --endpoint <ENDPOINT>  gRPC endpoint [env: FLOCK_ENDPOINT=] [default: http://[::1]:50051]
  -f, --format <FORMAT>      Output format [default: json]
                              [possible values: debug, debug-pretty, json, json-pretty, yaml]

Connecting to a Remote Router

By default, flock_cli connects to the local flockd instance via IPv6 loopback. To connect to a remote router, set the endpoint:

flock_cli -e http://10.0.1.1:50051 sys overview

Or via the FLOCK_ENDPOINT environment variable:

export FLOCK_ENDPOINT=http://10.0.1.1:50051
flock_cli sys overview

System Commands

System Overview

Returns system information including hostname, software version, uptime, enabled protocols, and software error count.

$ flock_cli -f json-pretty sys overview
{
  "host_info": {
    "hostname": "R01",
    "domain": null,
    ...
  },
  "system_info": {
    "name": "flockd",
    "description": "The Flock Networks Ltd Routing Suite Daemon",
    "version": "0.0.2"
  },
  "fire_info": {
    "name": "fire",
    "description": "Flock Internet Routing Engine (FIRE)",
    "version": "0.1.0"
  },
  "pid": 12345,
  "uptime": "days: 1, hours: 3, mins: 22, secs: 15",
  "enabled_protocols": ["BGPv4", "OSPFv2", "Static"],
  "software_errors": 0,
  "global_thread_pool_size": 10
}

List VRFs

Returns all VRFs configured on the system.

$ flock_cli sys list-vrfs
[{"vrf_id":254,"vrf_name":"default","underlay_vrf_name":null,...}]

BGP Commands

BGP Overview

Returns the BGP instance overview including BGP ID, ASN, route counts per VRF, and neighbor summary.

$ flock_cli -f json-pretty bgp overview
{
  "id": "70.0.100.70",
  "asn": 70,
  "route_server": false,
  "route_reflector": false,
  "total_event_loops": 571,
  "routes": {
    "unicast_routes": {
      "default": {
        "ipv4_unicast": { "route_count": 3, "attr_list_store_count": 1 },
        "ipv6_unicast": { "route_count": 0, "attr_list_store_count": 0 }
      }
    },
    "vpn_routes": {}
  },
  "neighbor_summary": {
    "default": {
      "count": 4,
      "established": 4,
      "send_converged": 4,
      "recv_converged": 4
    }
  }
}

List BGP VRFs

$ flock_cli bgp list-vrfs
[{"vrf_id":254,"vrf_name":"default","vrf_info":{"neighs":["90.0.93.61","70.0.100.73 70.0.100.70",...]},...}]

BGP Event Log

Returns the BGP event log showing neighbor state changes.

$ flock_cli -f json-pretty bgp event-log
{
  "object_oper": {
    "db": [
      {
        "event": "AddOrUpdate",
        "object_desc": "BGP Peer VrfId(254) 90.0.93.61",
        "time": "2026-02-27T13:42:00.015Z"
      },
      {
        "event": "Up",
        "object_desc": "BGP Peer VrfId(254) 90.0.93.61",
        "time": "2026-02-27T13:42:00.185Z"
      },
      ...
    ]
  }
}

BGP VRF Commands

All VRF-scoped commands use the form flock_cli bgp vrf <name> <command>.

Unicast RIB Lookup

Look up a single prefix in the BGP unicast RIB.

$ flock_cli bgp vrf default rib lookup 50.0.0.0/8

Unicast RIB Walk

Walk the BGP unicast RIB starting from a root prefix. Supports pagination via --start-from and --max-entries.

# Walk all routes
$ flock_cli bgp vrf default rib walk 0.0.0.0/0

# Walk with pagination (1 entry at a time)
$ flock_cli bgp vrf default rib walk 0.0.0.0/0 --max-entries 1

# Continue from a specific prefix
$ flock_cli bgp vrf default rib walk 0.0.0.0/0 --start-from 60.0.0.0/8 --max-entries 1

Each entry in the walk response includes a finished field indicating whether there are more entries to retrieve.

Unicast RIB Statistics

# Basic counters
$ flock_cli bgp vrf default rib stats

# Include memory usage details
$ flock_cli bgp vrf default rib stats --memory

Returns IPv4 and IPv6 counters: total prefixes, path counts by type (iBGP, eBGP, originated, redistributed, VPN-imported), and optionally memory usage per RIB entry.

De-aggregation Labels

$ flock_cli bgp vrf default rib deagg-labels

Redistribution Policy

$ flock_cli bgp vrf default redist-policy

Returns the redistribution policy status and statistics including hit counts and accept/reject counters.

VPN Policy Statistics

$ flock_cli bgp vrf default vpn policy-stats

BGP Neighbor Commands

Neighbor commands use the form flock_cli bgp vrf <vrf> neigh <addr> <command>.

The neighbor address can be a simple IP (90.0.93.61) or include a source IP (70.0.100.72 70.0.100.70, space-separated and quoted).

Show Neighbor

# Basic neighbor info
$ flock_cli bgp vrf default neigh 90.0.93.61 show

# Include detailed statistics
$ flock_cli bgp vrf default neigh 90.0.93.61 show --stats

Returns neighbor state (Established/Idle/etc.), capabilities, timers, keepalive counters, and optionally per-AFI statistics including policy hit counts and adj-rib prefix counts.

Reset Neighbor

$ flock_cli bgp vrf default neigh 90.0.93.61 reset soft-in
$ flock_cli bgp vrf default neigh 90.0.93.61 reset soft-out
$ flock_cli bgp vrf default neigh 90.0.93.61 reset hard
$ flock_cli bgp vrf default neigh 90.0.93.61 reset refresh-in
$ flock_cli bgp vrf default neigh 90.0.93.61 reset refresh-out

Adjacency RIB Walk

Walk routes received from or advertised to a specific neighbor.

# Routes received from neighbor (adj-rib-in)
$ flock_cli bgp vrf default neigh 90.0.93.61 adj-rib-unicast in walk 0.0.0.0/0

# Routes advertised to neighbor (adj-rib-out)
$ flock_cli bgp vrf default neigh 90.0.93.61 adj-rib-unicast out walk 0.0.0.0/0

# RTC adj-rib
$ flock_cli bgp vrf default neigh 90.0.93.61 adj-rib-rtc in walk

BGP VPN RIB Commands

VPN RIB Lookup

$ flock_cli bgp vpn-rib lookup <rd> <ip-net>

VPN RIB Walk

$ flock_cli bgp vpn-rib walk <rd> <ip-net> [--start-from-rd <rd>] [--start-from-ip-net <ip-net>] [--max-entries N] [--walk-rds]

VPN RIB Statistics

$ flock_cli bgp vpn-rib stats [--ip-version <4|6>] [--memory]

BGP RTC RIB Commands

RTC RIB Lookup

$ flock_cli bgp rtc-rib lookup <rtc-net>

RTC RIB Walk

$ flock_cli bgp rtc-rib walk

BGP AFI RIB Commands

Query AFI-specific RIBs (e.g., EVPN).

$ flock_cli bgp vrf default afi-rib <afi> lookup <prefix>
$ flock_cli bgp vrf default afi-rib <afi> walk <root> [--start-from <prefix>] [--max-entries N]

BGP Configuration Commands

The gRPC API supports reading and updating BGP configuration. Configuration changes use a pending-config workflow: changes are staged locally in a .bgp_pending_config file, then applied atomically to the running router.

Show Current Configuration

# Full configuration
$ flock_cli bgp config show

# Configuration for a specific VRF
$ flock_cli -f json-pretty bgp config show --vrf default
[
  {
    "name": "default",
    "multipath": false,
    "neighs": [
      {
        "key": { "dst": { "version": { "V4": 1509973309 } }, "src": null },
        "asn": 60,
        "local_as": 70,
        "route_reflector_client": false,
        "next_hop_self": false,
        "disabled": false,
        "connect_mode": "Both",
        ...
      }
    ],
    "networks": [
      { "prefix": "70.0.0.0/8", "originate_always": false }
    ],
    ...
  }
]

Pending Configuration Workflow

Configuration changes follow a four-step workflow: initialize, stage, review, apply.

Step 1: Initialize

Create a fresh pending configuration staging file:

$ flock_cli bgp config init
initialized .bgp_pending_config

Step 2: Stage Changes

Stage one or more changes. Multiple changes can be staged before applying:

$ flock_cli bgp config set --asn 65001
$ flock_cli bgp config vrf default set --multipath
$ flock_cli bgp config vrf default neigh 10.0.0.1 set --asn 65002

Step 3: Review Pending Changes

Inspect what will be applied before committing:

$ flock_cli bgp config show-pending

Step 4: Apply

Apply all staged changes atomically to the running router:

$ flock_cli bgp config apply
config applied

Example: Adding a BGP Neighbor

# Initialize pending config
$ flock_cli bgp config init
initialized .bgp_pending_config

# Stage the new neighbor
$ flock_cli bgp config vrf default neigh 10.99.99.1 set --asn 65099 --local-as 70

# Review - the pending config shows the staged addition
$ flock_cli bgp config show-pending
{"asn":null,...,"vrfs":{"clear":false,"ops":[{"name":"default","delete":false,
  "neighs":{"clear":false,"ops":[{"key":{"dst":{"version":{"V4":174285569}},
  "src":null,"zone":null},"delete":false,"asn":{"v":{"Set":65099}},
  "local_as":{"v":{"Set":70}},...}]}}]}}

# Apply to the running router
$ flock_cli bgp config apply
config applied

# Verify - neighbor count increased and the new neighbor is visible
$ flock_cli bgp overview
{...,"neighbor_summary":{"default":{"count":5,"established":4,...}}}

$ flock_cli bgp vrf default neigh 10.99.99.1 show
{"common":{"neigh_key":"10.99.99.1","asn":65099,"local_as":70,
  "neigh_type":"External","connect_mode":"Both",...},...}

Example: Deleting a BGP Neighbor

# Initialize pending config
$ flock_cli bgp config init
initialized .bgp_pending_config

# Stage the deletion
$ flock_cli bgp config vrf default neigh 10.99.99.1 delete

# Review - the pending config shows delete:true
$ flock_cli bgp config show-pending
{"asn":null,...,"vrfs":{"clear":false,"ops":[{"name":"default","delete":false,
  "neighs":{"clear":false,"ops":[{"key":{"dst":{"version":{"V4":174285569}},
  "src":null,"zone":null},"delete":true,...}]}}]}}

# Apply to the running router
$ flock_cli bgp config apply
config applied

# Verify - neighbor count decreased and the neighbor is gone
$ flock_cli bgp overview
{...,"neighbor_summary":{"default":{"count":4,"established":4,...}}}

$ flock_cli bgp vrf default neigh 10.99.99.1 show
null

Configuration Set Options

Global BGP settings:

$ flock_cli bgp config set [--asn <ASN>] [--route-server] [--route-reflector]

VRF settings:

$ flock_cli bgp config vrf <name> set [--multipath] [--redist-policy <policy>]
$ flock_cli bgp config vrf <name> delete

Neighbor settings:

$ flock_cli bgp config vrf <name> neigh <addr> set \
    [--asn <ASN>] [--local-as <ASN>] \
    [--route-reflector-client] [--next-hop-self] \
    [--disabled] [--connect-mode <mode>]
$ flock_cli bgp config vrf <name> neigh <addr> delete

Output Formats

flock_cli supports five output formats via the -f flag:

Format	Description
`json`	Compact JSON (default)
`json-pretty`	Pretty-printed JSON
`yaml`	YAML format
`debug`	Rust debug format
`debug-pretty`	Rust debug format, pretty-printed

Example using YAML:

$ flock_cli -f yaml bgp vrf default rib stats
v4:
  counters:
    total_prefixes: 3
    num_ibgp_regular_paths: 0
    num_ebgp_regular_paths: 2
    num_originated_paths: 1
    ...
v6:
  counters:
    total_prefixes: 0
    ...

3rd Party gRPC Clients

Since the gRPC API uses standard Protocol Buffers and HTTP/2, any gRPC client library can connect to flockd. The .proto service definitions are located in the flock_grpc/proto/ directory.

Available Services

Service	Proto File	Description
`SysService`	`sys.proto`	System overview and VRF listing
`BgpOperService`	`bgp_oper.proto`	BGP operational state queries
`BgpConfigService`	`bgp_config.proto`	BGP configuration read/write

Example: Connecting with `grpcurl`

# List available services
grpcurl -plaintext [::1]:50051 list

# Get system overview
grpcurl -plaintext [::1]:50051 sys.SysService/GetOverview

# Get BGP overview
grpcurl -plaintext -d '{}' [::1]:50051 bgp_oper.BgpOperService/GetOverview

Example: Connecting with Python

import grpc
# Generated from proto files using grpc_tools.protoc
import sys_pb2, sys_pb2_grpc

channel = grpc.insecure_channel('[::1]:50051')
stub = sys_pb2_grpc.SysServiceStub(channel)
response = stub.GetOverview(sys_pb2.GetOverviewRequest())
print(response)

gRPC Command Reference

System Service RPCs

RPC	Description
`GetOverview`	System overview (hostname, version, uptime, protocols, errors)
`ListVrfs`	List all VRFs

BGP Operational Service RPCs

RPC	Description
`GetOverview`	BGP instance overview (ASN, BGP ID, route counts, neighbor summary)
`ListVrfs`	List BGP VRFs with neighbor info
`GetNeighbor`	Neighbor details with optional statistics
`ResetNeighbor`	Reset a BGP neighbor session
`GetEventLog`	BGP event log
`GetRedistPolicy`	Route redistribution policy and stats
`GetVpnPolicyStats`	VPN import/export policy stats
`UnicastRibLookup`	Look up a single unicast route
`UnicastRibWalk`	Walk the unicast RIB with pagination
`UnicastRibStats`	Unicast RIB statistics with optional memory
`GetSourceDeAggLabels`	De-aggregation label information
`VpnRibLookup`	Look up a single VPN route
`VpnRibWalk`	Walk the VPN RIB with pagination
`VpnRibStats`	VPN RIB statistics
`RtcRibLookup`	Look up a single RTC route
`RtcRibWalk`	Walk the entire RTC RIB
`AfiRibLookup`	AFI-specific RIB lookup
`AfiRibWalk`	AFI-specific RIB walk
`AdjRibUnicastLookup`	Adjacency RIB unicast lookup
`AdjRibUnicastWalk`	Adjacency RIB unicast walk
`AdjRibVpnLookup`	Adjacency RIB VPN lookup
`AdjRibVpnWalk`	Adjacency RIB VPN walk
`AdjRibRtcLookup`	Adjacency RIB RTC lookup
`AdjRibRtcWalk`	Adjacency RIB RTC walk
`AdjRibPrivateLookup`	Adjacency RIB private lookup
`AdjRibPrivateWalk`	Adjacency RIB private walk

BGP Configuration Service RPCs

RPC	Description
`GetBgpConfig`	Read the full BGP configuration
`UpdateBgpConfig`	Apply a configuration update

Keyboard shortcuts

User Guide: Flock Networks Routing Suite [Copyright 2019]