System Tune

SoodarOS consist of services working together. These services can be categorized into three classes:

Data-plane Services
Control-plane Services
Management Services

Data-plane Services

Data-plane services are responsible for forwarding, routing, and filtering incoming packets.

Members

VPP: VPP is a fast, scalable layer 2-4 multi-platform network stack.

Control-plane Services

All services related to signaling and routing protocols are in this class.

Members

FRRouting: FRRouting (FRR) is a free and open-source Internet routing protocol suite.
Strongswan: IPSec/IKEv2 implementation

Management Services

Services responsible for connecting from outside to SoodarOS.

Members

OpenSSH: OpenSSH is a suite of secure networking utilities based on the SSH protocol.

Tweaks

Depending on each use case’s needs, admins can tweak services and their resources. These tweaks are defined as Tuning profiles.

Common tweaks

For all above three categories of services, the following parameters can be set:

CPU cores: Admin can specify CPU cores that the service is allowed to utilize. This core pinning is done in two ways:
- Exclusive: The core is only assigned to this service and is not used in OS Scheduler.
- Shared: The core lists in OS Scheduler and other processes can use them.
Note

For data-plane services, CPU core assigning is done differently. refer to Data-plane tweaks
CPU weight: In case of shared CPUs, The available CPU time is split up among all services relative to their CPU time weight. A higher weight means more CPU time, a lower weight means less. The allowed range is 1 to 10000. Defaults to 100 for all processes.
Maximum memory: Specify the absolute limit on memory usage of the executed processes in this class.

Data-plane tweaks

VPP as a sophisticated and high-performance network has lots of customizable options.

Data-plane CPU configuration

VPP needs at least one main thread and more optional worker threads. Each of These threads is pinned to a CPU core. The default configuration uses core 0 as the main thread( in shared mode) and no workers. Admin can set main thread/ worker threads, CPU exclusiveness, and service CPU weight.

Data-plane Memory configuration

VPP uses its memory management system, so in the beginning, it ``mmap``s needed memories from OS heap.

Main heap: VPP’s main memory. Used to store ACL rules, IP Routes,… . Defaults to 1GB.
Stat heap: Memory assigned to storing stats( like packets count). Defaults to 32MB.

Admin can change these values when needed.

Data-plane buffer configuration

VPP uses per NUMA pre-allocated buffers to process incoming packets. It’s advised not to change default values, but admins can change the number of buffers and buffer size.

Data-plane poll sleep

VPP’s main loop poll always fetches new packets and uses 100% CPU. An admin can add a fixed sleep between the main loop poll to lower the power usage, CPU usage, and heat production in small devices.

Warning

Enabling poll sleep can cause performance degradation.

Interface order configuration

Network interfaces are sorted by their PCI address. For example, a router with 4 network interfaces and their respective addresses is mapped like:

Interfaces’ list( original)
PCI Address	Order	Interface Name	MAC Address
00:04.0	0	ge0	0c:61:0c:83:00:00
00:05.0	1	ge1	0c:61:0c:83:00:01
00:06.0	2	ge2	0c:61:0c:83:00:02
00:07.0	3	ge3	0c:61:0c:83:00:03

Under some circumstances, admins may be willing to change this order. SoodarOS Tuning Profiles supports static interface order based on MAC address. Interfaces without a static order use the explained order. For instance, in the above example admin sets the order of MAC address 0c:61:0c:83:00:00 as 2. The resulting interface list is:

Interfaces’ list( reordered)
PCI Address	Order	Interface Name	MAC Address
00:04.0	2	ge2	0c:61:0c:83:00:00
00:05.0	0	ge0	0c:61:0c:83:00:01
00:06.0	1	ge1	0c:61:0c:83:00:02
00:07.0	3	ge3	0c:61:0c:83:00:03

CLI Commands

Adding/ Removing tuning profile

system tune profile TPROF: Create and Enter Tuning Profile configuration

no system tune profile TPROF: Removes a Tuning profile.

Note

Removing current profile does not apply default configuration to the system.

soodar(config)# system tune profile max-perf
soodar(tune-profile)#

Configuring Data-plane

data-plane: Enter Data-plane configuration node.

Configuring Data-plane CPU

cpu main [exclusive] (1-256): Assign entered CPU core as VPP main thread core.
If exclusive keyword is used, isolate the CPU core from the OS scheduler.

no cpu main [[exclusive] (1-256)]: Change current assigned CPU to the main thread to default( Shared CPU 0).

cpu worker [exclusive] (1-256)...: Add a list of CPU cores to be used as VPP worker threads.
If exclusive keyword is used, isolate CPU cores from the OS scheduler.

no cpu worker [[exclusive] (1-256)]: Remove a list of CPU cores from VPP worker threads.
If command is used without any input, remove all VPP worker threads.

cpu weight (1-10000): Set the service CPU time weight. Useful when CPU cores are shared between the OS scheduler and the service.

no cpu weight [(1-10000)]: Change the service CPU time weight to the default value of 100.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# cpu main exclusive 1
soodar(tune-dp-cfg)# cpu worker exclusive 2 3 4 5
soodar(tune-dp-cfg)# cpu worker 6 7

Configuring Data-plane Memory

memory max SIZE: Limit the service maximum available memory to SIZE value. SIZE can be in bytes, human-readable format, or a percentage of the system’s total memory.

no memory max [SIZE]: Remove the service memory usage limit.

memory heap main SIZE: Set the VPP main heap size to SIZE value.

no memory heap main [SIZE]: Reset the VPP main heap size to the default value of 1GB.

memory heap stats SIZE: Set the VPP stats heap size to SIZE value.

no memory heap stats [SIZE]: Reset the VPP stats heap size to the default value of 32MB.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# memory heap main 2G
soodar(tune-dp-cfg)# memory heap stats 64M
soodar(tune-dp-cfg)# memory max 4G

Configuring Data-plane Buffers

memory packet-buffer size (2048-65536): Set the packet buffer’s data segment size.

no memory packet-buffer size [(2048-65536)]: Reset the packet buffer’s data segment size to the default value of 2048.

memory packet-buffer count (16384-1049776): Set the data-plane packet buffers’ count.

no memory packet-buffer count [(16384-1049776)]: Reset the data-plane packet buffers’ count to the default value of 16384

Warning

Changing buffer values can affect performance, and default values are the optimized values for most use cases.

Configuring Data-plane Poll Sleep

poll sleep (0-10000): Set fixed sleep between VPP’s main loop polls. The sleep is in microseconds.

no poll sleep [(0-10000)]: Disable fixed sleep between VPP’s main loop polls.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# poll sleep 100

Configuring Control-plane

control-plane: Enter Control-plane configuration node.

Configuring Control-plane CPU

cpu [exclusive] (1-256)...: Add a list of CPU cores for threads of this class’s services.
If exclusive keyword is used, isolate CPU cores from the OS scheduler.

no cpu worker [[exclusive] (1-256)]: Remove a list of CPU cores from services of this class.
If the command is used without any input, remove all CPUs.

cpu weight (1-10000): Set the service CPU time weight. Useful when CPU cores are shared between the OS scheduler and the service.

no cpu weight [(1-10000)]: Change the service CPU time weight to the default value of 100.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-cp-cfg)# cpu exclusive 8
soodar(tune-cp-cfg)# cpu 9 10
soodar(tune-cp-cfg)# cpu weight 1000

Configuring Control-plane Memory

memory max SIZE: Limit the service maximum available memory to SIZE value. SIZE can be in bytes, human-readable format, or a percentage of the system’s total memory.

no memory max [SIZE]: Remove the service memory usage limit.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-cp-cfg)# memory max 16G

Configuring Management Plane

management-plane: Enter Management-plane configuration node.

Configuring Management-plane CPU

cpu [exclusive] (1-256)...: Add a list of CPU cores for threads of this class’s services.
If exclusive keyword is used, isolate CPU cores from the OS scheduler.

no cpu worker [[exclusive] (1-256)]: Remove a list of CPU cores from services of this class.
If the command is used without any input, remove all CPUs.

cpu weight (1-10000): Set the service CPU time weight. Useful when CPU cores are shared between the OS scheduler and the service.

no cpu weight [(1-10000)]: Change the service CPU time weight to the default value of 100.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-mgmt-cfg)# cpu exclusive 11

Configuring management-plane Memory

memory max SIZE: Limit the service maximum available memory to SIZE value. SIZE can be in bytes, human-readable format, or a percentage of the system’s total memory.

no memory max [SIZE]: Remove the service memory usage limit.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-mgmt-cfg)# memory max 1G

Applying tuning profile

After creating a tuning profile, one can apply this profile to SoodarOS by using the following commands.

system tune apply PROFILE: Apply PROFILE to SoodarOS. To changes take effect, a machine restart is needed.

system tune apply default: Apply default profile to SoodarOS. To changes take effect, a machine restart is needed.

Tips

CPU 0 is always used by the SoodarOS kernel. Try to assign services to other cores.
It is best to use exclusive CPU cores for data-plane’s threads( main or workers).
One or 2 cores should be enough for control-plane and management-plane services, even they can be shared with the system kernel.
Memory usage usually is not a concern, and limiting the memory usage could be skipped.
Put all cores of your router to use!