System Tune

SoodarOS consist of services working together. These services can be categorized into three classes:

  • Data-plane Services

  • Control-plane Services

  • Security-plane services

  • Management Services

  • System services

Data-plane Services

Data-plane services are responsible for forwarding, routing, and filtering incoming packets.

Members

  • VPP: VPP is a fast, scalable layer 2-4 multi-platform network stack.

Control-plane Services

All services related to signaling and routing protocols are in this class.

Members

  • FRRouting: FRRouting (FRR) is a free and open-source Internet routing protocol suite.

Security-plane Services

All services related to IPSec secure tunnels are in this class.

Members

  • Strongswan: Strongswan is an open-source IPsec-based VPN solution for secure communication over networks.

Management Services

Services responsible for connecting from outside to SoodarOS or managing internal services.

Members

  • SooSLA: SooSLA is the underlying service that does the IP SLA scheduling and running

  • Sooweb: The web server for Soodar( used in Amnet product only).

System services

All other services that are used to make the system running are under this class.

Tweaks

Depending on each use case’s needs, admins can tweak services and their resources. These tweaks are defined as Tuning profiles.

Common tweaks

For all above three categories of services, the following parameters can be set:

  • CPU cores: Admin can specify CPU cores that the service is allowed to utilize. This core pinning is done in two ways:

    • Exclusive: The core is only assigned to this service and is not used in OS Scheduler.

    • Shared: The core lists in OS Scheduler and other processes can use them.

    Note

    For data-plane services, CPU core assigning is done differently. refer to Data-plane tweaks

  • CPU weight: In case of shared CPUs, The available CPU time is split up among all services relative to their CPU time weight. A higher weight means more CPU time, a lower weight means less. The allowed range is 1 to 10000. Defaults to 100 for all processes.

  • Maximum memory: Specify the absolute limit on memory usage of the executed processes in this class.

Hugepages

The Hugepages subsystem is responsible for managing and configuring the usage of hugepages in the system. Hugepages are larger memory pages compared to regular pages, typically ranging from 2MB to 1GB in size. These larger pages can improve system performance by reducing the overhead associated with managing a large number of smaller pages.

Note

The hugepages configured in this subsystem are currently utilized by the data-plane component exclusively.

Parameters

The Hugepages subsystem provides the following configuration parameters:

  • Hugepages Size: The hugepages size parameter specifies the size of each hugepage. It determines the amount of memory allocated for each hugepage. Supported sizes may vary depending on the system architecture, but typical sizes include 2MB and 1GB.

  • Hugepages Number: The hugepages number parameter specifies the total number of hugepages to be allocated in the system. It determines the overall amount of memory reserved for hugepages. The number of hugepages allocated should be carefully determined based on the system’s memory requirements and workload characteristics.

Data-plane tweaks

VPP is a sophisticated and high-performance network that has lots of customizable options.

Data-plane CPU configuration

VPP needs at least one main thread and more optional worker threads. Each of These threads is pinned to a CPU core. The default configuration uses core 0 as the main thread( in shared mode) and no workers. Admin can set main thread/ worker threads, CPU exclusiveness, and service CPU weight.

Data-plane Memory configuration

VPP uses its memory management system, so in the beginning, it mmap needed memories from OS heap.

  • Main heap: VPP’s main memory. Used to store ACL rules, IP Routes,… . Defaults to 1GB.

  • Stat heap: Memory assigned to storing stats( like packets count). Defaults to 32MB.

Admin can change these values when needed.

Data-plane buffer configuration

VPP uses per NUMA pre-allocated buffers to process incoming packets. It’s advised not to change default values, but admins can change the number of buffers and buffer size.

Data-plane poll sleep

VPP’s main loop poll always fetches new packets and uses 100% CPU. An admin can add a fixed sleep between the main loop poll to lower the power usage, CPU usage, and heat production in small devices.

Warning

Enabling poll sleep can cause performance degradation.

Interface order configuration

Network interfaces are sorted by their PCI address. For example, a router with 4 network interfaces and their respective addresses is mapped like:

Interfaces’ list( original)

PCI Address

Order

Interface Name

MAC Address

00:04.0

0

ge0

0c:61:0c:83:00:00

00:05.0

1

ge1

0c:61:0c:83:00:01

00:06.0

2

ge2

0c:61:0c:83:00:02

00:07.0

3

ge3

0c:61:0c:83:00:03

Under some circumstances, admins may be willing to change this order. SoodarOS Tuning Profiles supports static interface order based on MAC address. Interfaces without a static order use the explained order. For instance, in the above example, admin sets the order of MAC address 0c:61:0c:83:00:00 as 2. The resulting interface list is:

Interfaces’ list( reordered)

PCI Address

Order

Interface Name

MAC Address

00:04.0

2

ge2

0c:61:0c:83:00:00

00:05.0

0

ge0

0c:61:0c:83:00:01

00:06.0

1

ge1

0c:61:0c:83:00:02

00:07.0

3

ge3

0c:61:0c:83:00:03

Auto-generated tuning profiles

Auto-generated tuning profiles are pre-configured profiles designed to optimize system performance and resource allocation based on the system’s hardware configuration. These profiles provide a balanced and power-saving configuration by assigning CPU cores to the data-plane and setting memory limits for the system and adjusting data-plane memory usage.

CLI Commands

Adding/ Removing tuning profile

system tune profile TPROF

Create and Enter Tuning Profile configuration

  • TPROF: specifies the tuning profile name.

Note

Removing current profile does not apply default configuration to the system.

soodar(config)# system tune profile max-perf
soodar(tune-profile)#

Configuring Hugepages

hugepages (2-100000)

The command is used to configure the number of hugepages allocated in the system.

  • (2-100000): Specifies the number of hugepages to be allocated. The valid range is from 2 to 100,000.

hugepages size <2|1024>

The command is used to configure the size of hugepages in the system.

  • <2|1024>: Specifies the size of the hugepages. You can choose either 2 or 1024, which represents the page size in megabytes.

Configuring Data-plane

data-plane

Enter the Data-plane configuration node.

Configuring Data-plane CPU

cpu main [exclusive] (1-256)

Assign entered CPU core as VPP main thread core.

  • exclusive: Specifies if the CPU core should be isolated from the OS scheduler.

  • (1-256): CPU core number.

Note

CPU Core 0 is reserved for OS kernel.

no cpu main [[exclusive] (1-256)]

Change current assigned CPU to the main thread to default( Shared CPU 0).

cpu worker [exclusive] (1-256)...

Add a list of CPU cores to be used as VPP worker threads.

  • exclusive: Specifies if the CPU core should be isolated from the OS scheduler.

  • (1-256): CPU core number.

no cpu worker [[exclusive] (1-256)]

Remove a list of CPU cores from VPP worker threads. If the command is used without any input, remove all VPP worker threads.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# cpu main exclusive 1
soodar(tune-dp-cfg)# cpu worker exclusive 2 3 4 5
soodar(tune-dp-cfg)# cpu worker 6 7

Configuring Data-plane Memory

memory heap main SIZE

Set the VPP main heap size.

  • SIZE: specifies the VPP main heap size. can be in bytes or human-readable format.

Note

Default value for main heap size is 1GB.

no memory heap main [SIZE]

Reset the VPP main heap size to the default value of 1GB.

memory heap stats SIZE

Set the VPP stats heap size to SIZE value.

  • SIZE: specifies the VPP stats heap size. can be in bytes or human-readable format.

Note

Default value for stats heap size is 32MB.

no memory heap stats [SIZE]

Reset the VPP stats heap size to the default value of 32MB.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# memory heap main 2G
soodar(tune-dp-cfg)# memory heap stats 64M
soodar(tune-dp-cfg)# memory max 4G

Configuring Data-plane Buffers

memory packet-buffer size (2048-65536)

Set the packet buffer’s data segment size.

  • (2048-65536): specifies the packet buffer’s data segment size.

Note

Default value for packet buffer’s data segment size is 2048.

Warning

Packet buffer’s size is carefully chosen and changing it is not advised.

no memory packet-buffer size [(2048-65536)]

Reset the packet buffer’s data segment size to the default value of 2048.

memory packet-buffer count (16384-1049776)

Set the data-plane packet buffers’ count.

  • (16384-1049776): specifies the number of buffers to be used for packets.

Note

Default packet buffers’ count is 16384.

Note

Each network interface consumes about 2000 packet buffer. As a rule of thumb, to calculate the needed packet buffers one can use 2000 * Number of interfaces + 1000.

no memory packet-buffer count [(16384-1049776)]

Reset the data-plane packet buffers’ count to the default value of 16384

Configuring Data-plane Poll Sleep

poll sleep (0-10000)

Set fixed sleep between VPP’s main loop polls. The sleep is in microseconds.

no poll sleep [(0-10000)]

Disable fixed sleep between VPP’s main loop polls.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# data-plane
soodar(tune-dp-cfg)# poll sleep 100

Configuring Control-plane

control-plane

Enter Control-plane configuration node.

Configuring Security Plane

security-plane

Enter security-plane configuration node.

Configuring Management Plane

management-plane

Enter Management-plane configuration node.

Configuring System services

system

Enter system services configuration node.

Common CPU Configuration

cpu [exclusive] (1-256)...

Add a list of CPU cores for threads of this class’s services.

  • exclusive: Specifies if the CPU core should be isolated from the OS scheduler.

  • (1-256): CPU core number.

no cpu [[exclusive] (1-256)]

Remove a list of CPU cores from services of this class. If the command is used without any input, remove all CPUs.

cpu weight (1-10000)

The cpu weight command is used to adjust the CPU usage priority for a process or group of processes. Useful when CPU cores are shared between the OS scheduler and the service.

  • (1-10000): specifies the weight of the process or group of processes, and it must be a number between 1 and 10000. A higher weight means higher priority, and a weight of 1 means the lowest priority.

When using this command, it is important to note that it does not guarantee a certain amount of CPU time to the process or group of processes. Instead, it adjusts the relative priority among them. Therefore, if the system is heavily congested, the processes with higher weights may still be starved of CPU time.

Note

Default weight value is 100

no cpu weight [(1-10000)]

Change the service CPU time weight to the default value of 100.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-cp-cfg)# cpu exclusive 8
soodar(tune-cp-cfg)# cpu 9 10
soodar(tune-cp-cfg)# cpu weight 1000

Common Memory Configuration

memory max SIZE

Limit the service’s maximum available memory.

  • SIZE: specifies the service’s maximum available memory. SIZE can be in bytes, human-readable format, or a percentage of the system’s total memory.

soodar(config)# system tune profile max-perf
soodar(tune-profile)# control-plane
soodar(tune-cp-cfg)# memory max 16G

Applying tuning profile

After creating a tuning profile, one can apply this profile to SoodarOS by using the following commands.

system tune apply PROFILE

The command is used to apply a tuning profile to SoodarOS. To changes take effect, a machine restart is needed.

  • PROFILE: is the name of the tuning profile to be applied.

system tune apply default

The command is used to apply the default profile to SoodarOS. To changes take effect, a machine restart is needed.

Note

On the first boot of the device, 2 balanced and powersaving profiles are generated and are ready to be used( But none of them is applied).

Tips

  • CPU 0 is always used by the SoodarOS kernel. Try to assign services to other cores.

  • It is best to use exclusive CPU cores for the data plane’s threads( main or workers).

  • One or 2 cores should be enough for control-plane and management-plane services, they can even be shared with the system kernel.

  • Memory usage usually is not a concern, and limiting memory usage could be skipped.

  • Put all cores of your router to use!