System

System Logging

SoodarOS uses systemd-journald as the central logging solution.

debug service snmp

Enable logging for SNMP service. All SNMP logs appear in journald.

debug service mender

The command is used to enable debug logging for the Mender service on a network device. Mender is an over-the-air (OTA) software updater for devices. When the debug logging is enabled using this command, it will display detailed information about the communication between the Mender client and server on journald.

debug service ntpd

Enable logging for NTP service. All NTP logs appear in journald.

debug service dhcp4

Enable logging for DHCP4 server service. All logs appear in journald.

debug dplane fib

Enable data plane( VPP) FIB logs.

debug dplane ipsec

Enable data plane( VPP) IPSec logs.

log rotate max-file-size SIZE

set the limit of how sizeable individual journal files may grow at most. When a limit is reached, it rotates to the next journal file.

  • SIZE: specifies the maximum size of the log file, and can be specified in bytes, kilobytes (K), megabytes (M), or gigabytes (G).

Note

Default value for the maximum file size is 10M.

log rotate max-files (1-1000)

The command is used in devices to set the maximum number of archived log files.

  • (1-1000): This specifies the maximum number of log files that can be created. The value must be within the range of 1 to 1000.

For example, if you want to limit the number of log files to 10, you can use the following command:

soodar(config)# log rotate max-files 10

This command will ensure that only 10 log files are kept at any given time, and older log files will be overwritten as new ones are created.

Note

Default value for the maximum number of log files is 40.

log rotate max-use <SIZE>

Control how much disk space the journal may use up at most. The size is capped at 4G. After reaching the limit, it starts removing elder journal files.

  • SIZE: specifies the maximum amount of disk space that can be used by log files in kilobytes, megabytes, or gigabytes. The format for specifying size is [number][unit], where the unit can be K (kilobytes), M (megabytes), or G (gigabytes).

For example, to configure the maximum size of log files to 500 MB, the following command can be used:

soodar(config)# log rotate max-use 500M

Once the specified maximum size is reached, the router automatically rotates the log files, starting with the oldest file. This helps to prevent log files from consuming too much disk space and causing issues with router performance.

Note

Default value for the maximum disk space usage of log files is 30% of the disk.

log rotate max-file-life (1-1000)

The maximum time( in days) to store entries in a single journal file before rotating to the next one.

  • (1-1000): is the maximum number of days that a log file will be kept before it is rotated.

For example, the following command sets the maximum file life for log files to 7 days:

soodar(config)# log rotate max-file-life 7

This means that log files will be rotated and archived after 7 days of being created, even if they have not reached their maximum file size.

Note

Default value for log files life is 30 days.

log rotate max-retention (1-1000)

The maximum time( in days) to store journal entries. This controls whether journal files containing entries older than the specified period are deleted.

  • (1-1000): is the number of days to retain log files for.

log syslog [LEVEL]

Enable logging output to syslog. If the optional second argument specifying the logging level is not present, the default logging level (typically debugging, but can be changed using the deprecated log trap command) is used. The no form of the command disables logging to syslog. Default log level for syslog is set to error level.

log syslog [X:X::X:X|A.B.C.D|HOST] tcp [tls [skip-host-verify]] [port (100-65535)]

The command is used to configure the router or switch to send system log messages to a remote syslog server over a TCP connection with optional Transport Layer Security (TLS) encryption.

  • X:X::X:X|A.B.C.D|HOST: This parameter specifies the IP address or hostname of the remote syslog server.

  • tcp: This parameter specifies that the log messages should be sent over a TCP connection.

  • tls: This optional parameter specifies that the connection should be secured with TLS encryption.

  • skip-host-verify: This optional parameter specifies that the hostname of the remote syslog server should not be verified when using TLS.

  • port (100-65535): This parameter specifies the TCP port number on which the remote syslog server is listening for syslog messages. The default port number for syslog over TCP is 514.

Example:

soodar(config)# ip host logServer 1.1.1.1
soodar(config)# log syslog logServer tcp tls port 6514

This command configures the router to send syslog messages over a TCP connection with TLS encryption to the remote syslog server at IP address 192.168.1.100 on port number 6514.

log syslog [HOST] loki [skip-host-verify] [port (100-65535)]

The command is used to configure the router or switch to send system log messages to a remote Loki server.

  • X:X::X:X|A.B.C.D|HOST: This parameter specifies the IP address or hostname of the Loki server.

  • skip-host-verify: This optional parameter specifies that the hostname of the Loki server should not be verified when using TLS.

  • port (100-65535): This parameter specifies the TCP port number on which the Loki server is listening for syslog messages. The default port number for syslog over TCP is 514.

Note

Loki connection uses http or https protocols to communicate. User must provide the http or https in address.

Note

Port is a different option. User must not provide a port in an address like http://temp.ir:3100. It’s wrong!

Example:

soodar(config)# log syslog https://192.168.1.1 loki skip-host-verify port 3100
log monitor [LEVEL]

Enable logging output to terminal shell. By default, monitor logging is enabled at the informational level, but this command can be used to change the monitor logging level. If the optional second argument specifying the logging level is not present, the default logging level (typically informational) is used. The no form of the command disables logging to terminal monitors.

log facility [FACILITY]

This command changes the facility used in syslog messages. The default facility is daemon. The no form of the command resets the facility to the default daemon facility.

log record-priority

To include the severity in all messages logged to a file. Use the log record-priority global configuration command. To disable this option, use the no form of the command. By default, the severity level is not included in logged messages.

log timestamp precision [(0-6)]

This command sets the precision of log message timestamps to the given number of digits after the decimal point. Currently, the value must be 0 to 6 (i.e., the maximum precision is microseconds). To restore the default behavior (1-second accuracy), use the no form of the command, or set the precision explicitly to 0.

log timestamp precision 3

In this example, the precision is set to provide timestamps with millisecond accuracy.

log commands

This command enables the logging of all commands typed by a user to all enabled log destinations.

show log all [follow]

Show all journals logs. If the follow mode is enabled, it follows the updates.

show log mender [follow]

Show mender update service logs. If the follow mode is enabled, it follows the updates.

show log ssh [follow]

Show SSH service logs. If the follow mode is enabled, it follows the updates.

show log soolog [follow]

Show Soodar service logs. We are using vector for logging. If the follow mode is enabled, it follows the updates.

show log snmpd [follow]

Show SNMP service logs. If the follow mode is enabled, it follows the updates.

show log ntpd [follow]

Show NTP service logs. If the follow mode is enabled, it follows the updates.

show log vpp [follow]

Show VPP service( data plane) logs. If the follow mode is enabled, it follows the updates.

show log frr [follow]

Show FRR service( control plane) logs. If the follow mode is enabled, it follows the updates.

show log ipsec [follow]

Show IPSec service logs. If the follow mode is enabled, it follows the updates.

show log kernel [follow]

Show kernel and boot logs. If the follow mode is enabled, it follows the updates.

show log ppp [follow]

Show PPD process logs. If the follow mode is enabled, it follows the updates.

clear log [syslog]

Clear all generated logs. Using the syslog keyword makes the journald logs vacuumed; otherwise, the log file is truncated.

System update

SoodarOS uses mender as its system update solution. It supports both online and offline updates, and in case of failure, it can roll back to the previous version

Online update

Update the system from a server. It is disabled by default. When an online update is enabled, the system automatically checks the server for available updates and installs if any are present.

Configuration

system update enable

Enable/Disable online update

system update server-url URL

The command is used to configure the URL of the update server on the device.

  • URL: represents the URL of the update server where the software image is located. This URL should be reachable from the device for the update to be successful. The update server may be located on a local network or on the internet.

Note

Update server address, should be a URL, and an IP address can’t be set.

system update update-poll-interval (5-2147483647)

The command is used to configure the interval at which a device polls the software update server for available software updates.

  • (5-2147483647): specifies the number of seconds between each poll.

When this command is configured, the device will periodically contact the software update server to check for available software updates. If an update is available, the device will download and install the update.

Note

Default update poll interval is 120 seconds.

system update inventory-poll-interval (5-2147483647)

Inventory polling is a process that retrieves and updates the device inventory information such as software and hardware components, serial numbers, firmware versions, and other details and sends them to the update server. By configuring the inventory-poll-interval command, administrators can customize the frequency of inventory polling based on their requirements.

  • (5-2147483647): specifies the number of seconds between each poll.

Setting a shorter polling interval can provide more up-to-date inventory information but may consume more device resources, while setting a longer interval can conserve resources but may result in outdated inventory information.

Note

Default inventory poll interval is 150 seconds.

Example:

soodar(config)# system update enable
soodar(config)# system update server-url https://update.soodar.ir
soodar(config)# system update update-poll-interval 300
soodar(config)# system update inventory-poll-interval 400

Offlline update

Update system from removable storage. The procedure to offline update is simple. One need to:

  1. Install an update

  2. Reboot

  3. Commit the update( to make it persistent) or rollback the update( in case of failure. Reboot without a commit to rollback)

Note

To use offline update, the online update should be disabled

Configuration

system update offline list

List available updates on removable storage

Example:

n1(config)# system update offline list
    1   rls-20
    2   rls-21
    3   rls-21.1
system update offline install ARTIFACT

Install update from removable storage. ARTFICAT is the relative path of update file from removable storage root, without .mender postfix

system update offline commit

Commit latest installed update.

Warning

During the system’s booting, no removable storage should be plugged into the router device, or the boot fails.

System backup and restore

The router is equipped with a set of backup/restore tools. Users can choose to create snapshots from running-config or startup-config. But backups can only be restored to startup-config.

A backup snapshot contains: * The router configuration( including tuning profiles and current tune profile). This configuration could be based on running-config or startup-config. * Private keys and imported CAs/Certificates. There’s an option to exclude this data. * Router’s licensing materials( license, license request and router license keys).

Each snapshot is saved with a unique user-provided tag. The same tag is used to restore the snapshot. The snapshots could be stored in two ways:

  1. To remote host and via SFTP

  2. To local storage

The snapshots are arranged in repositories of snapshots for a finer grained control. The user can define as many repo as possible in both local and remote sites.

Note

The default repository is router-backup.

Note

For keeping integrity and privacy, Each repository is encrypted with the user-provided password and should not be tampered with.

Warning

Backing up private keys to a remote host is ill-advised and should be avoided but if it’s needed, consider further safety measures for remote snapshots and their accessibilities.

Commands

copy <system:startup-config|system:running-config> <sftp:|backup:> [no-pki]

Create a snapshot from the current startup-config and save it to a remote host. the sftp: URI could contain the username, password and address of the remote computer with the path in remote, repository name and snapshot tag. The system: URI contains the repo’s name and snapshot tag. If URI is provided, all fields are shown to the user for confirmation; Otherwise, the user is asked for the required information. The no-pki option disables backing-up private keys and imported CAs/Certificates.

Note

sftp URI is: sftp:[user]:[password]@[host]:[path]/[repo]:[repo-password]/[tag].

Note

backup URI is: backup:[tag].

Examples:

soodar# ! copy startup-config with full URI
soodar# copy system:startup-config sftp:john:1234@test:/data/backups/router-1:9876/backup1
Address or name of remote host [test]?
Remote host user [john]?
Remote host password [*****]?
Repository path [/data/backups]?
Repository name [router-1]?
Repository password [*****]?
Destination tag [backup1]?
soodar# ! copy startup-config without providing password in URI
soodar# copy system:startup-config sftp:john@192.168.1.2:backup2
Address or name of remote host [192.168.1.2]?
Remote host user [john]?
Remote host password [admin]?
Repository path [/home/john]?
Repository name [router-backup]?
Repository password []?
Destination tag [backup2]?
soodar# ! copy startup-config with providing only repo and tag name
soodar# copy system:startup-config sftp:backups/backup3
Address or name of remote host [192.168.1.1]?
Remote host user [admin]?
Remote host password [admin]?
Repository path [/home/admin]?
Repository name [backups]?
Repository password []?
Destination tag [backup3]?
soodar# ! copy startup-config without providing anything
soodar# copy system:startup-config sftp:
Address or name of remote host [192.168.1.1]?
Remote host user [admin]?
Remote host password [admin]?
Repository path [/home/admin]?
Repository name [router-backup]?
Repository password []?
Destination tag [router-config]?
soodar# ! copy to local backup storage
soodar# copy system:running-config backup:
Destination tag [router-config]? backup4
copy <sftp:|backup:> <system:startup-config> [no-license]

Restore a snapshot from the provided source. The no-license option disables restoring licensing materials.

Note

restored snapshot takes effect after rebooting the device.

soodar# copy backup: system:startup-config
Tag to restore [router-config]? backup4
show archive snapshots [sftp:|backup:]

List available snapshots in the source.

soodar# show archive snapshots backup:
Repository name [router-backup]?
Repository password []?


   Tag        Host              Time
==============================================
r1           soodar   Wed Jun 15 14:07:45 2022
----------------------------------------------
keybackup1   soodar   Fri Jun 24 14:13:22 2022
----------------------------------------------
backup4      soodar   Sun Jul  3 14:50:37 2022
show archive config <sftp:|backup:>

Show snapshot contents. only config snapshots can be shown.

soodar# show archive config backup:admin-1:1234
Repository name [admin-1]?
Repository password [*****]?
Destination tag [router-config]? r1

r1
==
hostname soodar
no ipv6 forwarding
no zebra nexthop kernel enable
security passwords min-length 8
log syslog errors
log monitor
no banner motd
!
no ntp
!
interface ge1
 no ip address
!
interface ge2
 no ip address
!
interface ge3
 no ip address
!
interface lo
 no ip address
!
interface ge0
 no shutdown
 ip address 192.168.1.55/24
exit
!
end
show archive config differences <system:startup-config|system:running-config|sftp:|backup:> <system:startup-config|system:running-config|sftp:|backup:>
delete <backup:|sftp:>

Delete snapshot from provided source

soodar# delete sftp:
Address or name of remote host [192.168.1.1]? 192.168.1.2
Remote host user [admin]? john
Remote host password [admin]?
Repository name [router-backup]?
Repository password []?
Destination tag [router-config]? backup3

Prometheus Monitoring

SoodarOS supports both SNMP and Prometheus for monitoring purposes. Users can enable Prometheus monitoring by running soomon service on the router. After running and enabling soomon service, the router can provide metrics on port 9200.

system service enable soomon

Start soomon service to provide Prometheus monitoring.

Note

Currently, soomon only works on port 9200. This behavior could change in the future.

System Services

Services are running in the background for accomplishing tasks. These services include:

  • NTP: Network Time Protocol service.

  • Mender: System update service.

  • Soolog: Remote and local syslog service.

  • SNMPD: SNMP Services

  • VPP: Router service. Restarting this service is like restarting the router.

  • soomon: Soodar Prometheus monitoring service.

Commands

show system service status SERVICE

Show service status based on the output of the systemd.

system service restart SERVICE

Restart a service. If the service is not running, starts the service.

Note

An explicitly disabled service can not be restarted( for example, when a user has set no ntp command, one can not restart the NTP service).

System Security

The admin can set the maximum TCP SYN limit to protect the system from SYN flood attacks.

tcp syn-flood limit (1-4294967295)

The command is used in devices to configure the maximum number of allowed half-open TCP connections. Half-open connections occur during a SYN flood attack, which is a type of denial of service (DoS) attack. During a SYN flood attack, an attacker sends a large number of TCP SYN packets to the target device, but never completes the connection by sending the ACK packet.

  • (1-4294967295): specifies the maximum number of allowed half-open TCP connections.

Note

The default limit is 256.