System
System Logging
SoodarOS uses systemd-journald
as the central logging solution.
- debug service snmp
Enable logging for SNMP service. All SNMP logs appear in journald.
- debug service mender
The command is used to enable debug logging for the Mender service on a network device. Mender is an over-the-air (OTA) software updater for devices. When the debug logging is enabled using this command, it will display detailed information about the communication between the Mender client and server on journald.
- debug service ntpd
Enable logging for NTP service. All NTP logs appear in journald.
- debug service dhcp4
Enable logging for DHCP4 server service. All logs appear in journald.
- debug dplane fib
Enable data plane( VPP) FIB logs.
- debug dplane ipsec
Enable data plane( VPP) IPSec logs.
- log rotate max-file-size SIZE
set the limit of how sizeable individual journal files may grow at most. When a limit is reached, it rotates to the next journal file.
SIZE
: specifies the maximum size of the log file, and can be specified in bytes, kilobytes (K), megabytes (M), or gigabytes (G).
Note
Default value for the maximum file size is 10M.
- log rotate max-files (1-1000)
The command is used in devices to set the maximum number of archived log files.
(1-1000)
: This specifies the maximum number of log files that can be created. The value must be within the range of 1 to 1000.
For example, if you want to limit the number of log files to 10, you can use the following command:
soodar(config)# log rotate max-files 10
This command will ensure that only 10 log files are kept at any given time, and older log files will be overwritten as new ones are created.
Note
Default value for the maximum number of log files is 40.
- log rotate max-use <SIZE>
Control how much disk space the journal may use up at most. The size is capped at 4G. After reaching the limit, it starts removing elder journal files.
SIZE
: specifies the maximum amount of disk space that can be used by log files in kilobytes, megabytes, or gigabytes. The format for specifying size is [number][unit], where the unit can be K (kilobytes), M (megabytes), or G (gigabytes).
For example, to configure the maximum size of log files to 500 MB, the following command can be used:
soodar(config)# log rotate max-use 500M
Once the specified maximum size is reached, the router automatically rotates the log files, starting with the oldest file. This helps to prevent log files from consuming too much disk space and causing issues with router performance.
Note
Default value for the maximum disk space usage of log files is 30% of the disk.
- log rotate max-file-life (1-1000)
The maximum time( in days) to store entries in a single journal file before rotating to the next one.
(1-1000)
: is the maximum number of days that a log file will be kept before it is rotated.
For example, the following command sets the maximum file life for log files to 7 days:
soodar(config)# log rotate max-file-life 7
This means that log files will be rotated and archived after 7 days of being created, even if they have not reached their maximum file size.
Note
Default value for log files life is 30 days.
- log rotate max-retention (1-1000)
The maximum time( in days) to store journal entries. This controls whether journal files containing entries older than the specified period are deleted.
(1-1000)
: is the number of days to retain log files for.
- log syslog [LEVEL]
Enable logging output to syslog. If the optional second argument specifying the logging level is not present, the default logging level (typically debugging, but can be changed using the deprecated
log trap
command) is used. Theno
form of the command disables logging to syslog. Default log level for syslog is set toerror
level.
- log syslog [X:X::X:X|A.B.C.D|HOST] tcp [tls [skip-host-verify]] [port (100-65535)]
The command is used to configure the router or switch to send system log messages to a remote syslog server over a TCP connection with optional Transport Layer Security (TLS) encryption.
X:X::X:X|A.B.C.D|HOST
: This parameter specifies the IP address or hostname of the remote syslog server.tcp
: This parameter specifies that the log messages should be sent over a TCP connection.tls
: This optional parameter specifies that the connection should be secured with TLS encryption.skip-host-verify
: This optional parameter specifies that the hostname of the remote syslog server should not be verified when using TLS.port (100-65535)
: This parameter specifies the TCP port number on which the remote syslog server is listening for syslog messages. The default port number for syslog over TCP is 514.
Example:
soodar(config)# ip host logServer 1.1.1.1 soodar(config)# log syslog logServer tcp tls port 6514
This command configures the router to send syslog messages over a TCP connection with TLS encryption to the remote syslog server at IP address 192.168.1.100 on port number 6514.
- log syslog [HOST] loki [skip-host-verify] [port (100-65535)]
The command is used to configure the router or switch to send system log messages to a remote Loki server.
X:X::X:X|A.B.C.D|HOST
: This parameter specifies the IP address or hostname of the Loki server.
skip-host-verify
: This optional parameter specifies that the hostname of the Loki server should not be verified when using TLS.port (100-65535)
: This parameter specifies the TCP port number on which the Loki server is listening for syslog messages. The default port number for syslog over TCP is 514.
Note
Loki connection uses
http
orhttps
protocols to communicate. User must provide thehttp
orhttps
in address.Note
Port is a different option. User must not provide a port in an address like
http://temp.ir:3100
. It’s wrong!Example:
soodar(config)# log syslog https://192.168.1.1 loki skip-host-verify port 3100
- log monitor [LEVEL]
Enable logging output to terminal shell. By default, monitor logging is enabled at the informational level, but this command can be used to change the monitor logging level. If the optional second argument specifying the logging level is not present, the default logging level (typically informational) is used. The
no
form of the command disables logging to terminal monitors.
- log facility [FACILITY]
This command changes the facility used in syslog messages. The default facility is
daemon
. Theno
form of the command resets the facility to the defaultdaemon
facility.
- log record-priority
To include the severity in all messages logged to a file. Use the
log record-priority
global configuration command. To disable this option, use theno
form of the command. By default, the severity level is not included in logged messages.
- log timestamp precision [(0-6)]
This command sets the precision of log message timestamps to the given number of digits after the decimal point. Currently, the value must be 0 to 6 (i.e., the maximum precision is microseconds). To restore the default behavior (1-second accuracy), use the
no
form of the command, or set the precision explicitly to 0.log timestamp precision 3
In this example, the precision is set to provide timestamps with millisecond accuracy.
- log commands
This command enables the logging of all commands typed by a user to all enabled log destinations.
- show log all [follow]
Show all journals logs. If the
follow
mode is enabled, it follows the updates.
- show log mender [follow]
Show mender update service logs. If the
follow
mode is enabled, it follows the updates.
- show log ssh [follow]
Show SSH service logs. If the
follow
mode is enabled, it follows the updates.
- show log soolog [follow]
Show Soodar service logs. We are using vector for logging. If the
follow
mode is enabled, it follows the updates.
- show log snmpd [follow]
Show SNMP service logs. If the
follow
mode is enabled, it follows the updates.
- show log ntpd [follow]
Show NTP service logs. If the
follow
mode is enabled, it follows the updates.
- show log vpp [follow]
Show VPP service( data plane) logs. If the
follow
mode is enabled, it follows the updates.
- show log frr [follow]
Show FRR service( control plane) logs. If the
follow
mode is enabled, it follows the updates.
- show log ipsec [follow]
Show IPSec service logs. If the
follow
mode is enabled, it follows the updates.
- show log kernel [follow]
Show kernel and boot logs. If the
follow
mode is enabled, it follows the updates.
- show log ppp [follow]
Show PPD process logs. If the
follow
mode is enabled, it follows the updates.
- clear log [syslog]
Clear all generated logs. Using the
syslog
keyword makes the journald logs vacuumed; otherwise, the log file is truncated.
System update
SoodarOS uses mender
as its system update solution. It supports both online and offline updates, and in case of failure, it can roll back to the previous version
Online update
Update the system from a server. It is disabled by default. When an online update is enabled, the system automatically checks the server for available updates and installs if any are present.
Configuration
- system update enable
Enable/Disable online update
- system update server-url URL
The command is used to configure the URL of the update server on the device.
URL
: represents the URL of the update server where the software image is located. This URL should be reachable from the device for the update to be successful. The update server may be located on a local network or on the internet.
Note
Update server address, should be a URL, and an IP address can’t be set.
- system update update-poll-interval (5-2147483647)
The command is used to configure the interval at which a device polls the software update server for available software updates.
(5-2147483647)
: specifies the number of seconds between each poll.
When this command is configured, the device will periodically contact the software update server to check for available software updates. If an update is available, the device will download and install the update.
Note
Default update poll interval is 120 seconds.
- system update inventory-poll-interval (5-2147483647)
Inventory polling is a process that retrieves and updates the device inventory information such as software and hardware components, serial numbers, firmware versions, and other details and sends them to the update server. By configuring the inventory-poll-interval command, administrators can customize the frequency of inventory polling based on their requirements.
(5-2147483647)
: specifies the number of seconds between each poll.
Setting a shorter polling interval can provide more up-to-date inventory information but may consume more device resources, while setting a longer interval can conserve resources but may result in outdated inventory information.
Note
Default inventory poll interval is 150 seconds.
Example:
soodar(config)# system update enable soodar(config)# system update server-url https://update.soodar.ir soodar(config)# system update update-poll-interval 300 soodar(config)# system update inventory-poll-interval 400
Offlline update
Update system from removable storage. The procedure to offline update is simple. One need to:
Install an update
Reboot
Commit the update( to make it persistent) or rollback the update( in case of failure. Reboot without a commit to rollback)
Note
To use offline update, the online update should be disabled
Configuration
- system update offline list
List available updates on removable storage
Example:
n1(config)# system update offline list 1 rls-20 2 rls-21 3 rls-21.1
- system update offline install ARTIFACT
Install update from removable storage. ARTFICAT is the relative path of update file from removable storage root, without
.mender
postfix
- system update offline commit
Commit latest installed update.
Warning
During the system’s booting, no removable storage should be plugged into the router device, or the boot fails.
System backup and restore
The router is equipped with a set of backup/restore tools. Users can choose to create snapshots from running-config or startup-config. But backups can only be restored to startup-config.
A backup snapshot contains: * The router configuration( including tuning profiles and current tune profile). This configuration could be based on running-config or startup-config. * Private keys and imported CAs/Certificates. There’s an option to exclude this data. * Router’s licensing materials( license, license request and router license keys).
Each snapshot is saved with a unique user-provided tag. The same tag is used to restore the snapshot. The snapshots could be stored in two ways:
To remote host and via SFTP
To local storage
The snapshots are arranged in repositories of snapshots for a finer grained control. The user can define as many repo as possible in both local and remote sites.
Note
The default repository is router-backup.
Note
For keeping integrity and privacy, Each repository is encrypted with the user-provided password and should not be tampered with.
Warning
Backing up private keys to a remote host is ill-advised and should be avoided but if it’s needed, consider further safety measures for remote snapshots and their accessibilities.
Commands
- copy <system:startup-config|system:running-config> <sftp:|backup:> [no-pki]
Create a snapshot from the current startup-config and save it to a remote host. the sftp: URI could contain the username, password and address of the remote computer with the path in remote, repository name and snapshot tag. The system: URI contains the repo’s name and snapshot tag. If URI is provided, all fields are shown to the user for confirmation; Otherwise, the user is asked for the required information. The no-pki option disables backing-up private keys and imported CAs/Certificates.
Note
sftp URI is: sftp:[user]:[password]@[host]:[path]/[repo]:[repo-password]/[tag].
Note
backup URI is: backup:[tag].
Examples:
soodar# ! copy startup-config with full URI soodar# copy system:startup-config sftp:john:1234@test:/data/backups/router-1:9876/backup1 Address or name of remote host [test]? Remote host user [john]? Remote host password [*****]? Repository path [/data/backups]? Repository name [router-1]? Repository password [*****]? Destination tag [backup1]? soodar# ! copy startup-config without providing password in URI soodar# copy system:startup-config sftp:john@192.168.1.2:backup2 Address or name of remote host [192.168.1.2]? Remote host user [john]? Remote host password [admin]? Repository path [/home/john]? Repository name [router-backup]? Repository password []? Destination tag [backup2]? soodar# ! copy startup-config with providing only repo and tag name soodar# copy system:startup-config sftp:backups/backup3 Address or name of remote host [192.168.1.1]? Remote host user [admin]? Remote host password [admin]? Repository path [/home/admin]? Repository name [backups]? Repository password []? Destination tag [backup3]? soodar# ! copy startup-config without providing anything soodar# copy system:startup-config sftp: Address or name of remote host [192.168.1.1]? Remote host user [admin]? Remote host password [admin]? Repository path [/home/admin]? Repository name [router-backup]? Repository password []? Destination tag [router-config]? soodar# ! copy to local backup storage soodar# copy system:running-config backup: Destination tag [router-config]? backup4
- copy <sftp:|backup:> <system:startup-config> [no-license]
Restore a snapshot from the provided source. The no-license option disables restoring licensing materials.
Note
restored snapshot takes effect after rebooting the device.
soodar# copy backup: system:startup-config Tag to restore [router-config]? backup4
- show archive snapshots [sftp:|backup:]
List available snapshots in the source.
soodar# show archive snapshots backup: Repository name [router-backup]? Repository password []? Tag Host Time ============================================== r1 soodar Wed Jun 15 14:07:45 2022 ---------------------------------------------- keybackup1 soodar Fri Jun 24 14:13:22 2022 ---------------------------------------------- backup4 soodar Sun Jul 3 14:50:37 2022
- show archive config <sftp:|backup:>
Show snapshot contents. only config snapshots can be shown.
soodar# show archive config backup:admin-1:1234 Repository name [admin-1]? Repository password [*****]? Destination tag [router-config]? r1 r1 == hostname soodar no ipv6 forwarding no zebra nexthop kernel enable security passwords min-length 8 log syslog errors log monitor no banner motd ! no ntp ! interface ge1 no ip address ! interface ge2 no ip address ! interface ge3 no ip address ! interface lo no ip address ! interface ge0 no shutdown ip address 192.168.1.55/24 exit ! end
- show archive config differences <system:startup-config|system:running-config|sftp:|backup:> <system:startup-config|system:running-config|sftp:|backup:>
- delete <backup:|sftp:>
Delete snapshot from provided source
soodar# delete sftp: Address or name of remote host [192.168.1.1]? 192.168.1.2 Remote host user [admin]? john Remote host password [admin]? Repository name [router-backup]? Repository password []? Destination tag [router-config]? backup3
Prometheus Monitoring
SoodarOS supports both SNMP and Prometheus for monitoring purposes. Users can enable Prometheus monitoring by running soomon service on the router. After running and enabling soomon service, the router can provide metrics on port 9200.
- system service enable soomon
Start soomon service to provide Prometheus monitoring.
Note
Currently, soomon only works on port 9200. This behavior could change in the future.
System Services
Services are running in the background for accomplishing tasks. These services include:
NTP: Network Time Protocol service.
Mender: System update service.
Soolog: Remote and local syslog service.
SNMPD: SNMP Services
VPP: Router service. Restarting this service is like restarting the router.
soomon: Soodar Prometheus monitoring service.
Commands
- show system service status SERVICE
Show service status based on the output of the systemd.
- system service restart SERVICE
Restart a service. If the service is not running, starts the service.
Note
An explicitly disabled service can not be restarted( for example, when a user has set
no ntp
command, one can not restart the NTP service).
System Security
The admin can set the maximum TCP SYN limit to protect the system from SYN flood attacks.
- tcp syn-flood limit (1-4294967295)
The command is used in devices to configure the maximum number of allowed half-open TCP connections. Half-open connections occur during a SYN flood attack, which is a type of denial of service (DoS) attack. During a SYN flood attack, an attacker sends a large number of TCP SYN packets to the target device, but never completes the connection by sending the ACK packet.
(1-4294967295)
: specifies the maximum number of allowed half-open TCP connections.
Note
The default limit is
256
.