Real-Time System Setup Steps

The following example assumes that you are on a machine with 4 cores, Debian trixie and that you want to run the EtherCAT Master on enp1s0.
You might need to adjust some commands if you use e.g. Ubuntu.

1. Install a Real-Time Kernel

The most important step is to install a kernel that has the CONFIG_PREEMPT_RT option set (and optionally has the PREEMPT_RT patchset applied). Most distributions offer a pre-built RT kernel. E.g. on Debian this is linux-image-rt-amd64 for amd64.

sudo apt install linux-image-rt-amd64

After installation, set the kernel as default in the grub menu or uninstall the stock kernel. Reboot and verify the running kernel with:

uname -a
# Look for "PREEMPT_RT" in the output

2. Disable the GPU if not needed

The GPU can actually stall the CPU in certain situations. To sidestep this, it is usually best to just disable the GPU if it is not needed.

This can be done with the nomodeset kernel parameter.

Kernel parameters can be set by modifying GRUB_CMDLINE_LINUX_DEFAULT. Don't forget to run update-grub after that.

3. Core Isolation and Interrupt organization

Unfortunately the network performance is often not great without further steps.
The main reason for that is, that receiving or sending a packet requires disabling bottom halves (local_bh_disable()) to prevent data races.
Since code within these sections remains preemptible, other threaded interrupt handlers (e.g. for writing to disk) or softirqs can block a high priority networking task for quite a long time (see this LWN article for details).

This was addressed in 6.18, but is still the case for e.g. the kernel from Debian trixie.

As a workaround, we just isolate a single core per networking application.
Everything that is needed for receiving and sending packets will be moved onto that CPU, and everything that is not needed will be moved to the other CPUs.
Now nothing running on this core should disable bottom halves for a long time anymore.

The following example assumes an EtherCAT Master on enp1s0, and isolates CPU 1 for this.
When you want to run multiple fieldbusses at the same time, it is advised to isolate multiple CPUs.

Core Isolation (move everything that is not required of a CPU)

We will use the following kernel parameters:

isolcpus=managed_irq,domain,1 irqaffinity=0,2,3 rcu_nocbs=1 rcu_nocb_poll

Here is a brief explanation for the different parameters:

Parameter	Explanation
`isolcpus=managed_irq,domain,1`	Isolates CPU 1. `domain` removes it from the scheduler domain so the kernel will not migrate ordinary tasks onto it. `managed_irq` additionally moves managed (auto-affinity) IRQs away from the isolated CPU.
`irqaffinity=0,2,3`	Sets the default affinity mask for all hardware interrupts to CPUs 0, 2, and 3, ensuring interrupt handlers do not run on the isolated CPU.
`rcu_nocbs=1`	Offloads RCU (Read-Copy-Update) callbacks from CPU 1 to a dedicated kthread on another CPU, preventing RCU processing from blocking real-time threads.
`rcu_nocb_poll`	Makes the RCU offload threads poll for callbacks rather than being woken by the isolated CPU, eliminating the associated inter-processor interrupts and further reducing latency on CPU 1.

See the kernel parameter documentation for further details.

This is an example for 4 cores. On Systems with more cores, the irqaffinity should be adjusted accordingly. As an alternative to the kernel parameter it is also possible to configure interrupt affinity at runtime using procfs.

Moving the main thread of the application and the NIC interrupt to the isolated CPU

The voraus applications that do low-latency networking provide a configuration option to set the CPU affinity of the main thread (which actually sends and receives the packets). For the EtherCAT Master, this is the ECAT_CPU_AFFINITY key. For the Profinet Controller it is PNET_CPU_AFFINITY. For voraus-rt-ipc-eval this is NETWORK_RTT_TEST__CPU_AFFINITY. It is important to configure the fieldbus application to use the isolated CPU. So in this example, you should set NETWORK_RTT_TEST__CPU_AFFINITY=1 to use CPU1.

However, this just covers the send part.

The receive part involves a NAPI callback which is usually run by a dedicated threaded interrupt handler or ksoftirqd. Threaded interrupt handlers / softirqs will run on the CPU on which the interrupt occurred (if not forced otherwise).

So the interrupts of the NIC that is used must be moved to the isolated CPU. This can't be done statically with a Kernel Parameter.

We usually use tuna for that.

Example:

sudo tuna show_irqs

The output might contain something like:

enp1s0-TxRx-0       0,2,3  igb
enp1s0-TxRx-1       0,2,3  igb
enp1s0-TxRx-2       0,2,3  igb
enp1s0-TxRx-3       0,2,3  igb

These interrupts can be moved to CPU 1 with the following command.

sudo tuna move -q "enp1s0*" -c 1

after that the output looks like this:

enp1s0-TxRx-0           1  igb
enp1s0-TxRx-1           1  igb
enp1s0-TxRx-2           1  igb
enp1s0-TxRx-3           1  igb

The exact output varies with the NIC model.

warning

The interrupts only show up if the network interface is not down. Verify this with ip link if there are no interrupts.

4. Further Network tuning

Most NIC models benefit from further tuning. Most notably from disabling interrupt coalescing.

For the Intel I210, this can be done with:

sudo ethtool -C enp1s0 rx-usecs 0 tx-usecs 0

The exact command depends on the driver/model in use.

5. Additional steps

Sometimes a system requires additional steps that come with a potentially higher (performance) cost.

Fixed CPU frequency

CPU frequency changes can introduce latencies. They also lead to non deterministic behavior. E.g. a cycle that usually runs for 200µs when the system is not loaded can take 300µs when the CPU is under load, because it cannot maintain its turbo frequency. To prevent that, it is possible to pin CPUs at a fixed frequency. How to do that depends on the driver in use (see CPU Performance Scaling). If this is necessary depends on the actual hardware and the cooling situation. E.g. CPUs that are not properly cooled likely need this, but CPUs that stay at e.g. 60°C under full load may not need this.

Note: idle states are disabled by the voraus applications if not configured otherwise.

SMT

Modern systems often use Hyperthreading. Hyperthreading does in general introduce latency and reduces (single core) throughput but increases overall throughput. How much impact these effects have, depends on the exact CPU architecture. In our experience it is often acceptable to leave hyperthreading on.

However, if you don't need the additional CPUs and the additional performance, or SMT introduces high latencies on your system, you may want to disable that. This can be done with the nosmt Kernel parameter or often via the bios. As an alternative, it is also often sufficient to disable the hyperthreading partner of your isolated CPU if you have one.

EFI Runtime Services

EFI provides runtime services which can for example be used to determine the boot source. Unfortunately some EFI implementations can have quite long runtimes (in the millisecond range). Since they may be required to run with interrupts/preemption disabled, this runtime directly contributes to the scheduling latency. Normally, EFI runtime services are disabled by default on PREEMPT_RT kernels (CONFIG_EFI_DISABLE_RUNTIME defaults to y on CONFIG_PREEMPT_RT). However, some distributions (e.g. Ubuntu) configure this to n in their realtime kernel. This can also be the case when a "self-built" kernel, which was based on a "normal" config is used, since the default is not re-evaluated. This can be mitigated with the kernel parameter efi=noruntime. Disabling EFI runtime services does of course break everything that needs them. It is also possible to pin the services to certain CPUs instead. See the kernel documentation for more information.

Doing the Previous Steps with Ansible

As already described in Real-Time System Setup, you are free to use voraus-ipc-tools-ansible for the steps above.

The following playbook does steps 1-3:

---
- name: Run IPC tools
  ansible.builtin.import_playbook: voraus.ipc_tools.example

  vars:
    core_isolation_isolated_core_ids: 1
    core_isolation_isolated_network_interfaces:
      "enp1s0": 1

1. Install a Real-Time Kernel​

2. Disable the GPU if not needed​

3. Core Isolation and Interrupt organization​

Core Isolation (move everything that is not required of a CPU)​

Moving the main thread of the application and the NIC interrupt to the isolated CPU​

4. Further Network tuning​

5. Additional steps​

Fixed CPU frequency​

SMT​

EFI Runtime Services​

Doing the Previous Steps with Ansible​