Real Time Execution in ROS 2: Priority Scheduling and Composable Nodes

Real time software must meet a timing deadline on every cycle. For these systems the worst case latency, not the average, is the figure of merit.

This blog covers two topics:

1. Running critical ROS 2 callbacks with real time priority
2. Using composable nodes to reduce communication overhead

What real time means

Consider a controller for an inverted pendulum. Its control callback must execute at a fixed period. If one cycle misses its deadline, the controller operates on stale sensor data and the system becomes unstable. For such a callback, a low average latency is not sufficient. Every execution must complete before the next deadline.

The response time of any system running on a modern computer accumulates in three places:

1. Hardware latency from sensors, buses and interrupts
2. Operating system latency from scheduling, interrupts and context switches
3. Application latency from executors, callbacks and control code

When you design the critical path, you should exclude any operation whose worst case duration is unbounded or difficult to predict. In practice you should keep file access, network access, dynamic memory allocation and blocking locks out of your critical callbacks, because each of these can stall for an unbounded time when the system is under load.

Concurrency is not priority

A multithreaded executor allows several callbacks to run concurrently, but concurrency does not establish priority. It does not grant any callback precedence over another when the CPU is contended.

Consider a state machine that must execute at 20 Hz. If you place it in the same executor and the same scheduling class (SCHED_OTHER) as logging and telemetry, the operating system scheduler is free to defer it while lower value work runs.

The design I recommend uses two executors. One executor runs the critical callbacks on a thread configured with the SCHED_FIFO real time policy. The second executor runs logging, telemetry and housekeeping callbacks at the default priority. When a critical callback becomes ready, the real time thread preempts the normal priority work.

You can assign work to the real time executor at three levels of granularity:

1. An entire critical node.
2. Selected callback groups within a single node.
3. The callback groups that form a complete multi node pipeline.

Each design below is a self contained, runnable example. To emulate a loaded production system, I used stress-ng to saturate all CPU cores for the full duration of every 60 second measurement, so the reported figures describe behavior under sustained CPU contention rather than on an idle machine.

The example used in Design 1 and Design 2

Design 1 and Design 2 share the same three nodes. The full source is in design_1.cpp and design_2.cpp.

1. The producer node publishes an integer on int_topic every 10 ms, which is 100 Hz.
2. The consumer node is the critical node. It runs one subscription callback and one 100 Hz timer callback.
3. The logger node is not critical. It runs a subscription callback and a 100 Hz timer callback.

Every callback executes a fixed busy loop that represents a known amount of work. The exact durations and the comparison of interest differ between the two designs and are stated in each section.

Design 1: Assign an entire node to the real time executor

The simplest option is to assign a complete node to the real time executor. Here the consumer node runs on the real time executor, while the producer and logger nodes run on the normal executor.

The callbacks are created in the node constructor.

sub_ = create_subscription<std_msgs::msg::Int32>(  "int_topic", 100,  std::bind(&ConsumerNode::cb, this, _1)); timer_ = create_wall_timer(  10ms,  std::bind(&ConsumerNode::timer_cb, this));

In main(), the nodes are distributed across the two executors, and the real time executor is spun on its own thread configured with SCHED_FIFO priority 90.

rclcpp::executors::SingleThreadedExecutor no_rt_executor;rclcpp::executors::SingleThreadedExecutor rt_executor; no_rt_executor.add_node(producer_node);no_rt_executor.add_node(logger_node);rt_executor.add_node(consumer_node); auto rt_thread = std::thread(  [&]() {    sched_param sch;    sch.sched_priority = 90;     if (sched_setscheduler(0, SCHED_FIFO, &sch) == -1) {      throw std::runtime_error{        std::string("failed to set scheduler: ") +        std::strerror(errno)};    }     rt_executor.spin();  }); no_rt_executor.spin();rt_thread.join();

The consumer and logger nodes are configured identically: each runs a 0.5 ms subscription callback and a 2 ms timer callback at 100 Hz. Because the two nodes perform the same work, any difference in their measured timing is attributable to scheduling priority alone.

The two timer callbacks are each implemented to execute for 2 ms. The consumer timer runs under SCHED_FIFO and holds that duration, with a 99th percentile execution time of 2.003 ms and a maximum of 2.009 ms. The logger timer performs the same 2 ms of work at the default priority, but CPU contention preempts it repeatedly, so its 99th percentile execution time rises to 4.27 ms and its maximum to 16.85 ms.

Both timers are configured for a 10 ms period, which is 100 Hz. The consumer timer maintains this period closely, with a 99th percentile of 10.64 ms and a maximum of 12.68 ms. The logger timer diverges further from the target, reaching 13.30 ms at the 99th percentile and 17.18 ms at its maximum.

Takeaway: assign an entire node to the real time executor when every callback in that node lies on the critical path.

Design 2: Assign one callback group to the real time executor

A single node often contains both critical and non critical callbacks. Assigning the entire node to the real time executor would also promote its housekeeping callbacks, which is not the intent. Instead, you place only the critical callback into a dedicated callback group and assign that group to the real time executor.

In this example both callbacks in the consumer node execute for 2 ms: the subscription callback, which is on the critical path, and the timer callback, which is housekeeping. The subscription is assigned to a dedicated callback group, and the timer remains in the node's default group. Passing false to create_callback_group prevents the group from being attached to an executor automatically.

rt_callback_group_ = create_callback_group(  rclcpp::CallbackGroupType::MutuallyExclusive,  false); rclcpp::SubscriptionOptions sub_options;sub_options.callback_group = rt_callback_group_; sub_ = create_subscription<std_msgs::msg::Int32>(  "int_topic", 100,  std::bind(&ConsumerNode::cb, this, _1),  sub_options); timer_ = create_wall_timer(  10ms,  std::bind(&ConsumerNode::timer_cb, this));

A subscription callback executes only when a message arrives, so its rate is governed by the publisher. To isolate the effect of scheduling on the consumer, the producer's publishing timer is also assigned to a real time callback group. This guarantees that messages are published at a strict 100 Hz, so the arrival rate at the consumer subscription is not itself a source of jitter.

rt_callback_group_ = create_callback_group(  rclcpp::CallbackGroupType::MutuallyExclusive,  false);pub_ = create_publisher<std_msgs::msg::Int32>("int_topic", 100);timer_ = create_wall_timer(  10ms,  std::bind(&ProducerNode::timer_callback, this),  rt_callback_group_);

In main(), all three nodes are added to the normal executor. The producer's publishing group and the consumer's subscription group are then added to the real time executor.

rclcpp::executors::SingleThreadedExecutor no_rt_executor;rclcpp::executors::SingleThreadedExecutor rt_executor; no_rt_executor.add_node(node_producer);no_rt_executor.add_node(node_logger);no_rt_executor.add_node(node_consumer); rt_executor.add_callback_group(  node_producer->get_rt_callback_group(),  node_producer->get_node_base_interface());rt_executor.add_callback_group(  node_consumer->get_rt_callback_group(),  node_consumer->get_node_base_interface()); auto rt_thread = std::thread(  [&]() {    sched_param sch;    sch.sched_priority = 90;     if (sched_setscheduler(0, SCHED_FIFO, &sch) == -1) {      throw std::runtime_error{        std::string("failed to set scheduler: ") +        std::strerror(errno)};    }     rt_executor.spin();  }); no_rt_executor.spin();rt_thread.join();

Both callbacks execute for 2 ms. The subscription runs under SCHED_FIFO and holds that duration, with a 99th percentile execution time of 2.00 ms and a maximum of 2.01 ms. The timer runs at the default priority, and CPU contention stretches it to a 99th percentile of 7.29 ms and a maximum of 15.59 ms.

The update period shows the same separation. Because the producer publishes at a strict 100 Hz and the subscription is served by the real time executor, the subscription maintains its period, with a 99th percentile of 10.21 ms and a maximum of 15.11 ms. The timer, left at the default priority, drifts to a 99th percentile of 14.36 ms and a maximum of 30.49 ms.

Takeaway: to protect a single critical callback, place it in its own callback group and assign that group to the real time executor. Its execution time is then bounded under load. Its update period, however, is only as stable as whatever triggers it, so for a subscription you must also schedule the upstream publisher predictably, as done here by promoting the producer's timer.

Design 3: Assign a complete pipeline to the real time executor

The third example is a processing pipeline. A sensor driver publishes an image, an obstacle detector consumes it and publishes a detection, and a brake actuator consumes the detection and responds. Each node also carries a state reporting timer that is not on the critical path. The full source is in design_3.cpp.

The critical timer and subscriptions are placed in explicit callback groups. The state timers remain in the default groups.

// Sensor driverrt_callback_group_ = create_callback_group(  rclcpp::CallbackGroupType::MutuallyExclusive,  false);timer_scan_ = create_wall_timer(  10ms,  std::bind(&SensorDriverNode::produce_data, this),  rt_callback_group_); // Obstacle detectorsub_options.callback_group = rt_callback_group_;sub_ = create_subscription<sensor_msgs::msg::Image>(  "image", 100,  std::bind(&ObstacleDetectorNode::detect_obstacle, this, _1),  sub_options); // Brake actuatorsub_options.callback_group = rt_callback_group_;sub_ = create_subscription<vision_msgs::msg::Detection3D>(  "obstacles", 100,  std::bind(&BrakeActuatorNode::react_obstacle, this, _1),  sub_options);

All nodes are added to the normal executor so that their state timers run there. The three critical callback groups are added to a real time multithreaded executor with three threads.

rclcpp::executors::SingleThreadedExecutor no_rt_executor;rclcpp::executors::MultiThreadedExecutor rt_executor(  rclcpp::ExecutorOptions(), 3); no_rt_executor.add_node(node_sensor_driver);no_rt_executor.add_node(node_obstacle_detector);no_rt_executor.add_node(node_logger);no_rt_executor.add_node(node_brake_actuator); rt_executor.add_callback_group(  node_sensor_driver->get_rt_callback_group(),  node_sensor_driver->get_node_base_interface());rt_executor.add_callback_group(  node_obstacle_detector->get_rt_callback_group(),  node_obstacle_detector->get_node_base_interface());rt_executor.add_callback_group(  node_brake_actuator->get_rt_callback_group(),  node_brake_actuator->get_node_base_interface()); auto rt_thread = std::thread(  [&]() {    if (enable_rt) {      sched_param sch;      sch.sched_priority = 90;       if (sched_setscheduler(0, SCHED_FIFO, &sch) == -1) {        throw std::runtime_error{          std::string("failed to set scheduler: ") +          std::strerror(errno)};      }    }     rt_executor.spin();  }); no_rt_executor.spin();rt_thread.join();

The measured quantity is the end to end latency from the start of the sensor callback to the completion of the brake callback, which represents the full obstacle response path.

Under the real time policy the path is bounded: the median is 7.31 ms, the 99th percentile is 7.36 ms and the maximum is 8.92 ms. At the default priority the same path has a median of 9.32 ms, a 99th percentile of 22.26 ms and a maximum of 44.26 ms. Under CPU contention the default priority run accumulates delay at every stage, and these delays compound along the chain.

The update period follows the same pattern. The real time pipeline holds a 99th percentile period of 10.01 ms with a maximum of 10.03 ms. The default priority run reaches 13.97 ms at the 99th percentile and 23.28 ms at its maximum.

Takeaway: when a deadline spans a chain of nodes, assign every callback on that chain to the real time executor and leave the off path work at the default priority. This produced the largest improvement of the three designs.

SCHED_FIFO requires elevated privileges. Without them, sched_setscheduler() fails with Operation not permitted. Apply real time priority deliberately, because a high priority thread that never yields can starve the rest of the system.

Composable nodes

A conventional ROS 2 executable hosts one or more nodes in a single process. A composable node is instead compiled as a shared library and loaded at runtime into a container process. The source for this section is in composable_nodes_tutorial.

This makes the process layout a deployment decision rather than a compile time decision. The same component can run in a dedicated process during development and share a container with related components in production.

When a publisher and subscriber run in the same process, ROS 2 can use the intra process communication path, which passes the message by pointer and bypasses serialization, the loopback network layer and the associated copies. The cost is reduced fault isolation, because if one component crashes the container, every component in that process terminates with it.

ROS 2 provides three container executables:

1. component_container runs a single threaded executor.
2. component_container_mt runs a multithreaded executor.
3. component_container_isolated assigns a separate executor to each component.

The intra process path requires use_intra_process_comms on both the publisher and the subscriber, and it delivers a message by pointer only when the publisher hands ownership to the middleware, so the publisher must publish a std::unique_ptr rather than a value.

The tutorial launch file loads the publisher and one subscriber into a component_container_mt, both with use_intra_process_comms enabled, so messages between them travel over the intra process path. A second subscriber runs in a separate component_container; because it is in a different process, it always receives messages over the inter process path. Each subscriber measures the interval between the publisher timestamp and the start of its own callback. The plot below uses 30 samples from each subscriber.

The subscriber on the intra process path recorded a median latency of 197.0 µs with a 95th percentile of 297.7 µs. The subscriber on the inter process path recorded a median of 393.5 µs with a 95th percentile of 548.2 µs, so the intra process path roughly halved the median latency.

This measurement uses a small Header message, so the difference reflects the fixed cost of the inter process path, namely serialization and the loopback transport, rather than payload copying. The saving grows with message size. Transporting an image or a point cloud between separate processes serializes and copies every message, at a cost proportional to the payload. The intra process path passes the same message by pointer and eliminates those copies, so for large messages the reduction in latency, CPU usage and memory traffic is far larger than the figures above. The exact numbers depend on the middleware, the message ownership semantics and the system load, so you should measure on your target hardware.

Use separate processes when fault isolation and independent debugging are the priority. Use a shared container with the intra process path enabled when tightly coupled nodes exchange large messages and require low communication overhead.

Reference

For a broader introduction to these concepts, see A Concise Introduction to Robot Programming with ROS2 by Francisco Martín Rico.