Fast and low-latency IPS / RTLS with Ultrasound + IMU sensor fusion for autonomous indoor drones and robots

Executive summary

  • 80 Hz location update rate (it may be higher in the future with the new software releases on the same hardware)
  • 12-20 ms latency – depending on the type of mobile object movement and settings
  • 3D tracking. 1D and 2D tracking is supported as well (of course 🙂
  • Particularly good for autonomous drones and other noisy objects
  • This article focuses on NIA and MF NIA. Ultrasound + IMU sensor fusion for IA was released some months ago
  • The direction is provided with a single beacon. The Paired Beacons configuration for Location + Direction is supported too
  • Substantially higher robustness of tracking in temporary non-line-of-sight environment
  • A new Marvelmind Dashboard 3D is included in the pack for easier 3D visualization (in player-only mode yet)
  • The functionality is enabled by the MMSW0021: Realtime Ultrasound + IMU sensor fusion license (both IA and NIA)
  • It is fully commercially available. Download and start using it now

Minimal configuration

The NIA indoor positioning system with the IMU sensor fusion consists of several key elements:

  • The mobile beacon placed on the mobile object, for example, a drone
  • Stationary beacons placed around to provide the coverage – see the videos above to have an idea and check the Placement Manual for recommendations
  • The modem – to control them all. In NIA, it is not calculating the location anymore. The mobile beacon does it now. It is a new thing from now on
  • The RHU – to receive the fast IMU data on the modem’s side. Additionally, it does a few layers post-processing, thus, further improving robustness and quality of tracking. It is relatively a new kid in the block. However, it has been a key element of the Split-Modem Architecture, which becomes a mainstream, if you want to get the low-latency IMU sensor-fused data on the modem’s side. If you consume the data on the mobile beacon’s side, you don’t need the RHU

Thus, the recommended minimal configuration to enjoy high-speed, low-latency tracking:

Only the newer beacons with -4 or later in the name supported

Currently supported mobile beacons are:

If you purchase the hardware now, you can ask for the distributors to check and ship the -4 version. In our warehouse, we have only -4 left. But you can still find plenty of -1, -2, -3 versions in the field.

Newer hardware version of these and other mobile beacons likely to support the MMSW0021: Realtime Ultrasound + IMU sensor fusion.

To avoid doubts:

  • Super-Beacons, Super-Beacons-2, Super-Beacons-3, Industrial Super-Beacons-2 and -3 and Industrial-RX-2 and -3 and other more mature devices are not supporting the MMSW0021: Realtime Ultrasound + IMU sensor fusion as mobile devices
  • However, for stationary beacons, there are no restrictions and they can be used as stationary beacons easily, if they can work as as stationary beacons in IA. For example, Industrial-RX cannot act as a stationary beacon in IA, because it doesn’t have transducers at all and cannot emit ultrasound. It is not related to this feature, but the basics about the architecture requirements must not be forgotten. Even very mature Beacons HW v4.9 can be used as stationary beacons

Submaps, super-maps, MF NIA, PixHawk, DJI and everything else that is normally supported is supported by the system is supported by NIA IMU sensor fusion as well. IMU sensor fusion is an add-on top the existing layers.

Sensor fusion is a holy grail in robotics

With sensor fusion, you grab the best of both worlds:

  • Ultrasound provides high-accuracy positioning and no location drift but it can jump due to even short occlusions or noise
  • IMU provides extremely high angular resolution, no location jumps, but it drifts

Fusing both sources properly is technically very challenging, takes some assumptions about the mobile object, but returns results unachievable in standalone ultrasound-only or IMU-only systems: very high location update rate, low latency, more robustness for occlusions, and higher accuracy. So, the best of both worlds, literally.

Sensor fusion is a tree of technologies - not a single-solution-serving-them-all

As discussed in the previous article, IMU sensor fusion has several branches. In short:

  1. IMU sensor fusion for NIA – optimal for noisy objects such as drones
  2. IMU sensor fusion for IA – when the mobile beacon receives ultrasound – people tracking or drones/robots in swarms
  3. IMU sensor fusion post-processing – it is great when you need the best overall tracking, because you already have the whole view of the data but latency is not critical, for example, for sports or track analysis
  4. IMU sensor fusion for direction – the Paired Beacon configuration
  5. Extrapolation – technically, it is not sensor fusion. But it is an optimal solution for cases with a predictable path

Low-latency, high-speed, higher robustness, direction with single beacon and other benefits

With an article about the IMU sensor fusion for IA, we very much focused on the low latency of the location data that IMU sensor fusion brings and how to measure it. Thus, you can see multiple impact graphs and explanations of how we measured the latency from the mechanical impact to the IMU integration to processing and calculation of the location in that article.

With the new IMU sensor fusion for NIA software and further development where IMU sensor fusion for IA and NIA will be combined in a major software release, we stress more on the overall superiority of the IMU sensor fusion approach rather than the low-latency of the location data:

  • High update rate – 80 Hz or higher in the future – it is enough for the vast majority of the real-life industrial applications
  • Low-latency – 12-20 ms – it is difficult to measure in practice and it depends the settings, the map, the history of movements of the object, i.e., it is adaptivee. But the key point that it is also enough for the vast majority of the real-life applications. We are in the industrial automation business after all
  • Increased accuracy, particularly in static
  • Substantially higher robustness to short-time occlusions, acoustic noise and other adversary factors and improved quality of tracking with the layered approach
  • A single mobile beacon gives a direction/pose now. It is dynamic. You must move ~50 cm or so in any direction. The system determines the location and highly accurately measures it and maintains it even in static until you switch off the system

Summing up, the IMU sensor fusion is indeed a holy grail in robotics overall and in the indoor positioning as its subset with multiple benefits brought along with it.

Increased accuracy

IMU sensor fusion increases accuracy by combining more data and by eliminating some jumps.

Particularly beneficial IMU sensor fusion for accuracy in static conditions. At the same time, in dynamic, the IMU sensor fusion may increase the swing of error, but the result totally depends on the setup, conditions and the type of the movement. With more knowledge of the object, you can optimize the settings and the performance even further.

A single mobile beacon gives direction now as well

We used to require a Paired Beacon configuration to know the direction.

Now, we recommend flying with a single mobile beacon with the Paired Beacons remain an option.

Single mobile beacon:

  • A light option: no need for the second beacon, frame, base distance, two wires between mobile beacons
  • It requires ~50 cm of movement in any direction to determine the direction and then locks it
  • It maintains and remembers the direction even in static until you switch off the system
  • It is essentially the only option for smaller drones like DJI Mini or smaller

Paired Beacons:

  • Provides the direction even in static without a need to fly first to determine the direction
  • Particularly good for larger drones where the beacons’ weight is not critical and the base is large enough

Read more:

Increased robustness and quality of tracking

We employ a layered approach to tracking:

  1. Ultrasound tracking: it classic and you can see the blue dots in our Dashboard 2D
  2. Based on the ultrasound and IMU data, we generate a sensor-fused data. It is already brings all the benefits of low latency and high update rate. That data is available on the hedgehog’s (mobile beacon’s) side, but not yet on the modem’s side. So, a fully autonomous PixHawk drone can use it, but a DJI drone with brains outside of the drone, cannot
  3. The relatively slow-speed (~8 Hz) data is streamed to the modem as usual. But the high-speed (80 Hz) IMU-fused data is streamed to the RHU. You can see the very fast blue tail dots on the videos – that is the IMU-fused data
  4. The RHU not only receives the data but extensively process it both on the IMU layer but also on the ultrasound layer again but with more data available. There are several layers of processing. But some final layer is the golden one, which chooses the best information with the highest level of confidence from all the layers. Thus you as a user, can get both very fast IMU data with 12-20 ms latency and about 200-300 ms later even more verified golden data. As a result, the robustness of tracking increases notably. You can easily visually observe how cleaner and more coherent tracking is even not-so-good conditions or partial or temporary obstructions. 

Connectivity to user devices and methods to avoid extra latency

Since one of the biggest kicker for the feature MMSW0021: Realtime Ultrasound + IMU sensor fusion is its low-latency of the location data, it is crucial to clearly understand how to get the location data and not to introduce extra latency due to the interfaces or operating systems.

With the new version supported now in the new:

  • Regular UART from the mobile beacon (also called hedge or hedgehog) using the 4×4 pins of the Super-Beacon or pins inside the 7-pin data connector of Industrial beacons is the best and easiest way to get the low-latency location data without noticeable extra-latency
  • Via USB from the hedge (additional latency ~ 1 ms + latency of the USB driver (OS dependent)). The biggest concern is the unpredictability of the OS latency and its jitter. The extra latency is difficult to assess – from unnoticeable to substantial. The potential latency jitter due to the OS uncontrollable delays is another concern. In the reality, however, these concerns may be over-exaggerated and then such connectivity is even simpler than the wires from the UART (see above) because it uses USB
  • Via USB/UART of the RHU. Effectively, we employ the Split Modem Architecture. This is the best way to get the low-latency data on the modem’s side – not on the mobile beacons’ side – when you want to track something with the high location update rate and the low latency or to fly DJI drones, for example, because the “brain” is external in that system – not on the drone

There are other more theoretically possible rather than practically recommended ways to extract the location data:

  • Via UDP from Dashboard with hedge connected via USB (additional latency ~ 1 ms + latency of USB driver + latency of UDP transfer (typ. 3…50 ms depending on your network))
  • Via REST or MQTT from Dashboard with master hedge connected via USB (additional latency ~ 1 ms + latency of USB driver + latency of REST/MQTT transfer). Also, difficult to assess the total extra latency. But it is supported as well
  • Via UDP/REST/MQTT/log from Dashboard with modem HW v 5.1 with special firmware in the Split Modem Architecture (additional latency same as for hedge connection to Dashboard + ~ 1ms via radio at 500 kbps)
  • Via log file in Dashboard with master hedge connected via USB (for post processing – latency n/a)

Ever-evolving sensor fusion for precise, fast, low-latency indoor tracking

NIA and IMU sensor fusion is fully available already now and you can order, download, and enjoy it already today:

However, its evolution never stops. The approach – to fuse more data, combine it smartly, and provide tracking with even more speed, even lower latency, and even higher robustness – is an ever-winning approach.

Scroll to Top