High-Speed Tracking with IMU Sensor Fusion | Marvelmind

0:00 Hello colleagues. Today we will be talking about a faster positioning system for quick sports, quick industrial applications, and machinery, or quick drones. Of course, it's still about precise indoor positioning system, but the focus is on fast-moving objects and high update rate, and the ways how to achieve it. What's the problem? If you have a typical update rate of 8 Hertz, for very quick-moving objects it may be simply not enough because you will have too few location points per movement, and instead of smooth track you will have just sparse dots. What other options? Well, two options: either you slow the movement, which is typically not possible, or you increase the update rate. A regular way to increase the update rate we have shown already on our page.

1:00 Please check it. Basically, it's about reducing the size of their submap if possible, because our typical and default is 30 meters. If you have just five meters, you can increase the update rate significantly—basically six times. It's about increasing their radio profile because in some cases, radio may be the limiting factor, and reducing all kind of averaging, all kind of additional algorithms that can increase their latency. More or less, they do not decrease the update rate. But those typical cases we have discussed—when it's still not enough—there must be new ways to do this. So this is how it will look. This is your typical 8 Hertz submap trade, and as you can see, it's very precise.

1:57 Still, everything lovely—plus minus two centimeters. But we specifically moved very, very quickly just to show that the distances between the dots may be comparable with their clarity. The path is when I have a smooth track—like I don't know, ellipse—it wouldn't be a problem. But when I have sharp curves and sharp turns, the update rate maybe is not enough, and the only way for the solution is to increase the update rate of the location system. What's the problem? The ultrasound propagates slowly. That's great because it helps our system to be so precise. Remember that the speed of sound is around 340 meters per second, and the speed of light is one million

2:55 times more. So basically, it means that for radio-based systems—we are measuring time of flight, but time of flight of ultrasound as compared to time of flight of radio—for radio-based systems, it's actually a million times, million times more difficult to achieve the same accuracy. Of course, it's a very fundamental simplification, but nevertheless, time of flight to ultrasound is the biggest asset when we talk about the accuracy, and at the same time, it's the biggest limitation when we talk about their update rate. Why? Now, imagine that you have a large map—for example, 30 meters. So it meant that the time for the sound to propagate in this distance would be 340 meters per second divided by 30, so a gross maximum update rate that you can achieve without any other losses would be around 11 Hertz, which is the maximum. But of course, on top of this you will have losses due to processing, filtering, radio, etc. So effectively, on 30 meters you would expect roughly 5, 6, 7—depending on the settings, depending on which architecture you are using—but roughly that one. So lost about 50 percent, maybe 40 percent.

3:53 So which is okay for the majority of industrial applications because typically people said, "Okay, I want real time. How much?" Okay, one Hertz, maybe two Hertz. That's great. So we typically talk about 8 Hertz for let's say typical submap which is 10, 20 meters indoor. But for example, if you have a large map like 30 meters, you'll have around 11 Hertz theoretical limit, and 5, 6, 7 Hertz real, practical. And if you have a very quick cart moving, that would mean that their dots come to a rare. And if the object is moving too quickly, some data points may be simply missed. So it means that you have a curve, but instead of curve—for example, there's a curve—you have one location and another location, and this location you simply don't have it because it was so quick.

4:50 If you have a smaller map, you may, purely based on ultrasound, achieve a high update rate. So we experimented in the past, and it was around 40 Hertz—like an absolute minimum, minimum. As we show here in an air hockey—oh no, was this air hockey—case, the size is just 2–3 meters, so you can get around 40 Hertz update rate, which is great for this simple case. But for the larger case, for example, karting, that would be not enough. Why? I'm referring to karting because we have a couple of cases where people want, you know, very quick movements and they want a very precise track. Basically, in order to teach their players or drivers how to drive properly. So our system is perfect for this because it provides their high accuracy of track, and now it provides very high update rate, which is 100 Hertz, which is enough for even very, very fast-moving mobile objects.

5:49 And in order to achieve that, it's almost impossible to achieve by ultrasound because once again, return back to the basic calculations: 340 meters per second divided by 100 Hertz update rate. So the size of the map, even theoretical, would be just 3 meters, which is tiny. What to do? Sensor Fusion is a solution. Sensor Fusion is my personal favorite topic. It's extremely complex, but it allows the best from both worlds. It allows high accuracy of indoor positioning system, and at the same time, it allows very high update rate. It can be done even more than 100 Hertz from inertial measurement unit, from IMU. So let's talk about this deeper. But before we go there, what are the options? So basically, there are three: real-time player, which is already available in the Dashboard; IMU Plus ultrasound Sensor Fusion post-processing, which we are discussing now in this video and in this presentation; and IMU plus ultrasound Sensor Fusion real-time, a particularly complex topic. It will come later—it's not yet commercially available. But the first one, real-time player, and IMU and ultrasound Sensor Fusion post-processing, is available right away.

6:49 So what is real-time player? A real-time player is basically a functionality inside the Dashboard. It gets their location points, and then it does a relatively complex interpolation inside by using algorithms, by using filtering, by using everything. So it tries to guess—again, it's interpolation, guess, not measure—the location points. The result is pretty good, and it's suitable for relatively smooth curves. So when you don't have sharp turns or you have already sufficiently high update rate, real-time player is very good and is recommended. And it's great because it's already available—it's free of charge now. So you can basically use it right away and use it for the real-time data and for data that you already have recorded, like in your log file. So use it, try it, use it. It's very, very good.

7:46 So let's check all the performance. So this is the typical curve, and you can see this blue dots are ultrasound dots. This specifically and intentionally, I did it very, very quickly. So basically, we jumped very, very quickly. As you see, their scale is five centimeters between there, so this is five centimeters. And again, there are just a few dots—so three here because it already stopped moving, one, one. Okay, I hear because it was then one, two. I don't know, one, two here. And the track for real-time player is this pink one. Okay, the orange, we more or less reveal immediately what their IMU Sensor Fusion

8:44 time data and for data that you already are recorded like in your log file so use it try it use it it's very very a very good one so let's check all the performance so this is the typical uh curve and you can see this blue dots are ultrasound dots uh this specifically and intentionally did it very very quickly so basically we jumped very very quickly as you see their scale is five centimeters between there so this is five centimeters and again uh there are just a few doors so three here because it already stop moving one one okay if you hear because it was then one two I don't know one two here and the track for real-time player is this uh pink one okay the orange we more or less reveal immediately what their IMU Sensor Fusion

9:43 So I'm just, as a fusion, obviously shows much, much better. But let's not touch that one. So the topic is blue and pink. So as you can see, a real-time player is basically taking in some location points from the future—in this case zero from the future—and five points from the past. And then it's trying to put this information and place their, basically interpolate using these points. So as you see, it's trying its best, but it's not perfect. In this case, it's slightly better because there are no overshoots, et cetera. But as you see, this is the biggest drawback of their real-time player because it doesn't have this information. It doesn't have their acceleration information—unlike the IMU, which we will be discussing. So it's basically cutting the corners. It's making smooth...

10:42 ...smoothing of these corners. So, and it brings the errors. They're still not huge. So once again, this is five centimeter. So for example, this error is around five centimeters. But the problem is that you don't know in advance what will be your movement, so you cannot adapt this way or this way. So what kind of filtering? Some of them are greater for this type of movement. Some of them result you give a better result for this kind of movement. But the logic is very simple. Real-time player is great. It's suitable for smooth movements because it makes it real—100 Hertz, not real, but interpolated 100 Hertz. But in practice, it's basically helping as well. So it's not only producing 100 Hertz update rate and place those locations, but it's also doing the filtering. So if there are some...

11:41 ...occasional jumps, so when there's a sufficient number of ultrasound points, it basically helps in all ways. So one is it increased update rate. Second, it's filtering some occasional jumps or puts emissions on ultrasound. So if there is no location, for example, it will still work. And this is why this distance does matter. If I have a huge distance, then you would be able to guess pretty well or even two, three points or emissions on the allocation update of ultrasound. But at the same time, you will have more smoothing and your corners will be cut more and your error will be more. So there you can play with it and you can balance it. So for basic smoothing and for basic 100 Hertz, usual time player is a powerful tool. It...

12:41 ...is helpful. Again, it's post processing, so it's not real time from our perspective because our real time is basically 15, 10 millisecond. But it's real time, so you don't need to store the file and then play it. No, you can do this in real time, but it will be latency. What latency? This is the latency. So effectively, your latency would be in this case seven location update points. If I have eight Hertz, so the latency would be around one second, roughly one second. But again, key point is available now. It gives 100 Hertz. It works very well for relatively smooth curves. And it's not costly. It's free now. So just another example for another sharp, sharp curve. You see, it's more...

13:41 ...it's making even a smoother curve. But you know, cutting the corners even more. And you see, since it's expecting three or four points, but they are not available because again there was not available. So this is another example with rectangle. As you see, here you have a relatively more dots. So these are your blue ultrasound locations. This is your ground truth. I knew resulting IMU, and this is your real-time player. Well, I would say good enough, good enough for majority of cases.

14:40 Because once again, it's real. Everything two centimeters inaccuracy. But there were less corners are kind of smoothed and error is here accumulated. But if you do it properly, you can use it. But once again, it was with this one forward, five backward calculations. With this three forward and five backward, as you see, it's more smoothing, but the error is already somewhat five centimeters or more. It's nice. It's even closer to the reality when the curve is not, you know, too curvy. But the error in their corners or where there's a very quick change in direction becomes high. And it will be high if you increase their, let's say, the order of their player.

15:40 And this is a close look. Well, basically, this is already one centimeter. So you can calculate roughly five centimeters error due to real-time player. But again, in these areas, it helps a lot and it makes it even smoother. And the error is already lower. It's very difficult to calculate, but you can see it's basically reducing the jitter or spreads the jitter of ultrasound, making their location update 100 Hertz. And at the same time, this spot smaller than two centimeters, so better than their original ultrasound. So and I mentioned for smooth curves, it's almost ideal. So as you see, I wouldn't guess which one is better. Is it real-time player or I'm using some fusion? So use real-time player when you have smooth curves and you are okay with post processing.

16:39 Remember that it's all post processing. You'll have around one second delay or latency. The same. But you know, slower moving triangle. It's already good enough. You see over there we had just a few dots. I don't know, one, two, three dots per line. Here we have, whatever, ten dots. And it's already good. Once again, this is five centimeters. So it's plus minus one centimeter difference from the ground truth, which is perfect. So again, the same conclusion: if you have smooth curves or you have sufficiently high number of ultrasound location updates, you can achieve even better results by using real-time player. This is why, for example, in many of...

17:37 ...our robots and slow moving videos, we do have real-time player enabled by default because it basically makes a much nicer, much more reliable and smoother track. But when all this is not enough, when you have a really fast moving thing, when real-time player is not great, then there is a need for sensor fusion. Sensor fusion is extremely powerful. And sensor fusion, of course, is better than the real-time player, no doubt. Why? Now let's talk first about the physical limitations. Physical limitation about ultrasound we already mentioned. So speed of sound is the greatest asset of system because it allows very high accuracy of indoor positioning system. But at the same time, it's the main reason for a relatively slow update rate and limitation to increase this update rate further. So this is the physical...

18:35 ...limitation. And the types of sensor fusion we already discussed, so it's real time, which will come later. It's IMU sensor fusion post processing. And today we'll be discussing only about the post processing because we have it commercially. And this is exactly the video about this—to be basically informing all our existing and future customers that this functionality is available. Please start using it. And the final and probably the most important reason is that sensor fusion in our system is done using IMU. Because sensor fusion may be done with many things—for example, in our robots, we do have sensor fusion between odometry, ultrasound, and IMU. We have in our boxy robot, for example...

19:35 Also Intel RealSense—a visual positioning system—so it can be sensor fused as well. This is why it's my favorite topic: because you can achieve the best of the best by smartly, commercially, technically combining all these data together. Because none of them are perfect. Every positioning system can have some limitations. For example, optical: light too dark, too bright, some obstructions. Odometry: it may slip and it may calculate that it's moving, but it's not moving. Ultrasound: okay, ultrasound noise can block the system completely. In all cases, if radio is not available, you cannot get the data, you cannot update it, et cetera. But by smartly combining all this data and arranging self-checking, so basically:

20:34 Sensor Fusion—the best of the best from both systems or more than one, or more than two systems can be utilized. So in our case, we are talking about IMU class ultrasound Sensor Fusion. What is IMU? IMU stands for Inertial Measurement Unit. In our case, it contains 3D accelerometer and 3D gyroscope. Each of our beacons, they have it already on board. So if you don't know, of course, use it. Let me highlight that: when we talk about their IMU and position and why it's complex—the, uh, complex to achieve location, angle, just raw angle data or measure acceleration—it's nothing. So you can get it. It's a very, very basic functionality.

21:31 But to measure the location using IMU is an extremely complex task. To measure their absolute angle, it's also a complex task, but significantly less complex. But for that, you basically need to have about beacons placed on your robot, and you will know the location of your one beacon and another beacon, and you know the direction where the robot is looking—not the change of direction, which you can achieve using only one robot, one beacon—but static direction. They call this paired beacons, and we have it. So it's not a subject for this discussion. So in this video, just to make absolutely clear and sure, we are discussing about only a single subject: which is fast location measurement with 100 Hz update rate using Sensor Fusion between IMU and ultrasound. So what's the difficulty? The difficulty:

22:31 Is in the double integration of acceleration. So the accelerometer data, if it was in, let's say, zero gravity, would be very simple because basically you know integrated, and that's good. Even the basic MEMS—so micro-electromechanical systems—anyway, MEMS are in use that are typical in your phones or in our beacons. They're good enough for this because they have low drift. And with low drift with zero gravity, when you have non-zero gravity—and obviously, when we are on Earth—it's very difficult to subtract a huge acceleration that gives the Earth from the acceleration from the movement. For example, if you are just moving like this:

23:28 You have nearly zero acceleration of your own movements, and at the same time you have one G acceleration due to the Earth. And if you don't know the location—or let's say, not location, but the position of your beacon, or orientation of your accelerometer—it's very, very difficult to distinguish between the acceleration caused by not properly aligning, or by not knowing exactly their acceleration due to the Earth, and your beacon's acceleration. It's very difficult, and the error is huge. And the result is very simple: drifting becomes out of our plus-minus two centimeters—everything—in a fraction of a second, okay? If you calibrate it well enough, so it's around one and a half seconds, sometimes one second. So we think:

24:25 One second calibrated, we can drift away. So this is why purely based on Inertial Measurement Unit, without any Sensor Fusion, without any cancellation of the drift—purely IMU-based systems—I don't know any of them really working. They are all using some sort of calibration, no, no calibration—some sort of cancellation of the accelerometer drift, because double integration obviously makes it very, very quick. So if in one second you have two centimeters error, then in two seconds you will have double of that, and double of that, double of that. So basically you are increasing very, very quickly, and in one minute your speed—moving away, even though your beacon is, you know, standing still—would be, you know, meters and hundreds of meters.

25:22 Mathematics. So this is why drifting is the major problem, and you need to cancel it. And it's canceled, of course, using our ultrasound, because ultrasound provides their absolute location. And by smartly combining—basically doing the Sensor Fusion—it's possible. And the result is the sensor-fused data, which is very accurate, because IMU provides no jumps, and IMU provides very quick 100 Hertz update rate, and no drift, because the drift was canceled using the ultrasound-based system. Gyro drifts, of course, but gyro is a lesser problem. You need to very accurately know the angle in order to subtract their acceleration from the Earth from your acceleration. But the drift of a gyroscope is less of a problem, simply because it's less drifting compared to:

26:22 Uh, drift due to imprecise knowledge of our accelerometer orientation and what we call overflow of gravitation to the axes of the accelerometer, that provides drift. Once again, let me stress: IMU Sensor Fusion is a complex functionality. It's really complex functionality. This is why, you know, it takes some time, but the fruits are, you know, perfect, because you really get the best of both worlds. Let me stress once again: IMU is perfect because there's no jumps like ultrasound. You can do it if you have an online line of sight—who cares for some time. So it means that if, for example, something is obstructing the ultrasound, okay, IMU will not pay:

27:21 Attention. And time. You will be still producing the location. And if, after a fractional second or one second, you are back to line of sight, then ultrasound will again correct it. And ultrasound corrects the drift, because there was all the time drifts of IMU—both accelerometer and gyroscope—and it does it a few times per second. And to get the location data from their double integration of the accelerometer data, which is all the time corrected by all the ultrasound data. So that's, in short, it's a very powerful, very strong, and you get basically everything perfect: 100 Hertz, you get additional robustness, and you get additional accuracy. Because ultrasound, obviously, can jump:

28:18 Plus-minus two centimeters, but it doesn't jump. It has a noise, which is a fraction of millimeters. So effectively, you could achieve—if focus would be on accuracy—fraction of millimeter accuracy. It will be, maybe in the future. It's not a subject for this discussion. But again, the best of both worlds is achievable, and the only drawback is complexity, which we solved for you. So this is, once again, our lovely tracking: very few location updates, and then you, you know, get the feature and apply their IMU Sensor Fusion post-processing, and it will look like this. So you can check and look at this. So this is wonderful. You can see how it was stopping, then many, many—it does, because, you know, there's momentum and mod dots in this, and then there was again acceleration. Did you see?

29:17 There's acceleration and then again sharp stops, so it's wonderful. You see, there's a noise of ultrasound, so it's five centimeters. So I guess it's around plus minus 1.5, sometimes plus minus two centimeters. So we think there, but the resulting inaccuracy is even better than that. So again, you have the best from both worlds: you have high update rate, more robustness against obstructions, and even better accuracy. So IMU Sensor Fusion is very good. So once again, look at the corner—you can play with it a bit later when you get there. The functionality in your hands. So let's look. This was one of those examples. So look, it's a very quick movement. So basically we did like this, very, very quick, and

30:17 update rate was around 8 Hertz. So it meant that if you have one, two, three—so I guess this was moved in, you know, one quarter of the second or maybe one third of the second—so very quick movement. And yeah, you can guess this is a triangle, but it's kind of very rudimentary triangle. By employing IMU Sensor Fusion, you get pretty decent track, which is unlike a real-time player. Shows the exact ground truth; it doesn't have those limitations of the real-time player. So this is why when you have the most demanding, the quickest moving, then IMU Sensor Fusion is your solution. Now again, we already discussed: for smooth, even real-time players, good enough. So for Vegas smooth, you can easily use the real-time player if you wish. If you want

31:16 for the location updates, but if you want to get more robustness against, for example, obstructions, then IMU Sensor Fusion once again is your choice. Let's now talk about the post-processing time. Now as I mentioned in the beginning, the Sensor Fusion that we have today is post-processing. So the location data is streamed in two ways to the computer. One is the typical way: so the Mobile Beacon is calculating, or the modem is calculating. Nevertheless, there is ultrasound locations. Those do not change the same. But at the same time, IMU from their Mobile Beacon has plenty of data, which is coming very quick: 100 Hertz update rate. And it's impossible to send it over typical radio channels because there will be not enough bandwidth.

32:15 Particularly when I have many mobile objects—for example, many cars moving on your track. So this is why to each of these Mobile Beacons, they will be modem and another modem. Special modems—they're the same modems hardware-wise, but in terms of software, it's special software that will be basically serving one purpose: taking very fast data, IMU data, and sending it to a counterpart modem. And then it will be collected by the computer. So you have two streams of data, and these streams of data they have, of course, timestamps. So this is why their Sensor Fusion algorithm is capable to combine the ultrasound data and IMU data, of course, with proper timing. And thus after the Sensor Fusion processing, it's able to return

33:13 perfect locations and 100 Hertz update rate. But it's post-processing, so it takes time. How much time? Now I gave some examples. So currently, again, we didn't optimize this yet, but it will be optimized later, purely by software. You don't need to change anything hardware-wise. But today, as of now, a basic laptop—again, it's not a desktop, it's not something very special, it's not in the cloud—the very basic laptop makes the calculation with the following formula: 10 seconds plus T of drive. So how many seconds of log is recorded multiplied by 0.12. So basically I have, you know, a half-pipe half-pipe jump or some BMX jump which takes five seconds all together. So then the processing time would be 10 seconds plus five, so 10.6

34:13 seconds. So after you jump, in 10.6 seconds you have your perfect 3D track. If I have 15 minutes car drive, then it will be 10 seconds plus 15 multiplied by 0.12, so resulting around 2 minutes. So after two minutes—after 15 minutes of track, you will have your perfect less than two centimeters accuracy, 100 Hertz update track, which you can play, see the core curves, everything. And it's done on the basic laptop as I said. So on the desktop, obviously it would be much faster. And in the future, we believe that we will increase this speed of post-processing several times. But this is as of today. So just to set the expectations right, let me one more time stress about what Sensor Fusion we are talking about, because we have several

35:12 senses of fusions now, and we will have even more in the future. So there are Sensor Fusion for direction, and there's Sensor Fusion for location. Sensor Fusion for direction, which is a part of our paired beacons configuration that has been available for at least two years or something. So that's a complex task, but not as complex. Why? Well, because it's basically taking the location and there's IMU, there's gyroscope inside. So it's not using the accelerometer. So it doesn't care about the drift due to accelerometer. And gyroscope drifts very little. But like all other sensor fusions, it takes the best from both worlds. Now, first of all, Sensor Fusion for direction returns also 100 Hertz even today. So you already have it. So if you don't know, you have it already. So for example, without even the Sensor Fusion for location, so you have had it. So you had

36:10 8 Hertz location update, but for direction you have 100 Hertz. Where is it coming? It's coming from Sensor Fusion for Direction. So once again, you have two beacons. Beacons are connected by two wires. So the data is streamed. One of these beacons is Master. Master's calculating their angular position. Master is aware about their location, and Master is capable to precise and sensor fuse their Direction, which is calculated based on ultrasound—basically based on the location of each of these points and the IMU. In this case, gyroscope. As a result, you have a very reliable internal Direction. So it's basically coming from gyroscope. So it doesn't jump. At the same time, it doesn't drift because it's constantly.

37:08 The drift is constantly eliminated based on the ultrasound. So once again, the best of both worlds: no drift, 100 Hertz, and very high accuracy and extremely high resolution—so it's a fraction of degrees. But today, we were talking about their location Sensor Fusion for location, which is a new feature, significantly more complex feature, but very, very powerful. Because as I mentioned, it returns the best: 100 Hertz, more robustness, and at the same time more accuracy, because it reduces their jitter of ultrasound location measurements. What's the future? The future is the Holy Grail, which is real-time Sensor Fusion. In real-time, it's even more complex tasks because as of now you've seen, so it takes us some seconds to post-process. We somehow need to pack it and to make it in real-time. It's not

38:06 an impossible task, but we are not there yet. The biggest benefit of that will be low latency. Because now we give you 100 Hertz, they give you very high accuracy, we give you robustness, but the latency is seconds. So if you need, for example, to control your robot, if you need to stop very quickly, then we are not there yet. That requires a very low latency. Latency is the most complex task. It's solvable. It will be solved using the Sensor Fusion. It will be our future item, but not there yet. So just for you to know where we are coming and what you shall expect sometime in the future. Summary: well, let me just reinforce. So now we have several options: you have the real-time player, which is currently free of charge. You have the Sensor Fusion based on IMU, so IMU Sensor Fusion.

39:04 It's available now today. It gives you 100 Hz. The new 100 Hz, unlike the real-time player which is interpolation, it gives you at the same time more robustness and more accuracy. And it's used for fast machinery, for quick sports, for very quick drones, for example. I'm sure you have many more other applications which we are not aware. And by the way, this project has been driven mostly by cutting-edge cases. Uh, this article and this video is related to the article which is already published on the web page. Check it. It's basically showing the same information but written. If I have more questions, don't hesitate to ask. We are happy to answer. Just drop us a mail to Marvelmind.com, and thank you very much.