Data performance and validation (Telraam S2 and V1)

Data performance and validation of Telraam S2

As we are getting closer to the release of our Telraam S2 device, we are continuously working on improving the AI which is responsible for the traffic counts. This is a long and incremental process, but to stay completely transparent, we would like to provide some updates on the current performance of the AI chip, even though this will improve both before and after the initial release. We will update this article regularly to keep you up to date with the validation of the S2 data.

🆕   Update Dec 20th, 2022

We use a debugging tool to test any kind of change that we make on the firmware of the AI chip, which can emulate the performance of the firmware on simple video files (recorded with, e.g., a GoPro camera). As input we have assembled a set of 12 short videos (2-6 minutes) covering various streets across Leuven, Brussels, and London; ranging from cycling and pedestrian areas to 2×3 lane express roads, and representing various traffic and weather conditions. We derived the true number of road users from these videos by manual counting, and we used the debugging tool to get the totals that would be counted by our S2 devices.

When looking at the segments together (in the figure above), we can see that the performance is already very good. 874 objects were counted by Telraam S2, while in reality there were 870 objects, meaning that the error on the total number of objects is half percent (or that the accuracy is 99.5%). When looking at the accuracy on a road user type basis (in the figure below, orange graph of full sample), we see accuracy values of 90% for heavy vehicles, 95% for cars and light trucks, 90% for bicycles and motorcycles, and 85% for pedestrians.

As some segments perform better than others, and various conditions can result in overcounts or undercounts, we also calculated the accuracy on a segment-by-segment basis where the absolute errors are averaged over the 12 segments and weighted by traffic volume. This results in accuracy values at or above 90% for all modes except for heavy vehicles and pedestrians, but the total number of these objects is relatively low in our sample, and heavy vehicles suffer from a still-to-be-corrected detection bias, while pedestrians were only present in large numbers in two of the 12 videos, both cases in very challenging situations (groups of pedestrians on crossing trajectories), where we don’t expect the AI to be fully capable of detecting every object.

We have already identified a few key areas where we see big potential for further improvements even before the release of the S2 devices and will keep you up to date in this FAQ article.

Data performance and validation of Telraam V1

See blogpost https://telraam.net/en/blog/precision-accuracy-and-validations-of-the-original-telraam-sensor