To maintain platform responsiveness and interactivity, the platform avoids looping over all the data in a movement dataset.
One way this is achieved is by only visiting the data within the spatial and temporal bounds defined by the user. Another way is by pre-processing the data and generating multiple overview (or resolution) levels, where each level contains progressively fewer records by down-sampling in time (as explained here).
For instance, if the input movement data contains for each asset records every second, the overview levels will contain records every 2 seconds, 4 seconds, and so on. When looking at the movement of an asset at the scale of a country, it is often not necessary to use 1-second intervals.
This article explains how the platform selects the appropriate resolution level to evaluate filters and compute statistics, and how you can influence this selection.
The resolution level selected is determined solely by the geospatial area that the platform needs to evaluate. In short: the larger the area under evaluation, the coarser the data resolution the platform will use. This ensures that the number of records to process remains within a manageable threshold.
As explained in the Introduction to advanced filters article, filter pipeline evaluation first assesses the asset and asset route filters, followed by record filtering. However, there are some nuances not covered in the introductory article.
In the first phase of the filter pipeline, the asset and asset route filters are evaluated.
Since asset filters only use metadata properties and the properties/time/location of the first and/or last record of an asset, there is no need to access the overview levels. This information is stored separately and can be accessed without losing responsiveness.
The situation is different for the asset route filter.
An asset route filter describes a trajectory that the asset should follow, and optionally specifies which properties the asset should have along the trajectory.
A separate level is selected for each segment of the trajectory, based on the geospatial area for that segment. The platform loops over the records (available at that level) in the specified area, and checks whether they match the optional property and time filters.
Thus, if your asset route filter has one step with a large area and another step with a small area, different overview levels will be used for each step.
The result of evaluating the asset (route) filter is a set of matching assets, and for each asset, a set of time intervals where the asset matches the asset (route) filter.
For example, the result might be:
Asset 1 matches between April 17, 2020, 05:30:00 and April 17, 2020, 07:45:00
Asset 3 has two matching intervals: one between April 17, 2020, 06:23:00 and April 17, 2020, 07:38:00, and another between April 18, 2020, 06:13:00 and April 18, 2020, 08:07:00
…
To evaluate the record filter, the platform selects an overview level based on the area of interest:
On the visual analytics page, this area is defined by the spatial map (or the restriction bounds drawn on the map).
On the distribution and regular trend analytics pages, there are both global and local evaluations:
Global distribution/trends use no area restriction, so a very coarse overview level is selected.
Local distributions/trends use specific areas for level selection, where the chosen level depends on the size of the area.
On the categorized trend analytics page, the user selects a single area that determines the overview level.
Once a level is selected based on an area, the platform loops over all records from that level within the specified area.
For each of these records, the platform checks:
Whether the record matches the record filters in the pipeline
Whether the record matches the asset (route) filter, which involves checking if the record belongs to an asset that passed the asset (route) filter and whether its timestamp falls within one of the matching intervals for that asset.
The fact that the asset route filter and the record filter might use different overview levels (and thus data with different levels of detail) has implications.
For a record to pass through the filter pipeline, it must:
Be present at the level selected by the record filter, as these are the records over which the pipeline iterates.
Pass the record filter component of the pipeline.
Have an asset ID that passes the asset (route) filter.
Have a timestamp matching one of the time intervals identified by the asset route filter.
It may happen that the asset route filter identifies a specific asset as matching between times t1 and t2.
However, records for that asset will only appear in the results if, while looping over the records in the record filter phase,
the platform encounters records for that asset within that time range.
If the area used for record filtering is too large, all records for that asset between t1 and t2 might have been removed at that overview level, meaning the asset will not match the filter.
This is why limiting the record filtering area as much as possible ensures the most accurate results. For example, when analyzing traffic on a highway between two cities 100 km apart, but only interested in the results along a 1 km stretch, it is best to use that 1 km stretch as the area for record filtering.