Synchronization of Multi-User Event Media (SEM) Dataset
Task in Mediaeval 2015

• time synchronization error
• sub-event detection error

As far as the first metric is concerned, the goal of the participants is to maximize the number of galleries for which the synchronization error is below a predefined threshold, and to minimize the time shift of those galleries. The synchronization error for a gallery with respect to the reference is defined as the difference between the real offset (computed on the ground truth) and the the current offset computed by the participant. The threshold depends on the duration of the sub-events in the dataset, and represents the maximum accepted time lapse within which we consider a gallery as reasonably well-synchronized.

As far the metrics for evaluation are concerned, we have considered for the temporal alignment the precision and accuracy. For the quality of the clustering, we use the Jaccard index (JI), and the F1 score, where P and R represent the Precision and Recall, respectively.

Precision measures the number of galleries (M) over the total number of galleries (N-1, excluding the reference), that have been correctly synchronized. With the accuracy we instead evaluate the capabilities of the teams in minimizing the average time lapse calculated over the M synchronized galleries, normalized with respect to the maximum accepted time lapse.

The synchronization task provides a basis for the clustering task. Once the galleries are synchronized, it is possible to cluster the whole event collection to detect sub-events occurring within the entire event, for instance, the single competitions, or the ceremonies of the different disciplines. Sub-events are defined in a neutral and unbiased way (e.g., making reference to the calendar/schedule of the event) and coded into the ground truth. We measure the performance of the sub-event clustering over the whole synchronized collection of media. For this, we use the Jaccard index JI and the clustering F1 score, where for computing the latter we use P and R, which represent the Precision and Recall, respectively. In the formulation we declare a true positive (TP) when two images related to the same sub-event are put in the same cluster, and a true negative (TN) when two images belonging to different sub-events are assigned to two different clusters). False positives (FP) occur instead when two images are assigned to the same cluster although belonging to different sub-events.