Synchronization of Multi-User Event Media (SEM) Dataset
Task in Mediaeval 2014
Synchronization of Multi-User Event Media (SEM) Dataset |
|
|
Dataset description |
For this challenge we make available two different datasets, consisting of a collection of images gathered from Flickr and made available under Creative Commons license. Both datasets refer to well known and structured sport events, namely the Olympic Games held in London in 2012 and the Vancouver Winter Olympic Games of 2010. We have chosen to work with these two events because on the one hand they exhibit a clear and organized schedule with precise tim- ing. On the other hand they still exhibit a high variabil- ity in terms of visual content, due to the common features across different competitions in the same discipline, as well as strong similarities in the environments, in which the pictures are collected, making the synchronization a non-trivial task. As far as this task is concerned, the images within a gallery are consistent in terms of timestamp, and might include the GPS information. Therefore the temporal offsets are at gallery level thus assuming that every user uses one single device for acquisition. The dataset collected from the London Olympics includes 2124 images, divided into 37 galleries. The first gallery comprises a subset of the data provided in the development set and is defined as the reference gallery. The dataset collected from the Vancouver Winter Olympic Games includes 1351 pictures representing most of the competitions, divided into 35 galleries with a variable number of pictures in each gallery. Also in this case, the first gallery is set as the reference. |
Evaluation Metrics |
Two objective metrics have been used to evaluate the results:
As far as the first metric is concerned, the goal of the participants is to maximize the number of galleries for which the synchronization error is below a predefined threshold, and to minimize the time shift of those galleries. The synchronization error for a gallery with respect to the reference is defined as the difference between the real offset (computed on the ground truth) and the the current offset computed by the participant. As far the metrics for evaluation are concerned, we have considered for the temporal alignment the precision and accuracy. For the quality of the clustering, we use the Rand Index (RI), the Jaccard index (JI), and the F1 score, where P and R represent the Precision and Recall, respectively. Precision measures the number of galleries (M) over the total number of galleries (N-1, excluding the reference), that have been correctly synchronized, namely those galleries, for which the alignment error with respect to the reference gallery, is below a threshold. With the accuracy we instead evaluate the capabilities of the teams in minimizing the average time lapse calculated over the M synchronized galleries, normalized with respect to the maximum accepted time lapse. The synchronization task provides a basis for the clustering task. Once the galleries are synchronized, it is possible to cluster the whole event collection to detect sub-events occurring within the entire event, for instance, the single competitions, or the ceremonies of the different disciplines. Sub-events are defined in a neutral and unbiased way (e.g., making reference to the calendar/schedule of the event) and coded into the ground truth. We measure the performance of the sub-event clustering over the whole synchronized collection of media. In this case, we use the three performance indicators reported above, namely RI, JI, and F1. In the formulation we define a true positives (TP), in case two images related to the same sub-event are associated the same cluster, and the true negative (TN), when two images associated to different sub-events are assigned to two different clusters). False positives (FP) occur instead when two images are assigned to the same cluster although belonging to different sub-events. |
Reference |
|
@article{conci2014synchronization, |
Download files |
London Olympics Dataset |
|||||
Galleries 1-7 |
Galleries 8-9 |
Gallery 10 |
Galleries 11-19 |
Galleries 20-25 |
Galleries 26-37 |
 |
|
|
|
|
|
Vancouver Olympics Dataset |
|||||
Galleries 1-8 |
Galleries 9-16 |
Gallery 17-24 |
Galleries 25-35 |
||
|
|
|
|
||
Ground Truth |
|||
Contacts |
For further details please contact: |