TRECVID 2013 Collaborative Annotation Short Help

Organized by LIG (Laboratoire d'Informatique de Grenoble)
and LIF (Laboratoire d'Informatique Fondamentale de Marseille)
More info on TRECVID 2013 Collaborative Annotation.
Authors: Georges Quénot (LIG) and Stéphane Ayache (LIF)

The TRECVID 2013 Collaborative Annotation system has been designed to be efficient and easy to use. It operates through the Web and you can access to it from your favorite browser at: http://mrim.imag.fr/tvca2013/al.html. Participation is restricted to groups that are registered TRECVID participants and that have signed the IACC.1 license agreement

The system operates in a parallel mode in which the images are displayed by groups in a two-dimensionnal array.

Access page

The access page form has eight inputs:

Groupname: the name of the group you belong to. It will be given to you by the contact person for your group. Please refer to the group table to identify the contact person for your group.
Username: an identifier for the person doing the annotations. There should be one and only one for each physical person doing annotations in your group. It can be chosen arbitrarily (e.g. steve or user#). It is local to the group and will not be confused with a possibly identical username in another group. Check with the contact person for your group to ensure unicity within your group. There can be as many username as you want in one group. The username is actually used only to ensure a minimal continuity of the feature to be annotated within an annotation session and to make statistics about the annotation effort.
Password: the password for the group you belong to. It is related to the groupname and not to the username. It will be given to you by the contact person for your group.
Keyframes path: This field should contain the path of a local copy of the set of keyframes (e.g. Z:\tv2013\IACC.1 or /data/tv2013/IACC.1). If the path is valid and if your browser is properly configured (see below), a local copy of the keyframes will be used for display. We no longer support the version with online keyframe download since it is often too slow and unreliable. If the keyframes does not display properly, you may have specified a wrong path information and/or your browser is not properly configured.
Columns and Rows: These entries determine the number of rows and of columns of images that are displayed for annotation. This allows you to adapt the number of displayed images to your screen resolution.
Width and Height: These entries determine the size in pixels of images that are displayed for annotation. The default size is 224x126 pixels. You may choose larger values if you have a high resolution screen. You have to adapt both the number of images and the size of the images to the size of your screen. It may be better to have fewer but larger images. It is not so efficient to have too many images to annotate at once.

Using a local copy of the keyframes

The "keyframes" can be downloaded from the LIG site as a TGZ or ZIP archive (about 8.5GB). The "Keyframes path" field should not include the "keyframes" part (in the above example, the keyframes are in "Z:\tv2013\IACC.1\keyframes" or "/data/tv2013/IACC.1/keyframes"). The keyframes are distributed as a unix gzip-compressed tar file ("keyframes.tgz") or a winzip archive ("keyframes.zip"). You need to expand the archive so that the system can access its contents.

For (apparently good) security reasons, most recent versions of browsers do not accept to execute/display something with a "file://..." URL included in something with a "http//..." URL. You have to configure your browser so that it allows that, at least for pages/programs from "http//mrim.imag.fr/" (the collaboratiove annotation server). Here is how to proceed:

If you are using a recent version of Firefox (based on Mozilla 1.8 or higher):

(this will probably work for any Mozilla-based Web browser)

You have to locate your profile directory. It is normally something like: D:\Documents and Settings\<username>\Application Data\Mozilla\Firefox\Profiles\xxxxxxxx.default (XP) or D:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\xxxxxxxx.default (Vista) where xxxxxxxx is a randomly generated 8-character string.
You have to put in this directory the following user.js file. If you already have there a user.js file, you should rather append to it the content of the user.js file and if the already present file already have a similar section, you should simply edit it and add "http://mrim.imag.fr" to the list of authorized hosts. DO NOT execute user.js in your browser, just save it and copy it where appropriate or open it with a text editor to copy and paste its content where appropriate.

If you are using a less recent version of Firefox:

(this will probably work for any Mozilla-based Web browser)

Type about:config in the URL input
Double click on the line security.checkloaduri to set it to false

We strongly recommend however that you ugrade your version of Firefox instead since this operation will allow any host (not just mrim.imag.fr) to execute/display something with a "file://..." URL included in something with a "http//..." URL which is a potential security problem.

If you are using Internet Explorer:

Click on "Internet Options" in the "Tools" menu.
Click on "Security".
Click on "Trusted Sites".
Click on "Sites".
Type "http//mrim.imag.fr" into the textbox.
Click the Add button.
Uncheck the "Require server verification (https://)" box.
Click on "Close".
Click on "OK".

Annotation page

Once you click on "Login", you see the typical annotation page. You should choose "Rows" ans "Columns" values in the annotation page so that you display a maximum number of images on your screen without having to scroll. You can try different values and use the "Back" button of your browser to go back to the access page until you find the optimal values. Nothing will be stored if you do not click on the "VALIDATE" button.

An array of reduced size images with the chosen number of rows and columns is displayed on the right part of the screen. Each image comes with a set of three radio buttons labelled 'N', 'S' and 'P'. Selecting one of them corresponds to annotate the image as "Negative", "Skipped" or "Positive" for the target feature. By default, all images are selected as Negative as this is the most frequent case. There is also a right arrow on the right below each image. Clicking on it will play the associated shot in a separate window (not recommended if you have a slow Internet access). You will need a video player plugin able to display MPEG-4 files for TRECVID 2013. In order to ease the annotation, the image track of short shots has been slowed down by frame repilcation while the sound has not been slowed down but extended outside of the shot.

There are two ways of changing the annotation of an image. The first one is to check the corresponding button just below the image. The second one is to click on the image. This will change its annotation from "Negative" to "Positive", from "Positive" to "Skipped" or from "Skipped" to "Negative". This is because it is easier and faster to click on the image than on a radio button and because the first most frequent change is from "Negative" to "Positive".

When an image is annotated as "Negative", "Skipped" or "Positive", the background color of the corresponding cell is set to red, yellow or grenn respectively. This currently works fine with the Firefox browser but not with Internet Explorer (see the "Bugs" section).

An enlarged (full size) view of any reduced size image can be displayed in the upper left part of the screen simply by putting the mouse pointer over the reduced size image.

The feature to be annotated is displayed in the lower left part of the screen. All the displayed images have to be annotated for this feature.

There are two action buttons in the annotation page:

VALIDATE: Store the selected annotation for all the currently displayed images and go to the next series of images. It is very important to check and properly annotate all the displayed images before clicking on this button.If anything goes wrong (e.g. images are not displayed), please DO NOT hit this button and go back to the Login page.
EXIT: Terminates your current annotation session. The annotations of the currently displayed images are NOT stored. Please do not use the "Back" button of your browser afterwards to start new annotations but go through the http://mrim.imag.fr/tvca2013/al.html entry page again.

Warning page

Each time the system switches the feature to be annotated (and in the beginning), a warning message is displayed with two action buttons:

OK: Acknowledges the warning message and continue.
EXIT: Terminates your current annotation session. Please do not use the "Back" button of your browser afterwards to start new annotations but go through the http://mrim.imag.fr/tvca2013/al.html entry page again.

The warning message page was added because feature switches are quite frequent due to the active learning approach and they are easy to miss otherwise.

General

We are really interested into having a very high quality annotation since we know that both false positives and false negatives really harm supervised learning processes. Therefore, please take your time and, if you have any doubt about the annotation (if you are not sure of what you see or not sure that you fully understood the limits of the concept definition or ...), please prefer the "SKIP" button.

Please note that, in case of doubt, you can play the shot associated to a keyframe. It is almost always neccessary in some cases (like confimation of a positive "Singing"). The rule is that the shot is considered positive even if the feature is visible in the shot but not in the keyframe (as it is judged in TRECVID).

There is an Active Learning engine working in the background of the annotation system. Its goal is to select first for annotation the samples (keyframes) that are expected to be the most informative for systems' training. This is normally transparent to the users. The only expected effect is that we expect to see more positives in the early stages of the annotation (the effect is not always terrific...). This process is more efficient if annotations are done in small chunks. We therefore encourage several small sessions (typically 30-60 minutes or even shorter) separated by a few hours or one day rather than a continuous day of work. This is probably better for both the quality of the annotation and the comfort of the annotators. The minimum volume of annotations to be done by a single site should not exceed two or three full days of work but doing more is also encouraged.

Bugs

It sometimes happens that the server fails. The next page may not come or it may come with no image. If this happens, and only in this case, please use the "Back" button of your browser to go the previous page ant try again. We have seen this only when we use the dynamic loading of images. We recommend the use of a local copy of the keyframes when possible. This would also speed up the refresh.

It may happen that the system asks you again your groupname, username and password. In that case, just enter them again an then continue.

When an image is annotated as "Negative", "Skipped" or "Positive", the background color of the corresponding cell should be set to red, yellow or green respectively. This currently works fine with the Firefox browser but not with Internet Explorer (which indicate some error when clicking on the image but no further information seems available). If somebody knows how to fix that, we are interested (the javascript code is quite simple (use view source)).

Feedback to: Georges.Quenot@imag.fr