ActEV Sequestered Data Leaderboard

The ActEV SDL evaluation is based on the Multiview Extended Video with Activities (MEVA) dataset (mevadata.org) collected at the Muscatatuck Urban Training Center with a team of over 100 actors performing in various scenarios. The data was built by the Intelligence Advanced Research Projects Activity (IARPA) Deep Intermodal Video Analytics (DIVA) program to support activity detection in multi-camera environments for both DIVA performers and the broader research community.
The MEVA dataset has two parts: the public training and development data and sequestered evaluation data used only by NIST to test systems. The data is accompanied by activity annotations.

The MEVA data GIT repo is the data distribution mechanism for MEVA Related annotations and documentation. The repo presently consists of schemas for the activity annotations https://gitlab.kitware.com/meva/meva-data-repo.

The ActEV data GIT repo, is the data distribution mechanism for the ActEV evaluation. The repo presently consists of a collection of corpora and partition definition files to be used for the evaluations https://gitlab.kitware.com/actev/actev-data-repo.

We have updated the activity names for the SDL. We intend for these names to be used for the duration of the DIVA program. The list below shows the "ActEV 2020 SDL Activity Name" and "ActEV 2019 SDL Activity Name" of the 37 activities to be detected for the ActEV SDL evaluation. Detailed activity definitions are in the ActEV Annotation Definitions for MEVA Data document. Note that current scoring for the SDL is based exclusively on activity detection, and object detection is not considered. The link to the activity-name-mapping.csv .

In the SDL evaluation, there is one Activity Detection (AD) task for detecting and temporally localizing activities.

System delivery to the leaderboard must be in a form compatible with the ActEV Command Line Interface (ActEV CLI) and submitted to NIST for testing. The command line interface implementation that you will provide formalizes the entire process of evaluating a system, by providing the evaluation team a means to: (1) download and install your software via a single URL, (2) verify that the delivery works AND produces output that is “consistent” with output you produce, and (3) process a large collection of video in a fault-tolerant, parallelizable manner.

To complete this task you will need the following items described in detail below:

1. FAQ - Validation Phase Processing
2. CLI Description
3. The Abstract CLI Git Repository
4. The CLI Implementation Primer
5. The Validation Data Set
6. Example CLI-Compliant Implementation
7. NIST Hardware and Initial Operating System Description
8. SDL Submission Processing Pipeline

1. FAQ - Validation Phase Processing

The ActEV SDL - Validation Phase Processing

2. CLI Description

The ActEV CLI description

3. The Abstract CLI GIT Repository

The Abstract CLI GIT repository. The repo contains documentation.

4. The CLI Implementation Primer

There are 6 steps to adapt your code to the CLI. The ActEV Evaluation CLI Programming Primer describes the steps to clone the Abstract CLI and begin adapting the code to your implementation.

5. The Validation Data Set

As mentioned above, the CLI is used to verify the downloaded software is correctly installed and produces the same output at NIST as you produce locally. In order to do so, we have provided a small validation data set (ActEV-Eval-CLI-Validation-Set3) as part of the ActEV SDL Dataset that will be processed both at your site and at NIST. Please use this data in Step 3 of the Primer described above.

6. Example CLI-Compliant Implementation

The links below provide two example CLI implementations for the leaderboard baseline algorithm:

• RC3D baseline system with CLI implementation see for details
• Framework based RC3D baseline system with CLI implementation see for details.

7. NIST Independent Evaluation Infrastructure Specification

NIST will begin installing your system from a fresh Ubuntu 18.04 cloud image available from https://cloud-images.ubuntu.com/releases/18.04/release/ on the following hardware.

• Chassis: Asus ESC4000 G4S
• CPU: 2 x Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz (12 cores/CPU)
• Motherboard: Asus Intel® C621 PCH chipset
• HDD/SSD: 2x 1.92GB Intel SSD DC S4500
• RAM: 12x 16GB DDR4-2400 ECC RDIMM
• GPU: Four PNY RTX2080Ti blower style
• OS: Ubuntu 18.04
• Storage Volume- 1TB (variable)
• Supplied object store (read only) for source video

8. SDL Submission Processing Pipeline

There are three stages for the submission processing pipeline. They are:

• Validation: during this stage we run through each of the ActEV CLI commands to install your system, run the system on the validation set, compare the produced output to the validation set, and finally, take a snapshot of your system to re-use during execution.
• Execution: during this stage, we use the snapshot to process the sequestered data. Presently, we can divide the sequestered data into 1-hour sub-parts of the data set or process the whole dataset. Each sub-part is nominally 12, 5-minute files. We are only processing MEVA data through your system. Presently, we have a runtime limit per part and if a part fails to be processed, we retry it once.
• Scoring: After all the parts have been processed, the outputs are merged, and scored.

Datasets

• ActEV-supported data sets
  • Multiview Extended Video with Activities (MEVA)
  • VIRAT
  • Data access from mevadata.org: Accessing and using MEVA and MEVA Download Instructions (This includes, 4.6 hr UAV video collected, 3D model of the facility)
  • Data access : See actev-data-repo. Access credentials provided during signup
• Kinetics
• AVA
• Moments-in-Time
• ActivityNet
• NVIDIA's CityFlow dataset

Framework

• DIVA Framework Github Repository This is the main DIVA Framework site, all development of the framework happens here.
• DIVA Framework Issue Tracker Submit any bug reports or feature requests for the framework here.
• DIVA Framework Main Documentation PageThe source for the framework documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction is the UseCase section which will walk you though a number of typical use cases with the framework.

• KWIVER Github Repository This is the main KWIVER site, all development of the framework happens here.
• KWIVER Issue Tracker Submit any bug reports or feature requests for the KWIVER here. If there's any question about whether your issues belongs in the KWIVER or DIVA framework issues tracker, submit to the DIVA tracker and we'll sort it out..
• KWIVER Main Documentation Page The source for the KWIVER documentation is maintained in the Github repository using Sphinx. A built version is maintained on ReadTheDocs at this link. A good place to get started in the documentation, after reading the Introduction are the Arrows and Sprokit sections, both of which are used by the KWIVER framework.

Baseline Algorithms

• R-C3D
• ACT

Annotation Tools

• Kitware annotation tool (the tool natively supports the DIVA format)
• The VGG Image Annotator
• Scalabel (used for annotation of Berkeley DeepDrive project)
• VATIC - Video Annotation Tool
• BeaverDam
• VoTT: Visual Object Tagging Tool
• Computer Vision Annotation Tool (CVAT)
• Efficient Annotation of Segmentation Datasets with Polygon-RNN++

For ActEV Evaluation information (data, evaluation code, etc.) please email: actev-nist@nist.gov

For ActEV Evaluation Discussion Please Visit our Google Group.