Temporal Segment Networks:
Towards Good Practices for Deep Action Recognition

ECCV 2016

Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, Luc Van Gool


Introduction

An overview of TSN framework


Experimental Results

We conducted action recognition experiments on two standard benchmark datasets: UCF101 and HMDB51. Click dataset names to see detailed results.

* We do not use iDT in our experiments.
Modality RGB Flow Warp Flow RGB + Flow
(1:1.5)
RGB + Flow + Warp Flow
(1:1:0.5)
HMDB51 51.0% 64.2% 63.0% 68.5% 69.0%
UCF101 85.1% 89.7% 89.8% 94.0% 94.2%
Split 1 Split 2 Split 3 Average of 3 Splits
RGB 54.4% 50.0% 49.2% 51.0%
Flow 62.4% 63.3% 63.9% 63.2%
RGB + Flow (1:1.5) 69.5% 67.4% 68.5% 68.5%
* Note: it is fine to observe small differences in accuracy numbers of single streams, while the combined performance will be very stable.
Split 1 Split 2 Split 3 Average of 3 Splits
RGB 85.5% 84.9% 84.5% 85.1%
Flow 87.6% 90.2% 91.3% 89.7%
RGB + Flow (1:1.5) 93.5% 94.3% 94.5% 94.0%
* Note: it is fine to observe small differences in accuracy numbers of single streams, while the combined performance will be very stable.

Real-Time Action Recognition
The RGB diff modality is obtained by stacking difference frames of five continuous RGB frames. It requires no optical flow extraction, which is preferable for real-time action recognition.
Speed (on GPU) UCF101 Split 1 UCF101 3 Splits Average
Enhanced MV [1] 390 FPS 86.6% 86.4%
Two-stream 3Dnet [2] 246 FPS - 90.2%
RGB Diff w/o TSN 660FPS 83.0% N/A
RGB Diff + TSN 660FPS 86.5% 87.7%
RGB Diff + RGB (both TSN) 340 FPS 90.7% 91.0%

Incresing Segment Number (post-conference results)
In the ECCV publication we set the segment number to 3. Regarding requests from many users, here we evaluate the effect of increasing the segment number to 5 and 7 on UCF101.
Segment Number RGB Flow RGB + Flow (1:1.5) RGB + Flow (1:1)
3 85.1% 89.7% 94.0% 94.0%
5 85.1% 89.7% 94.5% 94.6%
7 85.4% 89.6% 94.6% 94.8%
9 85.3% 89.6% 94.8% 94.9%
[1] "Real-time Action Recognition with Enhanced Motion Vector CNNs", Bowen Zhang, et. al., CVPR 2016.
[2] "Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification", Ali Diba, et. al., Arxiv Preprint.

Code

The Python codes and trained models are release as a full-fledged action recognition toolbox on Github.

open_in_new Temporal Segment Network

Visualization

We visualized the parameters learned in CNNs using the DeepDraw toolbox.
Visualizing model parameters for five classes

open_in_new More Visualization Results

Citation

Please kindly cite the following paper if you find this project helpful.
@inproceedings{TSN2016ECCV,
  author    = {Limin Wang and Yuanjun Xiong and Zhe Wang
               and Yu Qiao and Dahua Lin and Xiaoou Tang and Luc {Val Gool}},
  title     = {Temporal Segment Networks: Towards Good Practices for Deep Action Recognition},
  booktitle   = {ECCV},
  year      = {2016},
}
                  

Related Projects

Contact