Rules

Goal: The development of tissue trackers for stereo endoscopic videos.

The SurgT challenge has been created for the purpose of exploring tracking methods for surgical scenes, where there is limited annotated data available. The participants of this challenge are expected to develop tracking methods, which can learn to perform accurate keypoint tracking on surgical tissue, without the use of human-annotated endoscopic videos. This challenge encourages the development of self-supervised tracking methods, and also generalised tracking methods which have been trained on annotated non-surgical scenes. Traditional computer vision methods are also welcomed.

For this challenge, the participants are asked to develop tracking methods for stereo endoscopic videos, as shown in the image above. More info about the data can be found in SurgT Data.

The participants are expected to: (1) create a docker image; (2) download the SurgT_benchmarking GitHub repository; (3) replace the sample_tracker.py with the developed self-supervised tracker.

The benchmarking tool is used to assess the tracker during the development stage and for getting the final results. The final results will be calculated on the test set, which will be hidden until the end of the challenge. The trackers are ranked according to their ability to track a bounding box throughout the video sequence. Specifically, the submissions will be ranked by computing the EAO (Expected Average Overlap) score, similar to what is used in the VOT Challenge.

At the end of the challenge, results will be published in a joint publication. In this publication, the developed methods will be described and all the members of the teams will be made authors. The participating teams may publish their own results separately only after the joint publication is published. The developed methods, including code and models, are to be made publicly available online at the end of the challenge.

Submission

Submissions are open from now until the end of the submission period (September 1st at 14:00 ET).

How to create a docker image?

  1. Download the docker_submission/ folder from the GitHub link: SurgT_challenge
  2. Create an environment requirements file docker_submission/surg_t_requirements.txt, to specify the libraries you used:
    For conda: conda env export > surg_t_requirements.txt
    For virtualenv: pip freeze > surg_t_requirements.txt
  3. Edit line 60 in the Dockerfile to upload your code to the docker image
    Option 1: copy SurgTbenchmarking folder (containing your code) from your local computer to the docker image using: COPY [path/to/your/SurgT_benchmarking] /home/newuser/surg_t/
    Option 2: upload the SurgT
    benchmarking folder (containing your code) on GitHub and use: RUN git clone https://github.com/YOUR_GIT_USERNAME/SurgT_benchmarking.git /home/newuser/surg_t/
  4. Run the following commands to build Docker and test it is working: sh build.sh , sh start.sh
  5. Compress docker image for uploading : docker save -o surgt.tar surgt:latest
  6. Upload the docker image .tar file to any remote file storage of choice Google Drive, Box, DropBox...;

Please send an email to jmc19@ic.ac.uk, containing the following information:

  1. Please send us a link to your docker image containing the latest version of [SurgT_benchmarking](https://github.com/Cartucho/SurgT_benchmarking)  configured to your algorithm. Make sure that the file src/evaluate.py is updated; Find instructions on how to create the docker image above;
  2. Team members (names + email) - One submission per team;
  3. Please attach a summary to be used in the final joint publication:
    1. 1 paragraph describing the method with references (example from VOT challenge)
    2. 1 paragraph describing the datasets used for training
At the end of the submission period, the latest received email from each team will be used to download the docker image. The winning team and runner-up will be announced at the MICCAI challenge's event.

FAQ

  1. Can I manually label the training data?

    No, you cannot label and cannot use any annotated surgical data. The only labelled data that can be used are the non-medical datasets that are publicly available online, e.g. Visual Tracker Benchmark Dataset, VOT 2021 Dataset.

  2. Can I train on my own data?

    Private datasets cannot be used. Public datasets can be used, except for data from other EndoVis challenges, and endoscopic data from the Hamlyn Centre for Robotic Surgery. Participants will be required to declare all datasets that have been used, during submission.

  3. Can I train on the validation ground truth?

    No, the validation ground truth can only be used to assess your method during development.

  4. Can I develop a monocular tracking method, instead of stereo?

    Yes, as exemplified in sample_tracker.py a monocular tracker can be used to track the bounding box on the left and right image separately.

  5. Are the images in the test set rectified?

    Yes, similarly to the validation dataset the images in the test set are rectified. So,  the left bounding box will always be centred along the same row as the right bounding box. In the training set, we also provide the stereo calibration parameters, which can be used to rectify the provided training videos.

  6. How will I submit my method at the end of the challenge?

    You will send the docker image to the organizers. In your docker image, SurgT_benchmarking  should have already been downloaded and running with your method. Simply, replace the sample_tracker.py with your own method. At the end of the challenge, we will load your docker image, and update the config.yaml file with the links to the test data and assess your method.

  7. Can I use pre-trained networks?
    Yes, pre-trained networks are allowed in this challenge.

Summary

  • Manual annotation of the training/development data is not allowed;
  • Validation ground truth cannot be used for training;
  • Publicly available non-surgical datasets are allowed, even with ground truth labels;
  • Docker image with SurgT_benchmarking is required;
  • Ranking by EAO score;
  • Results & Methods will be published in a joint paper;
  • Developed methods are to be made publicly available online.