Assessment

There are two tracks for each of the three tasks, one track for CTA modality and one track for MRA modality. The assessment of algorithms will be the same for both CTA and MRA modalities, and with the following metrics:

Metrics for Task-1-CoW-Segmentation

Seven evaluation metrics with equal weights for the multi-class (CoW vessels) segmentation task:

1. Class-average Dice similarity coefficient
2. Centerline Dice (clDice) on merged binary mask
3. Class-average 0-th Betti number error
4. Class-average Hausdorff Distance 95% Percentile (HD95)
5. Average F1 score (harmonic mean of the precision and recall) for detection of the "Group 2 CoW components" (as coined in our summary pre-print):
   • Acom
   • Pcoms
   • 3rd-A2
6. Variant-balanced graph classification accuracy
   • Same as one of the metrics for Task-3!
   • Classify the graph by detecting the edges
7. Variant-balanced topology match rate (as introduced in our summary pre-print)
   • Not only do the CoW components need to be correctly detected
   • Topology extracted from segmentation should also satisfy strict connectivity and neighborhood conditions:
     • Correct neighbourhood connectivity (connected to correct vessel classes)
     • No 0-th Betti number errors
     • No topological mistakes in connectivity, fragmentation, or crossover
   • Not trivial to get a match in our topology matching analysis
   • A more advanced and stringent metric than detection (metric #5) and graph classification (metric #6)
     • as it incorporates the detection and classification performance in its evaluation

Metrics for Task-2-CoW-ObjDet

For the CoW object detection task, we ask your algorithm to only provide one box per image, which is then compared with our ground-truth box on one metric:

1. Boundary intersection over union (Boundary IoU)

Please refer to this entry from "Metrics Reloaded" for more information on Boundary IoU.

Metrics for Task-3-CoW-Classification

Task 3 is to classify the CoW variant topology graph, which are represented by a graph with relevant anterior and posterior edge lists. The presence and absence of the edge of the edge list determine the topology graph class.

Your algorithm is expected to output the predicted anterior and posterior edge lists. We evaluate both lists using two metrics with equal weights:

1. Variant-balanced accuracy
2. Distance between ground-truth and predicted 4-element vectors

Evaluation Code

Whatever we use for the challenge evaluation will be released and synchronized to the following repo in a transparent manner. Please refer to this repo for our assessment implementations.

Please feel free to leave an issue or let us know if you have further feedback or questions.

Further Readings

• Yang, Kaiyuan, et al. "Benchmarking the cow with the topcow challenge: Topology-aware anatomical segmentation of the circle of willis for cta and mra." ArXiv (2023).
• Shit, Suprosanna, et al. "clDice-a novel topology-preserving loss function for tubular structure segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
• Stucki, Nico, et al. "Topologically faithful image segmentation via induced matching of persistence barcodes." International Conference on Machine Learning. PMLR, 2023.
• Menten, Martin J., et al. "A skeletonization algorithm for gradient-based optimization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
• Vos, Iris, Ynte Ruigrok, and Hugo Kuijf. "Results of the CROWN challenge on automated assessment of circle of Willis morphology." Medical Imaging with Deep Learning. 2024.
• Maier-Hein, Lena, et al. "Metrics reloaded: recommendations for image analysis validation." Nature methods 21.2 (2024): 195-212.