About the series

As you know, we at Humans in the Loop have a great love and appreciation of a well-designed annotation tool. After the great feedback on the reviews we published of our the best platforms on the market here and here, we decided that it’s time for a deep dive in some of our all-time favourites! We will be publishing one review every week.

This is the second review in our series of 10 reviews. Our first review on Supervise.ly can be found here.

The whole series is based on the premise of transparency and honesty and none of these reviews are sponsored. They are just our way to give props to the best teams out there working on making annotation easier for AI teams, and to share some of the know-how that we have been accumulating over the past few years as a professional annotation company.

As in previous reviews, our parameters are:

  • price
  • functions
  • project management
  • automation

If you have additional questions or want to get in touch with us to beta test or feature your tool in an upcoming article, feel free to email us at hello@humansintheloop.org

TrainingData.io

Screenshot_293

Trainingdata.io is a tool launched in 2019 which specializes above all in the medical imagery domain even though it can be used for all other purposes as well. The founder who is an ex-Netflix employee is based out of Palo Alto and is working closely with early clients to build a tool that works exceptionally well for their needs. 

In terms of pricing, the free version allows 2500 images per year and up to 3 collaborators, while the Pro and Enterprise version are more suitable for those interested in unlimited assets, API access, on-premise data, and other features.

Features

Annotating COVID-19 imagery 👩‍⚕️👨‍⚕️

The tool supports both vector annotations (boxes and polygons) and pixel-wise annotation. It offers some really neat parameters, such as superpixel segmentation, brushes of different shapes, using a magnifier, drawing polygons with a freehand outline and then sculpting them intuitively from the outside and inside, etc. The platform even has settings for disallowing invalid annotations, such has polygon lines running over each other. 

Trainingdata supports standard JSON annotation formats and PNG masks, and is the only tool on our list which allows for easy annotation of DICOM imagery in layers. 

The whole tool can be installed on-premise using Docker (even though we personally have had quite a lot of problems with the installation process!) which guarantees the security and privacy of the data. Another feature that ensures privacy is the importing images and clips from local host.

Project management

Ultimate consensus and queuing features 💪

The best features of Trainingdata.io in terms of project management are related to annotator performance monitoring. For quality control purposes, admins are able to rate each individual label or image on a scale of 1 to 10 and then analyse annotator aggregate performance.  

The platform has another distinctive feature that allows admins to decide the queuing strategy for the annotation and review of images. Admins can also set the ‘Duplication’ element that will send the images in a queued fashion to different annotators for consensus-based labeling. 

Quality work is guaranteed by the platform as it allows you to set a standard for your annotated images. Images can be checked against that standard under the ‘Benchmark’ feature. The ‘Review’ tool has a comment feature to enable coordination between different annotators and reviewers.   

Automation

Model library 📚

TrainingData.io is currently partnering with NVIDIA to deliver automatic pre-labeling using several medical imagery models (spleen segmentation, liver segmentation, etc). It also features several other models (Mask RCNN, YOLOv3) for annotation prediction of common classes.

The platform is also customer friendly as users can import their own ML models and use Active Learning for generating label predictions, as well as Transfer Learning between models. This is one of the most exciting functions of the platform and it’s promising to reduce labeling time considerably, especially on large-scale projects!

And finally, the tool also supports video annotation with interpolation/tracking, and is currently building its point cloud annotation suite. Overall, exciting times coming up for Trainingdata.io.

Hope this was helpful! If you are working on an AI project and are currently reviewing which tool might be the most appropriate for it, get in touch with us and we would be happy to have a call and advise you on the best way to build your pipeline.