About the series

As you know, we at Humans in the Loop have a great love and appreciation of a well-designed annotation tool. After the great feedback on the reviews we published of our the best platforms on the market here and here, we decided that it’s time for a deep dive in some of our all-time favorites!

This article is the first from a series of 10 reviews which will be published each week. Soon we will be uploading the links to the other articles as they are released.

The whole series is based on the premise of transparency and honesty and none of these reviews are sponsored. They are just our way to give props to the best teams out there working on making annotation easier for AI teams, and to share some of the know-how that we have been accumulating over the past few years as a professional annotation company.

As in previous reviews, our parameters are: 

  • price
  • functions
  • project management
  • automation

If you have additional questions or want to get in touch with us to beta test or feature your tool in an upcoming article, feel free to email us at hello@humansintheloop.org!

The tool

Supervise.ly was developed as an internal tool by Deep Systemsan AI development company established in 2013 in Russia. In order to realize their AI projects such as creating self-driving car prototypes, a human pose estimation system, a road scene understanding system, etc. Deep Systems created Supervise.ly and made it publicly available in 2017.

The team is really responsive and always available to address technical issues. In addition, they have a slack for their entire community which is quite useful as a knowledge base and a quick tech support hotline.

The annotation suite is completely free and in addition to it, Supervise.ly offers an Enterprise version which includes AI-powered tools, an API and the option for on-premise hosting. 

Features

Annotating kiwis 🥝 and berries 🍓
The annotation interface is equipped with the best annotation tools.: from keypoints, polygons, and bounding boxes, to cuboids, polylines and bitmap brushes for efficient semantic segmentation. It also offers a very customizable tag/attribute function which can accept multiple choice, numbers, and text as input. Users can also customize how their labeling interface looks like and what hotkeys they prefer.

For semantic segmentation tasks, Supervise.ly has always been our tool of choice because its interface allows for layering of the annotated objects just like in graphic design tools – something which is missing in most platforms. Other helpful features include adjusting the opacity of the figures and cutting holes inside of them.

The recently released suite for Video labeling is quite impressive and enables annotating the same object from several camera angles, making it easier to track the object across cameras. Supervise.ly also has a 3D point cloud annotation suite.

Finally, we are always grateful for the availability of different formats in the ‘Import’ section which supports the most common formats, including DICOM images, and for the Data Transformation Language and Python Scripts which allow the user to transform the annotated data in various ways.

Project management

Look at all of those labeling jobs 🔥🔥🔥

The main reason why Supervise.ly has been our go-to platform for large-scale annotation projects is the extensive functions for project management. The new ‘Labeling at Scale’ feature allows for creating Labeling jobs and distributing work seamlessly among the annotators team, in addition to monitoring their progress and conducting QC.

 

The Labeling at Scale also includes Guides and Exams which allow project managers to create all of the instructions for labeling within the tool and pre-screen the annotators prior to onboarding them on a project. Another feature called ‘Issues’ has made inspection of issues on invalid images and objects really easy and has created a way to communicate about them within the tool until they are resolved. 

 

All of these functionalities work within a layered system for project management: starting from a Team which shares the same users, to a Workspace meant for different projects, and the different datasets within them. Being able to control user permissions on a granular level for all of these is great.

 

Some other features that make our lives easier: The ‘Data Commander’ which allows you to manage folders and files in an easier way, unlike other platforms; and the ‘Statistics’ which gives you a useful overview of the objects, tags, and images in your datasets – even though user-centered stats (like images per user, objects per user, handling time, etc) are still missing.

Automation

Visiting the model zoo 🦙🐍🦘

Quite frankly, we haven’t been using the Smart tool that Supervise.ly offers too much and we haven’t taken the opportunity to train it within the platform. Supervise.ly does offer however a really neat suite for training Neural Networks, including access to a Model Zoo of state-of-the-art models such as Mask R-CNN, YOLO v3, Faster-RCNN and other pre-trained models. 

 

Through Docker technology, users can also integrate their custom neural networks into Supervise.ly by implementing a simple API and putting their code into docker image.

Hope this was helpful! If you are working on an AI project and are currently reviewing which tool might be the most appropriate for it, get in touch with us and we would be happy to have a call and advise you on the best way to build your pipeline.