Back in 2018, Humans in the Loop published a review of the best annotation tools that we regularly use and the article was received with great enthusiasm by AI professionals and non-experts alike.

We were even contacted by several newly established platforms to beta test their tools and provide UX and UI feedback based on our hands-on experience managing and delivering large-scale annotation projects for AI. 

 

Since 2018, we have witnessed a lot of developments in the annotation platforms landscape, including the successful fundraising rounds of Labelbox which has solidified its position as a leader in the space, as well as the announcements of amazing new functionalities of our personal absolute favorite Supervise.ly which we still use for the majority of our projects. 

We thought it’s a good time to feature some of the most exciting new tools that have emerged in the past year. In the previous article, our evaluation parameters were:

  • price
  • functions
  • project management
but we feel that we have to add a new one:
 
  • automation

given that each one of the new tools is offering great new ways of optimizing the manual annotation process. 

If you have additional questions or want to get in touch with us to beta test or feature your tool in an upcoming article, feel free to email us at hello@humansintheloop.org!

crowd on the street automatic object detection
Object detection with bounding boxes (after annotating 10 similar images)

We have used Hasty.ai for several of our recent projects and both we and our clients love it. Based out of Germany, the team behind it is always open to feedback and is working closely with users to develop the best features suitable for their needs.

  1. Price: Still in open beta
  2. Functions: Supports both vector annotations (boxes and polygons) and pixel-wise annotation (with a brush). Users are able to upload both data and pre-generated labels, and export their data as a JSON file or PNG masks. The UI and design are quite similar to Neurala’s BrainBuilder which maybe served as an inspiration for the tool. 
  3. Project management: Offers great workflow management: images can be sorted by status (new, in progress, to review, done), and users can be assigned permissions in a really granular way. We love Secret Sauce number 1, which is the manual review panel where you can visualize each and every instance that has been labeled and sort them by labeler, status, or class. E.g. show all polygons labeled as “people”. It makes the review process really easy and we haven’t seen such a functionality anywhere else.
  4. Automation: The platform has several smart tools like GrabCut, Contour or Dextr that detect objects’ edges or contours which can be manually adjusted with a threshold setting so as to best segment the image. It supports Label prediction as well once enough data has been labeled. In addition, Secret Sauce number 2 is the ability to train your own Object detector, Semantic Segmentation and Instance segmentation. These work especially well with large scale predictable datasets where after a while the smart tools are activated and can be used to drastically reduce the amount of time spent per object. The only downside is that the processing takes a while (up to 10 or 20 seconds) which is time lost that could have been used to do actual labeling.
image of a surfer with semantic segmentation
Semantic segmentation using GrabCut
semantic segmentation of a girl photo
Semantic segmentation with 15 to 500 segments

Superannotate is a Silicon Valley startup with a large engineering presence in Armenia. The founder developed the technology behind it during his PhD in Computer Vision and the possibilities it offers for optimizing image segmentation are really impressive. 

  1. Price:Free for the first 100 images and for academic research, paid versions start with Starter (up to 10.000 images), Pro (unlimited images), and Enterprise (unlimited, custom).
  2. Functions: Offers both vector annotations (boxes, polygons, lines, ellipses, keypoints with templates and cuboids) and pixel-wise annotation with a brush. Supports both images and video. Also includes tons of other useful advanced polygon features, image filtering, tracking objects between frames, and shortcuts.  
  3. Project management: Superannotate has been delivering fully-managed annotation solutions for a while and has therefore created a variety of awesome functionalities for project management and quality control based on their own experience and pain points. These include different user permissions, the possibility to assign images for annotation and review, submitting comments on images and returning them for annotation, and even arbitration by an admin if the annotator and the QC disagree. 
  4. Automation: Arguably the best part of the tool is its superpixel functionality. It can detect the edges of objects with extremely high accuracy and that makes semantic and instance segmentation faster than with any other tool. The only problem is that if the boundaries between the object and the background are not clear, it ends up taking more time to play around with the segments than doing the actual work. There is an important pre-labeling functionality underway which will allow users to use one of Superannotate’s models or eventually their own (currently only available for COCO). Otherwise, users can just upload their own pre-generated annotations together with the images. 
semantic segmentation of a photo of a girl
Segmentation of a more complicated image
bounding boxes of an image of a crowd
Automatic bounding box detection

Picsellia is the brainchild of two French engineers who studied at ENS and set out to create a tool which would optimize the manual annotation process and make it 10 times faster. Their business model is quite different from the usual one that annotation tools use (priced per API call) and we can’t wait to see the updates that they have in store for further improvements of label prediction!

  1. Price: $100/month for the use of the platform. There is also a separate pricing for API calls (0.02€/call) for users who want to integrate smart tools in their own UI
  2. Functions: Supports vector annotations (boxes, polygons, points, and lines).  The data can be exported (either all annotations or just the approved ones) as a JSON file. The UI and design seem to have been borrowed entirely from Labelbox but that hopefully will take a life of their own once the platform develops.   
  3. Project management: The platform supports all basic operations and has a nice dashboard for monitoring activity on the labeling project. You can create users with different permissions, assign different labeling jobs to them, and review the annotations using a review panel. We are sure more nice features will be added soon! 
  4. Automation: The reason we decided to include Picsellia in this list is its enormous potential for automatic prelabeling using a custom model. For now, the default model for all predictions is COCO and it does fare quite well with bounding box prediction. The only issue is that the tool itself cannot automatically assign classes to objects even if it detects them correctly. The automatic segmentation is a bit harder but you can edit the segmentation using a brush and an eraser – the results are not extremely precise but it does the job. When you click on “Enable Assistance” you can use 4 extreme points which will give you Dextr prediction of the object. Right click and “switch prediction” will give you GrabCut. You definitely need to experiment a bit to see which tool works best for your data but sometimes these work quite well.
semantic segmentation on a photo of a woman
Automatic segmentation
video annotation of a girl with feathers

Another one of our favorites, Diffgram has been around for a while now and is focused on providing not only an annotation tool but a complete training data management system. The founder is based out of California and is fully dedicated to helping companies where companies manage their entire training data process.

  1. Price: Has a free trial version, Explorer and Teams versions priced monthly at $398 and $994 monthly 
  2. Functions: Offers vector annotations (boxes, polygons, and lines) and is the only tool from this list which specializes in video labeling. The formats supported are JSON and YAML. One downside of the tool is that its UI is not very intuitive at first and it might be confusing to navigate.    
  3. Project management: Has some great features for workforce management, including assigning tasks automatically by creating batches of work, creating exams with awards for annotators to pass prior to starting the real annotation, as well as creating and managing datasets within the platform. You are able to import or export the data through the online interface, API, or SDK access.  
  4. Automation: If you are interested in using label predictions, you can upload the ready labels together with the images with Diffgram’s assistance. Diffgram is not using FAN anymore to generate predictions within the platform but the video interpolation is still a main feature they support. Essentially, users can annotate just a few keyframes and the tool will interpolate all of the frames in between them. The interpolation process can be rather slow and not completely accurate but does a good job for simple object tracking.

Hope this was helpful! If you are working on an AI project and are currently reviewing which tool might be the most appropriate for it, get in touch with us and we would be happy to have a call and advise you on the best way to build your pipeline.