HIPIE: Hierarchical Open-vocabulary Universal Image Segmentation

XuDong Frank Wang1*, Shufan Li1*, Konstantinos Kallidromitis2*, Yusuke Kato2, Kazuki Kozuka2, Trevor Darrell1
1UC Berkeley 2Panasonic AI Research
Paper arXiv Code

We present HIPIE, a novel HIerarchical, oPen-vocabulary and unIvErsal image segmentation and detection model that is capable of performing segmentation tasks at various levels of granularities (whole, part and subpart) and tasks, including semantic segmentation, instance segmentation, panoptic segmentation, referring segmentation, and part segmentation, all within a unified framework of language-guided segmentation.

Abstract

Open-vocabulary image segmentation aims to partition an image into semantic regions according to arbitrary text descriptions. However, complex visual scenes can be naturally decomposed into simpler parts and abstracted at multiple levels of granularity, introducing inherent segmentation ambiguity. Unlike existing methods that typically sidestep this ambiguity and treat it as an external factor, our approach actively incorporates a hierarchical representation encompassing different semantic-levels into the learning process. We propose a decoupled text-image fusion mechanism and representation learning modules for both “things” and “stuff”. Additionally, we systematically examine the differences that exist in the textual and visual features between these types of categories. Our resulting model, named HIPIE, tackles HIerarchical, oPen-vocabulary, and unIvErsal segmentation tasks within a unified framework. Benchmarked on diverse datasets, e.g., ADE20K, COCO, Pascal-VOC Part, and RefCOCO/RefCOCOg, HIPIE achieves the state-of-the-art results at various levels of image comprehension, including semantic-level (e.g., semantic segmentation), instance-level (e.g., panoptic/referring segmentation and object detection), as well as part-level (e.g., part/subpart segmentation) tasks.

Tasks: Open-vocabulary Hierarchical Universal Image Segmentation

We consider all relevant tasks under the unified framework of language-guided segmentation, which performs open-vocabulary segmentation and detection tasks for arbitrary text-based descriptions.

HIPIE can provide segmentation masks at various granularities (whole, part and subpart) and perform referring segmentation using a single model.

Integrating HIPIE into SAM for Class-aware Image Segmentation on SA-1B


HIPIE is capable of labeling all segmentation masks from SAM and can even identify additional masks that may have been overlooked by SAM.

More Demos on Hierarchical Image Segmentation

Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.

Framework

Interpolate start reference image.

Diagram of HIPIE for hierarchical, universal and open-vocabulary image segmentation and detection. Please check our paper for more details on the proposed method.

BibTeX

@misc{wang2023hierarchical,
  title={Hierarchical Open-vocabulary Universal Image Segmentation}, 
  author={Xudong Wang and Shufan Li and Konstantinos Kallidromitis and Yusuke Kato and Kazuki Kozuka and Trevor Darrell},
  year={2023},
  eprint={2307.00764},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}