[译]Segment Anything

Segment Anything 项目提出「可提示分割」任务并发布 SAM 模型与 SA-1B 数据集:11M 图像、1.1B 掩码,通过提示工程零样本迁移到边缘检测、目标提议、实例分割等任务。

原文地址:Segment Anything
官方实现:facebookresearch/segment-anything

摘要

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive – often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.

我们介绍 Segment Anything(SA)项目:一个用于图像分割的新任务、新模型和新数据集。在数据收集循环中使用我们的高效模型,我们构建了迄今规模最大的分割数据集(远超以往),在 11M 张经授权且尊重隐私的图像上包含超过 10 亿个掩码。该模型被设计并训练为可提示(promptable)的,因此可以零样本迁移到新的图像分布和任务。我们在众多任务上评估其能力,发现其零样本性能令人印象深刻——通常与先前的全监督结果相当甚至更优。我们在 https://segment-anything.com 发布 Segment Anything 模型(SAM)和相应的数据集(SA-1B,1B 掩码、11M 图像),以促进计算机视觉基础模型的研究。

引言

Large language models pre-trained on web-scale datasets are revolutionizing NLP with strong zero-shot and few-shot generalization [10]. These “foundation models” [8] can generalize to tasks and data distributions beyond those seen during training. This capability is often implemented with prompt engineering in which hand-crafted text is used to prompt the language model to generate a valid textual response for the task at hand. When scaled and trained with abundant text corpora from the web, these models’ zero and few-shot performance compares surprisingly well to (even matching in some cases) fine-tuned models [10, 21]. Empirical trends show this behavior improving with model scale, dataset size, and total training compute [56, 10, 21, 51]. Foundation models have also been explored in computer vision, albeit to a lesser extent. Perhaps the most prominent illustration aligns paired text and images from the web. For example, CLIP [82] and ALIGN [55] use contrastive learning to train text and image encoders that align the two modalities. Once trained, engineered text prompts enable zero-shot generalization to novel visual concepts and data distributions. Such encoders also compose effectively with other modules to enable downstream tasks, such as image generation (e.g., DALL·E [83]). While much progress has been made on vision and language encoders, computer vision includes a wide range of problems beyond this scope, and for many of these, abundant training data does not exist. In this work, our goal is to build a foundation model for image segmentation. That is, we seek to develop a promptable model and pre-train it on a broad dataset using a task that enables powerful generalization. With this model, we aim to solve a range of downstream segmentation problems on new data distributions using prompt engineering. The success of this plan hinges on three components: task, model, and data. To develop them, we address the following questions about image segmentation:

在网页级数据集上预训练的大语言模型正以强大的零样本和少样本泛化能力革新 NLP [10]。这些「基础模型」[8] 可以泛化到训练期间未见过的任务和数据分布。这种能力通常通过提示工程(prompt engineering)实现:手工构造的文本提示语言模型为当前任务生成有效的文本回复。当用来自网页的充足文本语料进行规模化训练后,这些模型的零样本和少样本性能与微调模型相比出奇地好(在某些情况下甚至持平)[10, 21]。经验趋势表明,这种行为随模型规模、数据集大小和总训练计算量的增加而改善 [56, 10, 21, 51]。基础模型在计算机视觉中也有探索,尽管程度较低。最突出的例子可能是对齐来自网页的成对文本和图像。例如,CLIP [82] 和 ALIGN [55] 使用对比学习训练文本和图像编码器以对齐两种模态。训练完成后,工程化的文本提示能够实现对新视觉概念和数据分布的零样本泛化。这类编码器还能与其他模块有效组合以支持下游任务,例如图像生成(如 DALL·E [83])。虽然视觉和语言编码器已取得很大进展,但计算机视觉包含超出此范围的众多问题,其中许多并不存在充足的训练数据。在这项工作中,我们的目标是为图像分割构建一个基础模型。也就是说,我们试图开发一个可提示(promptable)的模型,并使用一个能带来强大泛化能力的任务在广泛数据集上预训练它。借助这个模型,我们旨在通过提示工程在新数据分布上解决一系列下游分割问题。这一计划的成功取决于三个组成部分:任务、模型和数据。为了开发它们,我们解决以下关于图像分割的问题:

  1. What task will enable zero-shot generalization?
  2. What is the corresponding model architecture?
  3. What data can power this task and model?
  1. 什么任务能够实现零样本泛化?
  2. 对应的模型架构是什么?
  3. 什么数据能够支撑这个任务和模型?

These questions are entangled and require a comprehensive solution. We start by defining a promptable segmentation task that is general enough to provide a powerful pre-training objective and to enable a wide range of downstream applications. This task requires a model that supports flexible prompting and can output segmentation masks in real-time when prompted to allow for interactive use. To train our model, we need a diverse, large-scale source of data. Unfortunately, there is no web-scale data source for segmentation; to address this, we build a “data engine”, i.e., we iterate between using our efficient model to assist in data collection and using the newly collected data to improve the model. We introduce each interconnected component next, followed by the dataset we created and the experiments that demonstrate the effectiveness of our approach.

这些问题相互纠缠,需要一套全面的解决方案。我们首先定义一个可提示分割(promptable segmentation)任务,该任务足够通用,既能提供强大的预训练目标,又能支持广泛的下游应用。这个任务要求模型支持灵活的提示方式,并且在被提示时能够实时输出分割掩码以支持交互式使用。为了训练我们的模型,我们需要多样化、大规模的数据来源。不幸的是,分割任务没有网页级的数据来源;为了解决这个问题,我们构建了一个「数据引擎」,即在「使用高效模型辅助数据收集」和「使用新收集的数据改进模型」之间迭代。接下来我们逐一介绍每个相互关联的组件,随后介绍我们创建的数据集以及证明方法有效性的实验。

Task (§2). In NLP and more recently computer vision, foundation models are a promising development that can perform zero-shot and few-shot learning for new datasets and tasks often by using “prompting” techniques. Inspired by this line of work, we propose the promptable segmentation task, where the goal is to return a valid segmentation mask given any segmentation prompt (see Fig. 1a). A prompt simply specifies what to segment in an image, e.g., a prompt can include spatial or text information identifying an object. The requirement of a valid output mask means that even when a prompt is ambiguous and could refer to multiple objects (for example, a point on a shirt may indicate either the shirt or the person wearing it), the output should be a reasonable mask for at least one of those objects. We use the promptable segmentation task as both a pre-training objective and to solve general downstream segmentation tasks via prompt engineering.

任务(§2)。在 NLP 以及近来的计算机视觉中,基础模型是一个有前景的发展方向,通常通过「提示」技术对新数据集和新任务执行零样本和少样本学习。受这类工作的启发,我们提出可提示分割任务:给定任意分割提示,返回一个有效的分割掩码(见 Fig. 1a)。提示简单地指定了图像中要分割什么,例如提示可以包含标识物体的空间或文本信息。「有效」输出掩码的要求意味着,即使提示存在歧义、可能指向多个物体(例如,衬衫上的一个点可能表示衬衫,也可能表示穿衬衫的人),输出也应当是其中至少一个物体的合理掩码。我们将可提示分割任务同时用作预训练目标,以及通过提示工程解决通用下游分割任务的方法。

Model (§3). The promptable segmentation task and the goal of real-world use impose constraints on the model architecture. In particular, the model must support flexible prompts, needs to compute masks in amortized real-time to allow interactive use, and must be ambiguity-aware. Surprisingly, we find that a simple design satisfies all three constraints: a powerful image encoder computes an image embedding, a prompt encoder embeds prompts, and then the two information sources are combined in a lightweight mask decoder that predicts segmentation masks. We refer to this model as the Segment Anything Model, or SAM (see Fig. 1b). By separating SAM into an image encoder and a fast prompt encoder / mask decoder, the same image embedding can be reused (and its cost amortized) with different prompts. Given an image embedding, the prompt encoder and mask decoder predict a mask from a prompt in ∼50ms in a web browser. We focus on point, box, and mask prompts, and also present initial results with free-form text prompts. To make SAM ambiguity-aware, we design it to predict multiple masks for a single prompt allowing SAM to naturally handle ambiguity, such as the shirt vs. person example.

模型(§3)。可提示分割任务和现实使用的目标给模型架构带来了约束。具体来说,模型必须支持灵活的提示,需要以摊销的实时速度计算掩码以支持交互式使用,并且必须具备歧义感知(ambiguity-aware)能力。令人惊讶的是,我们发现一个简单的设计就能同时满足这三个约束:一个强大的图像编码器计算图像嵌入,一个提示编码器编码提示,然后两个信息源在一个轻量的掩码解码器中结合以预测分割掩码。我们将这个模型称为 Segment Anything Model,即 SAM(见 Fig. 1b)。通过将 SAM 分为图像编码器和快速的提示编码器 / 掩码解码器,同一个图像嵌入可以在不同提示之间复用(其成本被摊销)。给定图像嵌入,提示编码器和掩码解码器可以在网页浏览器中约 50ms 内根据提示预测掩码。我们重点关注点、框和掩码提示,同时给出自由形式文本提示的初步结果。为了让 SAM 具备歧义感知能力,我们设计它对单个提示预测多个掩码,使 SAM 能自然处理歧义,例如衬衫与人的例子。

Data engine (§4). To achieve strong generalization to new data distributions, we found it necessary to train SAM on a large and diverse set of masks, beyond any segmentation dataset that already exists. While a typical approach for foundation models is to obtain data online [82], masks are not naturally abundant and thus we need an alternative strategy. Our solution is to build a “data engine”, i.e., we co-develop our model with model-in-the-loop dataset annotation (see Fig. 1c). Our data engine has three stages: assisted-manual, semi-automatic, and fully automatic. In the first stage, SAM assists annotators in annotating masks, similar to a classic interactive segmentation setup. In the second stage, SAM can automatically generate masks for a subset of objects by prompting it with likely object locations and annotators focus on annotating the remaining objects, helping increase mask diversity. In the final stage, we prompt SAM with a regular grid of foreground points, yielding on average ∼100 high-quality masks per image.

数据引擎(§4)。为了实现对新数据分布的强泛化,我们发现必须在超越任何现有分割数据集的大规模多样化掩码集上训练 SAM。虽然基础模型的典型做法是在线获取数据 [82],但掩码并非天然充足,因此我们需要另一种策略。我们的解决方案是构建一个「数据引擎」,即通过模型在环(model-in-the-loop)的数据集标注来共同开发模型(见 Fig. 1c)。我们的数据引擎有三个阶段:辅助手动、半自动和全自动。在第一阶段,SAM 辅助标注员标注掩码,类似于经典的交互式分割设置。在第二阶段,SAM 可以通过用可能的物体位置提示来自动生成部分物体的掩码,标注员专注于标注剩余物体,从而帮助提高掩码多样性。在最后阶段,我们用规则网格的前景点提示 SAM,平均每张图像生成约 100 个高质量掩码。

Dataset (§5). Our final dataset, SA-1B, includes more than 1B masks from 11M licensed and privacy-preserving images (see Fig. 2). SA-1B, collected fully automatically using the final stage of our data engine, has 400× more masks than any existing segmentation dataset [66, 44, 117, 60], and as we verify extensively, the masks are of high quality and diversity. Beyond its use in training SAM to be robust and general, we hope SA-1B becomes a valuable resource for research aiming to build new foundation models.

数据集(§5)。我们的最终数据集 SA-1B 包含来自 11M 张经授权且保护隐私图像的超过 1B 个掩码(见 Fig. 2)。SA-1B 使用数据引擎的最后阶段全自动收集,其掩码数量是任何现有分割数据集的 400 倍 [66, 44, 117, 60],并且正如我们广泛验证的那样,掩码具有高质量和多样性。除了用于训练 SAM 使其鲁棒和通用之外,我们希望 SA-1B 能成为构建新基础模型研究的宝贵资源。

Responsible AI (§6). We study and report on potential fairness concerns and biases when using SA-1B and SAM. Images in SA-1B span a geographically and economically diverse set of countries and we found that SAM performs similarly across different groups of people. Together, we hope this will make our work more equitable for real-world use cases. We provide model and dataset cards in the appendix.

负责任 AI(§6)。我们研究并报告使用 SA-1B 和 SAM 时潜在的公平性问题和偏见。SA-1B 中的图像覆盖了地理和经济上多样化的国家集合,我们发现 SAM 在不同人群组之间的表现相似。总之,我们希望这能使我们的工作对现实用例更加公平。我们在附录中提供了模型卡和数据集卡。

Experiments (§7). We extensively evaluate SAM. First, using a diverse new suite of 23 segmentation datasets, we find that SAM produces high-quality masks from a single foreground point, often only slightly below that of the manually annotated ground truth. Second, we find consistently strong quantitative and qualitative results on a variety of downstream tasks under a zero-shot transfer protocol using prompt engineering, including edge detection, object proposal generation, instance segmentation, and a preliminary exploration of text-to-mask prediction. These results suggest that SAM can be used out-of-the-box with prompt engineering to solve a variety of tasks involving object and image distributions beyond SAM’s training data. Nevertheless, room for improvement remains, as we discuss in §8.

实验(§7)。我们对 SAM 进行了广泛评估。首先,使用一套多样化的 23 个分割数据集,我们发现 SAM 仅凭单个前景点就能生成高质量掩码,通常仅略低于人工标注的真值。其次,在基于提示工程的零样本迁移协议下,我们在各种下游任务上得到了一致强劲的定量和定性结果,包括边缘检测、目标提议生成、实例分割,以及对文本到掩码预测的初步探索。这些结果表明,SAM 可以开箱即用地配合提示工程,解决涉及超出 SAM 训练数据的物体和图像分布的各种任务。尽管如此,仍有改进空间,我们将在 §8 中讨论。

Release. We are releasing the SA-1B dataset for research purposes and making SAM available under a permissive open license (Apache 2.0) at https://segment-anything.com. We also showcase SAM’s capabilities with an online demo.

发布。我们发布 SA-1B 数据集用于研究目的,并在 https://segment-anything.com 以宽松的开源许可证(Apache 2.0)提供 SAM。我们还通过在线 demo 展示 SAM 的能力。

Segment Anything任务

We take inspiration from NLP, where the next token prediction task is used for foundation model pre-training and to solve diverse downstream tasks via prompt engineering [10]. To build a foundation model for segmentation, we aim to define a task with analogous capabilities.

我们从 NLP 中获得启发:NLP 中「下一个 token 预测」任务被用于基础模型预训练,并通过提示工程解决多样化的下游任务 [10]。为了构建分割的基础模型,我们旨在定义一个具有类似能力的任务。

Task. We start by translating the idea of a prompt from NLP to segmentation, where a prompt can be a set of foreground / background points, a rough box or mask, free-form text, or, in general, any information indicating what to segment in an image. The promptable segmentation task, then, is to return a valid segmentation mask given any prompt. The requirement of a “valid” mask simply means that even when a prompt is ambiguous and could refer to multiple objects (e.g., recall the shirt vs. person example, and see Fig. 3), the output should be a reasonable mask for at least one of those objects. This requirement is similar to expecting a language model to output a coherent response to an ambiguous prompt. We choose this task because it leads to a natural pre-training algorithm and a general method for zero-shot transfer to downstream segmentation tasks via prompting.

任务。我们首先把 NLP 中提示的概念迁移到分割上:提示可以是一组前景 / 背景点、一个粗略的框或掩码、自由形式的文本,或者一般来说,任何指示图像中要分割什么的信息。那么,可提示分割任务就是给定任意提示返回一个有效的分割掩码。「有效」掩码的要求仅仅意味着,即使提示存在歧义、可能指向多个物体(例如,回想衬衫与人的例子,并参见 Fig. 3),输出也应当是其中至少一个物体的合理掩码。这个要求类似于期望语言模型对模糊提示输出连贯的回复。我们选择这个任务,是因为它能引出自然的预训练算法,以及通过提示实现零样本迁移到下游分割任务的通用方法。

Pre-training. The promptable segmentation task suggests a natural pre-training algorithm that simulates a sequence of prompts (e.g., points, boxes, masks) for each training sample and compares the model’s mask predictions against the ground truth. We adapt this method from interactive segmentation [109, 70], although unlike interactive segmentation whose aim is to eventually predict a valid mask after enough user input, our aim is to always predict a valid mask for any prompt even when the prompt is ambiguous. This ensures that a pre-trained model is effective in use cases that involve ambiguity, including automatic annotation as required by our data engine §4. We note that performing well at this task is challenging and requires specialized modeling and training loss choices, which we discuss in §3.

预训练。可提示分割任务给出了一种自然的预训练算法:对每个训练样本模拟一个提示序列(如点、框、掩码),并将模型的掩码预测与真值比较。我们借鉴交互式分割 [109, 70] 调整了这一方法;但与交互式分割的目标不同——交互式分割旨在用户输入足够多之后最终预测出有效掩码——我们的目标是对于任意提示始终预测出有效掩码,即使提示存在歧义。这保证了预训练模型在涉及歧义的用例中有效,包括数据引擎 §4 所需的自动标注。我们注意到,在这项任务上表现良好具有挑战性,需要专门的建模和训练损失选择,我们将在 §3 中讨论。

Zero-shot transfer. Intuitively, our pre-training task endows the model with the ability to respond appropriately to any prompt at inference time, and thus downstream tasks can be solved by engineering appropriate prompts. For example, if one has a bounding box detector for cats, cat instance segmentation can be solved by providing the detector’s box output as a prompt to our model. In general, a wide array of practical segmentation tasks can be cast as prompting. In addition to automatic dataset labeling, we explore five diverse example tasks in our experiments in §7.

零样本迁移。直观地说,我们的预训练任务赋予模型在推理时对任意提示做出恰当响应的能力,因此下游任务可以通过设计合适的提示来解决。例如,如果有一个猫的边界框检测器,那么将检测器的框输出作为提示提供给我们的模型,就可以解决猫的实例分割问题。一般来说,大量实际分割任务都可以转化为提示形式。除了自动数据集标注之外,我们在 §7 的实验中探索了五个不同的示例任务。

Related tasks. Segmentation is a broad field: there’s interactive segmentation [57, 109], edge detection [3], super pixelization [85], object proposal generation [2], foreground segmentation [94], semantic segmentation [90], instance segmentation [66], panoptic segmentation [59], etc. The goal of our promptable segmentation task is to produce a broadly capable model that can adapt to many (though not all) existing and new segmentation tasks via prompt engineering. This capability is a form of task generalization [26]. Note that this is different than previous work on multi-task segmentation systems. In a multi-task system, a single model performs a fixed set of tasks, e.g., joint semantic, instance, and panoptic segmentation [114, 19, 54], but the training and test tasks are the same. An important distinction in our work is that a model trained for promptable segmentation can perform a new, different task at inference time by acting as a component in a larger system, e.g., to perform instance segmentation, a promptable segmentation model is combined with an existing object detector.

相关任务。分割是一个广阔的领域:有交互式分割 [57, 109]、边缘检测 [3]、超像素化 [85]、目标提议生成 [2]、前景分割 [94]、语义分割 [90]、实例分割 [66]、全景分割 [59] 等。我们的可提示分割任务的目标是产生一个能力广泛的模型,能够通过提示工程适应许多(尽管不是全部)现有和新的分割任务。这种能力是任务泛化 [26] 的一种形式。注意这与之前的多任务分割系统不同。在多任务系统中,单个模型执行一组固定任务,例如联合语义、实例和全景分割 [114, 19, 54],但训练和测试任务是相同的。我们工作的一个重要区别在于:为可提示分割训练的模型可以在推理时作为更大系统的组件执行新的、不同的任务——例如,要执行实例分割,将可提示分割模型与现有的物体检测器组合即可。

Discussion. Prompting and composition are powerful tools that enable a single model to be used in extensible ways, potentially to accomplish tasks unknown at the time of model design. This approach is analogous to how other foundation models are used, e.g., how CLIP [82] is the text-image alignment component of the DALL·E [83] image generation system. We anticipate that composable system design, powered by techniques such as prompt engineering, will enable a wider variety of applications than systems trained specifically for a fixed set of tasks. It’s also interesting to compare promptable and interactive segmentation through the lens of composition: while interactive segmentation models are designed with human users in mind, a model trained for promptable segmentation can also be composed into a larger algorithmic system as we will demonstrate.

讨论。提示和组合是强大的工具,能让单个模型以可扩展的方式被使用,甚至可能完成模型设计时未知的任务。这种方法与其他基础模型的使用方式类似,例如 CLIP [82] 是 DALL·E [83] 图像生成系统中的文本-图像对齐组件。我们预期,由提示工程等技术驱动的可组合系统设计,将带来比针对固定任务集专门训练的系统更广泛的应用。从组合的角度比较可提示分割和交互式分割也很有意思:交互式分割模型是为人机交互的用户设计的,而为可提示分割训练的模型还可以组合进更大的算法系统,正如我们将展示的那样。

Segment Anything模型

We next describe the Segment Anything Model (SAM) for promptable segmentation. SAM has three components, illustrated in Fig. 4: an image encoder, a flexible prompt encoder, and a fast mask decoder. We build on Transformer vision models [14, 33, 20, 62] with specific tradeoffs for (amortized) real-time performance. We describe these components at a high-level here, with details in §A.

接下来我们描述用于可提示分割的 Segment Anything Model(SAM)。SAM 有三个组件,如 Fig. 4 所示:图像编码器、灵活的提示编码器和快速的掩码解码器。我们基于 Transformer 视觉模型 [14, 33, 20, 62] 构建,并针对(摊销的)实时性能做了特定权衡。我们在这里高层次地描述这些组件,细节见 §A。

Image encoder. Motivated by scalability and powerful pre-training methods, we use an MAE [47] pre-trained Vision Transformer (ViT) [33] minimally adapted to process high resolution inputs [62]. The image encoder runs once per image and can be applied prior to prompting the model.

图像编码器。受可扩展性和强大预训练方法的启发,我们使用 MAE [47] 预训练的 Vision Transformer(ViT)[33],并做了最小化调整以处理高分辨率输入 [62]。图像编码器每张图像只运行一次,可以在提示模型之前应用。

Prompt encoder. We consider two sets of prompts: sparse (points, boxes, text) and dense (masks). We represent points and boxes by positional encodings [95] summed with learned embeddings for each prompt type and free-form text with an off-the-shelf text encoder from CLIP [82]. Dense prompts (i.e., masks) are embedded using convolutions and summed element-wise with the image embedding.

提示编码器。我们考虑两组提示:稀疏(点、框、文本)和密集(掩码)。我们用位置编码 [95] 与每种提示类型的学习嵌入相加来表示点和框,用 CLIP [82] 的现成文本编码器表示自由形式的文本。密集提示(即掩码)使用卷积嵌入,并与图像嵌入逐元素相加。

Mask decoder. The mask decoder efficiently maps the image embedding, prompt embeddings, and an output token to a mask. This design, inspired by [14, 20], employs a modification of a Transformer decoder block [103] followed by a dynamic mask prediction head. Our modified decoder block uses prompt self-attention and cross-attention in two directions (prompt-to-image embedding and vice-versa) to update all embeddings. After running two blocks, we upsample the image embedding and an MLP maps the output token to a dynamic linear classifier, which then computes the mask foreground probability at each image location.

掩码解码器。掩码解码器高效地将图像嵌入、提示嵌入和输出 token 映射为掩码。该设计受 [14, 20] 启发,采用 Transformer 解码器块 [103] 的修改版,后接动态掩码预测头。我们修改后的解码器块使用提示自注意力,以及两个方向的交叉注意力(提示到图像嵌入、图像嵌入到提示)来更新所有嵌入。运行两个块之后,我们对图像嵌入进行上采样,MLP 将输出 token 映射为动态线性分类器,随后在每个图像位置计算掩码的前景概率。

Resolving ambiguity. With one output, the model will average multiple valid masks if given an ambiguous prompt. To address this, we modify the model to predict multiple output masks for a single prompt (see Fig. 3). We found 3 mask outputs is sufficient to address most common cases (nested masks are often at most three deep: whole, part, and subpart). During training, we backprop only the minimum loss [15, 45, 64] over masks. To rank masks, the model predicts a confidence score (i.e., estimated IoU) for each mask.

歧义消解。只有一个输出时,如果给定模糊提示,模型会对多个有效掩码取平均。为了解决这个问题,我们修改模型,对单个提示预测多个输出掩码(见 Fig. 3)。我们发现输出 3 个掩码足以应对大多数常见情况(嵌套掩码通常最多三层:整体、部分和子部分)。训练时,我们只对掩码中的最小损失 [15, 45, 64] 进行反向传播。为了给掩码排序,模型为每个掩码预测一个置信度分数(即估计的 IoU)。

Efficiency. The overall model design is largely motivated by efficiency. Given a precomputed image embedding, the prompt encoder and mask decoder run in a web browser, on CPU, in ∼50ms. This runtime performance enables seamless, real-time interactive prompting of our model.

效率。整体模型设计在很大程度上由效率驱动。给定预先计算的图像嵌入,提示编码器和掩码解码器可以在网页浏览器中、在 CPU 上、约 50ms 内运行。这种运行时性能实现了对模型的无缝、实时交互式提示。

Losses and training. We supervise mask prediction with the linear combination of focal loss [65] and dice loss [73] used in [14]. We train for the promptable segmentation task using a mixture of geometric prompts (for text prompts see §7.5). Following [92, 37], we simulate an interactive setup by randomly sampling prompts in 11 rounds per mask, allowing SAM to integrate seamlessly into our data engine.

损失与训练。我们使用 [14] 中的 focal loss [65] 和 dice loss [73] 的线性组合来监督掩码预测。我们使用几何提示的混合来训练可提示分割任务(文本提示见 §7.5)。遵循 [92, 37],我们通过为每个掩码随机采样 11 轮提示来模拟交互式设置,使 SAM 能无缝集成到我们的数据引擎中。

Segment Anything数据引擎

As segmentation masks are not abundant on the internet, we built a data engine to enable the collection of our 1.1B mask dataset, SA-1B. The data engine has three stages: (1) a model-assisted manual annotation stage, (2) a semi-automatic stage with a mix of automatically predicted masks and model-assisted annotation, and (3) a fully automatic stage in which our model generates masks without annotator input. We go into details of each next.

由于互联网上的分割掩码并不充足,我们构建了一个数据引擎来支撑我们 1.1B 掩码数据集 SA-1B 的收集。数据引擎有三个阶段:(1)模型辅助的手动标注阶段,(2)自动预测掩码与模型辅助标注混合的半自动阶段,以及(3)模型在无标注员输入的情况下生成掩码的全自动阶段。下面逐一介绍细节。

Assisted-manual stage. In the first stage, resembling classic interactive segmentation, a team of professional annotators labeled masks by clicking foreground / background object points using a browser-based interactive segmentation tool powered by SAM. Masks could be refined using pixel-precise “brush” and “eraser” tools. Our model-assisted annotation runs in real-time directly inside a browser (using precomputed image embeddings) enabling a truly interactive experience. We did not impose semantic constraints for labeling objects, and annotators freely labeled both “stuff” and “things” [1]. We suggested annotators label objects they could name or describe, but did not collect these names or descriptions. Annotators were asked to label objects in order of prominence and were encouraged to proceed to the next image once a mask took over 30 seconds to annotate.

辅助手动阶段。在第一阶段,与经典交互式分割类似,一组专业标注员使用基于浏览器的、由 SAM 驱动的交互式分割工具,通过点击前景 / 背景物体点来标注掩码。掩码可以使用像素级精确的「画笔」和「橡皮擦」工具细化。我们的模型辅助标注直接在浏览器内实时运行(使用预先计算的图像嵌入),实现了真正的交互式体验。我们没有对物体标注施加语义约束,标注员自由地标注「stuff」和「things」[1]。我们建议标注员标注他们能命名或描述的物体,但没有收集这些名称或描述。要求标注员按显著程度顺序标注物体,并鼓励他们在某个掩码标注时间超过 30 秒后转到下一张图像。

At the start of this stage, SAM was trained using common public segmentation datasets. After sufficient data annotation, SAM was retrained using only newly annotated masks. As more masks were collected, the image encoder was scaled from ViT-B to ViT-H and other architectural details evolved; in total we retrained our model 6 times. Average annotation time per mask decreased from 34 to 14 seconds as the model improved. We note that 14 seconds is 6.5× faster than mask annotation for COCO [66] and only 2× slower than bounding-box labeling with extreme points [76, 71]. As SAM improved, the average number of masks per image increased from 20 to 44 masks. Overall, we collected 4.3M masks from 120k images in this stage.

在这一阶段开始时,SAM 使用常见的公开分割数据集训练。在数据标注足够多之后,SAM 仅使用新标注的掩码重新训练。随着收集到的掩码增多,图像编码器从 ViT-B 扩展到 ViT-H,其他架构细节也在演进;我们总共重新训练了模型 6 次。随着模型的改进,每个掩码的平均标注时间从 34 秒下降到 14 秒。我们注意到,14 秒比 COCO [66] 的掩码标注快 6.5 倍,仅比使用极点的边界框标注 [76, 71] 慢 2 倍。随着 SAM 的改进,每张图像的平均掩码数从 20 个增加到 44 个。总体上,我们在这一阶段从 120k 张图像中收集了 4.3M 个掩码。

Semi-automatic stage. In this stage, we aimed to increase the diversity of masks in order to improve our model’s ability to segment anything. To focus annotators on less prominent objects, we first automatically detected confident masks. Then we presented annotators with images prefilled with these masks and asked them to annotate any additional unannotated objects. To detect confident masks, we trained a bounding box detector [84] on all first stage masks using a generic “object” category. During this stage we collected an additional 5.9M masks in 180k images (for a total of 10.2M masks). As in the first stage, we periodically retrained our model on newly collected data (5 times). Average annotation time per mask went back up to 34 seconds (excluding the automatic masks) as these objects were more challenging to label. The average number of masks per image went from 44 to 72 masks (including the automatic masks).

半自动阶段。在这一阶段,我们旨在提高掩码的多样性,以改进模型「分割一切」的能力。为了让标注员专注于不太显著的物体,我们首先自动检测高置信度的掩码。然后向标注员展示预先填充了这些掩码的图像,要求他们标注任何其他未标注的物体。为了检测高置信度掩码,我们在第一阶段的所有掩码上训练了一个使用通用「物体」类别的边界框检测器 [84]。在这一阶段,我们额外从 180k 张图像中收集了 5.9M 个掩码(总计 10.2M 个)。与第一阶段一样,我们定期在新收集的数据上重新训练模型(5 次)。由于这些物体标注难度更大,每个掩码的平均标注时间回升到 34 秒(不包括自动掩码)。每张图像的平均掩码数从 44 个增加到 72 个(包括自动掩码)。

Fully automatic stage. In the final stage, annotation was fully automatic. This was feasible due to two major enhancements to our model. First, at the start of this stage, we had collected enough masks to greatly improve the model, including the diverse masks from the previous stage. Second, by this stage we had developed the ambiguity-aware model, which allowed us to predict valid masks even in ambiguous cases. Specifically, we prompted the model with a 32×32 regular grid of points and for each point predicted a set of masks that may correspond to valid objects. With the ambiguity-aware model, if a point lies on a part or subpart, our model will return the subpart, part, and whole object. The IoU prediction module of our model is used to select confident masks; moreover, we identified and selected only stable masks (we consider a mask stable if thresholding the probability map at 0.5 − δ and 0.5 + δ results in similar masks). Finally, after selecting the confident and stable masks, we applied non-maximal suppression (NMS) to filter duplicates. To further improve the quality of smaller masks, we also processed multiple overlapping zoomed-in image crops. For further details of this stage, see §B. We applied fully automatic mask generation to all 11M images in our dataset, producing a total of 1.1B high-quality masks. We describe and analyze the resulting dataset, SA-1B, next.

全自动阶段。在最后阶段,标注完全自动。这得益于对模型的两项重大增强。第一,在这一阶段开始时,我们已经收集了足够的掩码来大幅改进模型,包括上一阶段多样化的掩码。第二,到这一阶段我们已经开发出歧义感知模型,即使在存在歧义的情况下也能预测有效掩码。具体来说,我们用 32×32 规则网格的点提示模型,为每个点预测一组可能对应有效物体的掩码。有了歧义感知模型,如果某个点位于部分或子部分上,我们的模型将返回子部分、部分和整体物体。模型中的 IoU 预测模块用于选择高置信度掩码;此外,我们仅识别和选择稳定掩码(如果以 0.5 − δ 和 0.5 + δ 阈值化概率图得到相似的掩码,我们认为该掩码是稳定的)。最后,在选择高置信度和稳定的掩码后,我们应用非极大值抑制(NMS)过滤重复项。为了进一步提高较小掩码的质量,我们还处理了多个重叠的放大图像裁剪。这一阶段的更多细节见 §B。我们将全自动掩码生成应用于数据集中的全部 11M 张图像,共产生 1.1B 个高质量掩码。接下来我们描述并分析生成的数据集 SA-1B。

Segment Anything数据集

Our dataset, SA-1B, consists of 11M diverse, high-resolution, licensed, and privacy protecting images and 1.1B high-quality segmentation masks collected with our data engine. We compare SA-1B with existing datasets and analyze mask quality and properties. We are releasing SA-1B to aid future development of foundation models for computer vision. We note that SA-1B will be released under a favorable license agreement for certain research uses and with protections for researchers.

我们的数据集 SA-1B 由 11M 张多样化、高分辨率、经授权且保护隐私的图像,以及用我们的数据引擎收集的 1.1B 个高质量分割掩码组成。我们将 SA-1B 与现有数据集进行比较,并分析掩码质量和属性。我们发布 SA-1B 以助力计算机视觉基础模型的未来发展。我们注意到,SA-1B 将以对特定研究用途有利的许可协议发布,并为研究人员提供保护。

Images. We licensed a new set of 11M images from a provider that works directly with photographers. These images are high resolution (3300×4950 pixels on average), and the resulting data size can present accessibility and storage challenges. Therefore, we are releasing downsampled images with their shortest side set to 1500 pixels. Even after downsampling, our images are significantly higher resolution than many existing vision datasets (e.g., COCO [66] images are ∼480×640 pixels). Note that most models today operate on much lower resolution inputs. Faces and vehicle license plates have been blurred in the released images.

图像。我们从一家直接与摄影师合作的供应商处授权了一组新的 11M 张图像。这些图像分辨率很高(平均 3300×4950 像素),由此产生的数据规模可能带来可访问性和存储上的挑战。因此,我们发布短边设为 1500 像素的下采样图像。即使经过下采样,我们的图像分辨率仍显著高于许多现有视觉数据集(例如 COCO [66] 的图像约为 480×640 像素)。注意,如今大多数模型在低得多的分辨率输入上运行。发布图像中的人脸和车辆车牌已做模糊处理。

Masks. Our data engine produced 1.1B masks, 99.1% of which were generated fully automatically. Therefore, the quality of the automatic masks is centrally important. We compare them directly to professional annotations and look at how various mask properties compare to prominent segmentation datasets. Our main conclusion, as borne out in the analysis below and the experiments in §7, is that our automatic masks are high quality and effective for training models. Motivated by these findings, SA-1B only includes automatically generated masks.

掩码。我们的数据引擎生成了 1.1B 个掩码,其中 99.1% 全自动生成。因此,自动掩码的质量至关重要。我们将它们与专业标注直接比较,并考察各种掩码属性与主流分割数据集相比如何。正如以下分析和 §7 中的实验所证实的,我们的主要结论是:自动掩码质量高且对训练模型有效。基于这些发现,SA-1B 只包含自动生成的掩码。

Mask quality. To estimate mask quality, we randomly sampled 500 images (∼50k masks) and asked our professional annotators to improve the quality of all masks in these images. Annotators did so using our model and pixel-precise “brush” and “eraser” editing tools. This procedure resulted in pairs of automatically predicted and professionally corrected masks. We computed IoU between each pair and found that 94% of pairs have greater than 90% IoU (and 97% of pairs have greater than 75% IoU). For comparison, prior work estimates inter-annotator consistency at 85-91% IoU [44, 60]. Our experiments in §7 confirm by human ratings that mask quality is high relative to a variety of datasets and that training our model on automatic masks is nearly as good as using all masks produced by the data engine.

掩码质量。为了估计掩码质量,我们随机采样了 500 张图像(约 50k 个掩码),要求专业标注员改进这些图像中所有掩码的质量。标注员使用我们的模型和像素级精确的「画笔」「橡皮擦」编辑工具完成这一工作。这个过程产生了自动预测掩码与专业修正掩码的配对。我们计算了每对之间的 IoU,发现 94% 的配对 IoU 大于 90%(97% 的配对大于 75%)。作为对比,先前工作估计标注员间一致性为 85-91% IoU [44, 60]。§7 中的实验通过人类评分确认,相对于各种数据集,掩码质量很高,并且在自动掩码上训练模型几乎与使用数据引擎产生的全部掩码一样好。

Mask properties. In Fig. 5 we plot the spatial distribution of object centers in SA-1B compared to the largest existing segmentation datasets. Common photographer biases are present in all datasets. We observe that SA-1B has greater coverage of image corners compared to LVIS v1 [44] and ADE20K [117], the two most similarly distributed datasets, while COCO [66] and Open Images V5 [60] have a more prominent center bias. In Fig. 6 (legend) we compare these datasets by size. SA-1B has 11× more images and 400× more masks than the second largest, Open Images. On average, it has 36× more masks per image than Open Images. The closest dataset in this respect, ADE20K, still has 3.5× fewer masks per image. Fig. 6 (left) plots the masks-per-image distribution. Next, we look at image-relative mask size (square root of the mask area divided by image area) in Fig. 6 (middle). As expected, since our dataset has more masks per image, it also tends to include a greater percentage of small and medium relative-size masks. Finally, to analyze shape complexity, we look at mask concavity (1 minus mask area divided by area of mask’s convex hull) in Fig. 6 (right). Since shape complexity is correlated with mask size, we control for the datasets’ mask size distributions by first performing stratified sampling from binned mask sizes. We observe that the concavity distribution of our masks is broadly similar to that of other datasets.

掩码属性。在 Fig. 5 中,我们绘制了 SA-1B 中物体中心的空间分布,并与最大的现有分割数据集比较。所有数据集都存在常见的摄影师偏差。我们观察到,与分布最相似的两个数据集 LVIS v1 [44] 和 ADE20K [117] 相比,SA-1B 对图像角落的覆盖更大,而 COCO [66] 和 Open Images V5 [60] 的中心偏差更明显。在 Fig. 6(图例)中,我们按规模比较这些数据集。SA-1B 的图像数量是第二大数据集 Open Images 的 11 倍,掩码数量是其 400 倍。平均而言,其每张图像的掩码数是 Open Images 的 36 倍。在这方面最接近的数据集 ADE20K,每张图像的掩码数仍少 3.5 倍。Fig. 6(左)绘制了每张图像掩码数的分布。接下来,我们在 Fig. 6(中)考察图像相对掩码尺寸(掩码面积的平方根除以图像面积)。正如预期,由于我们的数据集每张图像的掩码更多,它也倾向于包含更大比例的小尺寸和中尺寸相对掩码。最后,为了分析形状复杂度,我们在 Fig. 6(右)中考察掩码凹度(1 减去掩码面积除以掩码凸包面积)。由于形状复杂度与掩码尺寸相关,我们首先按掩码尺寸分箱进行分层采样,以控制各数据集的掩码尺寸分布。我们观察到,我们的掩码凹度分布与其他数据集大致相似。

Segment Anything RAI分析

We next perform a Responsible AI (RAI) analysis of our work by investigating potential fairness concerns and biases when using SA-1B and SAM. We focus on the geographic and income distribution of SA-1B and fairness of SAM across protected attributes of people. We also provide dataset, data annotation, and model cards in §F.

接下来,我们通过调查使用 SA-1B 和 SAM 时潜在的公平性问题和偏见,对这项工作进行负责任 AI(RAI)分析。我们重点关注 SA-1B 的地理和收入分布,以及 SAM 在人的受保护属性上的公平性。我们还在 §F 中提供了数据集卡、数据标注卡和模型卡。

Geographic and income representation. We infer the country images were photographed in using standard methods (see §C). In Fig. 7 we visualize the per-country image counts in SA-1B (left) and the 50 countries with the most images (right). We note that the top-three countries are from different parts of the world. Next, in Table 1 we compare the geographic and income representation of SA-1B, COCO [66], and Open Images [60]. SA-1B has a substantially higher percentage of images in Europe and Asia & Oceania as well as in middle income countries. All datasets underrepresent Africa as well as low income countries. We note that in SA-1B, all regions, including Africa, have at least 28 million masks, 10× more than the total number of masks of any previous dataset. Finally, we observe that the average number of masks per image (not shown) is fairly consistent across region and income (94-108 per image).

地理和收入代表性。我们使用标准方法推断图像拍摄所在的国家(见 §C)。在 Fig. 7 中,我们可视化 SA-1B 的各国图像数量(左)以及图像最多的 50 个国家(右)。我们注意到,图像最多的三个国家来自世界不同地区。接下来,在 Table 1 中我们比较 SA-1B、COCO [66] 和 Open Images [60] 的地理和收入代表性。SA-1B 在欧洲以及亚洲与大洋洲的图像占比显著更高,中等收入国家的占比也更高。所有数据集都欠代表非洲和低收入国家。我们注意到,在 SA-1B 中,包括非洲在内的所有地区都至少有 28M 个掩码,是任何先前数据集掩码总数的 10 倍以上。最后,我们观察到每张图像的平均掩码数(未展示)在不同地区和收入水平之间相当一致(每张 94-108 个)。

Fairness in segmenting people. We investigate potential fairness concerns across perceived gender presentation, perceived age group, and perceived skin tone by measuring the performance discrepancy of SAM between groups. We use the More Inclusive Annotations for People (MIAP) [87] dataset for gender presentation and age and a proprietary dataset for skin tone (see §C). Our evaluation uses simulated interactive segmentation with random sampling of 1 and 3 points (see §D). Table 2 (top left) shows results for perceived gender presentation. We note that females have been shown to be underrepresented in detection and segmentation datasets [115], but observe that SAM performs similarly across groups. We repeat the analysis for perceived age in Table 2 (bottom left), noting that those who are perceived to be younger and older have been shown to be underrepresented in large-scale datasets [110]. SAM performs best on those who are perceived older (although the confidence interval is large). Finally, we repeat the analysis for perceived skin tone in Table 2 (right), noting that those with lighter apparent skin tones have been shown to be overrepresented and those with darker skin tones underrepresented in large-scale datasets [110]. As MIAP does not contain perceived skin tone annotations, we use a proprietary dataset that contains annotations for the perceived Fitzpatrick skin type [36], which ranges from 1 (lightest skin tone) to 6 (darkest skin tone). While the means vary somewhat, we do not find a significant difference across groups. We believe our findings stem from the nature of the task, and acknowledge biases may arise when SAM is used as a component in larger systems. Finally, in §C we extend the analysis to segmenting clothing where we find an indication of bias across perceived gender presentation.

分割人群的公平性。我们通过测量 SAM 在各组之间的性能差异,调查感知性别表达、感知年龄组和感知肤色方面的潜在公平性问题。我们使用 More Inclusive Annotations for People(MIAP)[87] 数据集评估性别表达和年龄,使用专有数据集评估肤色(见 §C)。我们的评估使用随机采样 1 个和 3 个点的模拟交互式分割(见 §D)。Table 2(左上)显示了感知性别表达的结果。我们注意到,有研究表明女性在检测和分割数据集中欠代表 [115],但我们观察到 SAM 在各组之间的表现相似。我们在 Table 2(左下)中对感知年龄重复该分析,注意到有研究表明被认为年轻和年长的人在大型数据集中欠代表 [110]。SAM 在被认为年长的人上表现最好(尽管置信区间很大)。最后,我们在 Table 2(右)中对感知肤色重复该分析,注意到有研究表明肤色较浅的人在大型数据集中过代表、肤色较深的人欠代表 [110]。由于 MIAP 不包含感知肤色标注,我们使用一个包含感知 Fitzpatrick 皮肤类型 [36] 标注的专有数据集,范围从 1(最浅肤色)到 6(最深肤色)。虽然均值有所不同,但我们没有发现各组之间存在显著差异。我们相信我们的发现源于任务本身的性质,并承认当 SAM 被用作更大系统的组件时可能会产生偏见。最后,在 §C 中我们将分析扩展到分割衣物,发现感知性别表达之间存在偏见的迹象。

零样本迁移实验

In this section, we present zero-shot transfer experiments with SAM, the Segment Anything Model. We consider five tasks, four of which differ significantly from the promptable segmentation task used to train SAM. These experiments evaluate SAM on datasets and tasks that were not seen during training (our usage of “zero-shot transfer” follows its usage in CLIP [82]). The datasets may include novel image distributions, such as underwater or ego-centric images (e.g. Fig. 8) that, to our knowledge, do not appear in SA-1B. Our experiments begin by testing the core goal of promptable segmentation: producing a valid mask from any prompt. We emphasize the challenging scenario of a single foreground point prompt, since it is more likely to be ambiguous than other more specific prompts. Next, we present a sequence of experiments that traverse low, mid, and high-level image understanding and roughly parallel the historical development of the field. Specifically, we prompt SAM to (1) perform edge detection, (2) segment everything, i.e. object proposal generation, (3) segment detected objects, i.e. instance segmentation, and (4), as a proof-of-concept, to segment objects from free-form text. These four tasks differ significantly from the promptable segmentation task that SAM was trained on and are implemented via prompt engineering. Our experiments conclude with an ablation study.

在本节中,我们介绍使用 SAM(Segment Anything Model)的零样本迁移实验。我们考虑五个任务,其中四个与用于训练 SAM 的可提示分割任务有显著差异。这些实验在训练期间未见过的数据集和任务上评估 SAM(我们对「零样本迁移」的用法沿用 CLIP [82] 中的用法)。这些数据集可能包含新的图像分布,例如水下或自我中心图像(如 Fig. 8),据我们所知这些图像不出现在 SA-1B 中。我们的实验首先测试可提示分割的核心目标:从任意提示生成有效掩码。我们强调单个前景点提示这一具有挑战性的场景,因为与其他更具体的提示相比,它更可能产生歧义。接下来,我们展示一系列横跨低、中、高层级图像理解的实验,大致对应这一领域的历史发展。具体来说,我们提示 SAM(1)执行边缘检测,(2)分割一切,即目标提议生成,(3)分割检测到的物体,即实例分割,以及(4)作为概念验证,从自由形式文本中分割物体。这四个任务与 SAM 训练所用的可提示分割任务有显著差异,并且通过提示工程实现。我们的实验以消融研究结束。

Implementation. Unless otherwise specified: (1) SAM uses an MAE [47] pre-trained ViT-H [33] image encoder and (2) SAM was trained on SA-1B, noting that this dataset includes only automatically generated masks from the final stage of our data engine. For all other model and training details, such as hyperparameters, refer to §A.

实现。除非另有说明:(1)SAM 使用 MAE [47] 预训练的 ViT-H [33] 图像编码器,(2)SAM 在 SA-1B 上训练,注意该数据集只包含数据引擎最后阶段自动生成的掩码。所有其他模型和训练细节(如超参数)参见 §A。

单点有效掩码的零样本评估

Task. We evaluate segmenting an object from a single foreground point. This task is ill-posed as one point can refer to multiple objects. Ground truth masks in most datasets do not enumerate all possible masks, which can make automatic metrics unreliable. Therefore, we supplement the standard mIoU metric (i.e., the mean of all IoUs between predicted and ground truth masks) with a human study in which annotators rate mask quality from 1 (nonsense) to 10 (pixel-perfect). See §D.1, §E, and §G for additional details. By default, we sample points from the “center” of ground truth masks (at a maximal value of the mask’s interior distance transform), following the standard evaluation protocol in interactive segmentation [92]. Since SAM is capable of predicting multiple masks, we evaluate only the model’s most confident mask by default. The baselines are all single-mask methods. We compare mainly to RITM [92], a strong interactive segmenter that performs best on our benchmark compared to other strong baselines [67, 18].

任务。我们评估从单个前景点分割物体。这个任务是病态的,因为一个点可能指向多个物体。大多数数据集中的真值掩码并没有枚举所有可能的掩码,这会使自动指标不可靠。因此,我们在标准 mIoU 指标(即预测掩码与真值掩码之间所有 IoU 的均值)之外补充一项人类研究:标注员将掩码质量从 1(无意义)到 10(像素级完美)评分。详见 §D.1、§E 和 §G。默认情况下,我们从真值掩码的「中心」采样点(位于掩码内部距离变换的最大值处),遵循交互式分割 [92] 的标准评估协议。由于 SAM 能够预测多个掩码,默认情况下我们只评估模型最置信的掩码。基线都是单掩码方法。我们主要与 RITM [92] 比较,它是一个强交互式分割器,在我们的基准上相比其他强基线 [67, 18] 表现最好。

Datasets. We use a newly compiled suite of 23 datasets with diverse image distributions. Fig. 8 lists the datasets and shows a sample from each one (see appendix Table 7 for more details). We use all 23 datasets for mIoU evaluation. For the human study, we use the subset listed in Fig. 9b (due to the resource requirements of such studies). This subset includes both datasets for which SAM outperforms and underperforms RITM according to automatic metrics.

数据集。我们使用新编译的一套 23 个数据集,具有多样化的图像分布。Fig. 8 列出了这些数据集并展示了每个数据集的一个样本(更多细节见附录 Table 7)。我们使用全部 23 个数据集进行 mIoU 评估。对于人类研究,我们使用 Fig. 9b 中列出的子集(由于此类研究的资源需求)。该子集既包括 SAM 在自动指标上优于 RITM 的数据集,也包括劣于 RITM 的数据集。

Results. First, we look at automatic evaluation on the full suite of 23 datasets using mIoU. We compare per-dataset results in Fig. 9a against RITM. SAM yields higher results on 16 of the 23 datasets, by as much as ∼47 IoU. We also present an “oracle” result, in which the most relevant of SAM’s 3 masks is selected by comparing them to the ground truth, rather than selecting the most confident mask. This reveals the impact of ambiguity on automatic evaluation. In particular, with the oracle to perform ambiguity resolution, SAM outperforms RITM on all datasets. Results of the human study are presented in Fig. 9b. Error bars are 95% confidence intervals for mean mask ratings (all differences are significant; see §E for details). We observe that the annotators consistently rate the quality of SAM’s masks substantially higher than the strongest baseline, RITM. An ablated, “ambiguity-unaware” version of SAM with a single output mask has consistently lower ratings, though still higher than RITM. SAM’s mean ratings fall between 7 and 9, which corresponds to the qualitative rating guideline: “A high score (7-9): The object is identifiable and errors are small and rare (e.g., missing a small, heavily obscured disconnected component, …).” These results indicate that SAM has learned to segment valid masks from a single point. Note that for datasets like DRAM and IBD, where SAM is worse on automatic metrics, it receives consistently higher ratings in the human study. Fig. 9c shows additional baselines, SimpleClick [67] and FocalClick [18], which obtain lower single point performance than RITM and SAM. As the number of points increases from 1 to 9, we observe that the gap between methods decreases. This is expected as the task becomes easier; also, SAM is not optimized for the very high IoU regime. Finally, in Fig. 9d we replace the default center point sampling with random point sampling. We observe that the gap between SAM and the baselines grows and SAM is able to achieve comparable results under either sampling method.

结果。首先,我们看全部 23 个数据集上使用 mIoU 的自动评估。我们在 Fig. 9a 中按数据集与 RITM 比较结果。SAM 在 23 个数据集中的 16 个上结果更高,最高超出约 47 IoU。我们还给出一个「oracle」结果:将 SAM 的 3 个掩码与真值比较,选择最相关的那个,而不是选择最置信的掩码。这揭示了歧义对自动评估的影响。特别是,有了 oracle 来执行歧义消解后,SAM 在所有数据集上都优于 RITM。人类研究的结果见 Fig. 9b。误差条是平均掩码评分的 95% 置信区间(所有差异均显著;详见 §E)。我们观察到,标注员对 SAM 掩码质量的评分始终显著高于最强基线 RITM。一个只有单个输出掩码的消融版「歧义不感知」SAM 评分始终较低,但仍高于 RITM。SAM 的平均评分落在 7 到 9 之间,对应定性评分准则:「高分(7-9):物体可辨认,错误小而罕见(例如,遗漏一个小的、严重遮挡的不连通分量……)」。这些结果表明 SAM 已经学会了从单个点分割出有效掩码。注意,对于 DRAM 和 IBD 这类 SAM 在自动指标上较差的数据集,它在人类研究中却获得了持续更高的评分。Fig. 9c 展示了额外的基线 SimpleClick [67] 和 FocalClick [18],它们的单点性能低于 RITM 和 SAM。随着点数从 1 增加到 9,我们观察到方法之间的差距缩小。这是预期的,因为任务变得更容易;另外,SAM 没有针对极高 IoU 区间优化。最后,在 Fig. 9d 中,我们将默认的中心点采样替换为随机点采样。我们观察到 SAM 与基线之间的差距变大,且 SAM 在两种采样方式下都能取得可比的结果。

零样本边缘检测

Approach. We evaluate SAM on the classic low-level task of edge detection using BSDS500 [72, 3]. We use a simplified version of our automatic mask generation pipeline. Specifically, we prompt SAM with a 16×16 regular grid of foreground points resulting in 768 predicted masks (3 per point). Redundant masks are removed by NMS. Then, edge maps are computed using Sobel filtering of unthresholded mask probability maps and standard lightweight postprocessing, including edge NMS (see §D.2 for details).

方法。我们使用 BSDS500 [72, 3] 在经典的低层级边缘检测任务上评估 SAM。我们使用自动掩码生成流水线的简化版本。具体来说,我们用 16×16 规则网格的前景点提示 SAM,产生 768 个预测掩码(每点 3 个)。冗余掩码通过 NMS 去除。然后,对未阈值化的掩码概率图进行 Sobel 滤波,并进行标准的轻量后处理(包括边缘 NMS),计算边缘图(详见 §D.2)。

Results. We visualize representative edge maps in Fig. 10 (see Fig. 15 for more). Qualitatively, we observe that even though SAM was not trained for edge detection, it produces reasonable edge maps. Compared to the ground truth, SAM predicts more edges, including sensible ones that are not annotated in BSDS500. This bias is reflected quantitatively in Table 3: recall at 50% precision (R50) is high, at the cost of precision. SAM naturally lags behind state-of-the-art methods that learn the biases of BSDS500, i.e., which edges to suppress. Nevertheless, SAM performs well compared to pioneering deep learning methods such as HED [108] (also trained on BSDS500) and significantly better than prior, though admittedly outdated, zero-shot transfer methods.

结果。我们在 Fig. 10 中可视化有代表性的边缘图(更多见 Fig. 15)。定性来看,我们观察到,尽管 SAM 没有为边缘检测训练过,它仍能生成合理的边缘图。与真值相比,SAM 预测了更多边缘,包括 BSDS500 中未标注的合理边缘。这种偏差在 Table 3 中得到定量反映:50% 精度下的召回率(R50)很高,代价是精度。SAM 自然落后于学习 BSDS500 偏差(即哪些边缘应该抑制)的最先进方法。尽管如此,与 HED [108](同样在 BSDS500 上训练)等开创性深度学习方法相比,SAM 表现良好,并且显著优于先前的(尽管公认过时的)零样本迁移方法。

零样本目标提议

Approach. Next, we evaluate SAM on the mid-level task of object proposal generation [2, 102]. This task has played an important role in object detection research, serving as an intermediate step in pioneering systems (e.g., [102, 41, 84]). To generate object proposals, we run a slightly modified version of our automatic mask generation pipeline and output the masks as proposals (see §D.3 for details). We compute the standard average recall (AR) metric on LVIS v1 [44]. We focus on LVIS because its large number of categories presents a challenging test. We compare to a strong baseline implemented as a ViTDet [62] detector (with cascade Mask R-CNN [48, 11] ViT-H). We note that this “baseline” corresponds to the “Detector Masquerading as Proposal generator” (DMP) method [16] that was shown to game AR, making it a truly demanding comparison.

方法。接下来,我们在中层任务——目标提议生成 [2, 102] 上评估 SAM。这个任务曾在目标检测研究中发挥重要作用,是开创性系统(如 [102, 41, 84])中的中间步骤。为了生成目标提议,我们运行自动掩码生成流水线的略微修改版本,将掩码作为提议输出(详见 §D.3)。我们在 LVIS v1 [44] 上计算标准平均召回率(AR)指标。我们重点关注 LVIS,因为其类别数量庞大,构成了具有挑战性的测试。我们与一个强基线比较:实现为 ViTDet [62] 检测器(级联 Mask R-CNN [48, 11] ViT-H)。我们注意到,这个「基线」对应「Detector Masquerading as Proposal generator」(DMP)方法 [16],该方法被证明能刷高 AR,使这成为一个真正严苛的比较。

Results. In Table 4 we see unsurprisingly that using the detections from ViTDet-H as object proposals (i.e., the DMP method [16] that games AR) performs the best overall. However, SAM does remarkably well on several metrics. Notably, it outperforms ViTDet-H on medium and large objects, as well as rare and common objects. In fact, SAM only underperforms ViTDet-H on small objects and frequent objects, where ViTDet-H can easily learn LVIS-specific annotation biases since it was trained on LVIS, unlike SAM. We also compare against an ablated ambiguity-unaware version of SAM (“single out.”), which performs significantly worse than SAM on all AR metrics.

结果。在 Table 4 中,不出所料,使用 ViTDet-H 的检测结果作为目标提议(即刷 AR 的 DMP 方法 [16])整体表现最好。然而,SAM 在若干指标上表现出色。值得注意的是,它在中等和大物体以及稀有和常见物体上超过了 ViTDet-H。事实上,SAM 只在小物体和频繁物体上不如 ViTDet-H,因为 ViTDet-H 在 LVIS 上训练,能够轻松学习 LVIS 特有的标注偏差,而 SAM 不能。我们还与消融的歧义不感知版 SAM(「single out.」)比较,后者在所有 AR 指标上都显著差于 SAM。

零样本实例分割

Approach. Moving to higher-level vision, we use SAM as the segmentation module of an instance segmenter. The implementation is simple: we run a object detector (the ViTDet used before) and prompt SAM with its output boxes. This illustrates composing SAM in a larger system.

方法。转向更高层级的视觉任务,我们将 SAM 用作实例分割器的分割模块。实现很简单:我们运行一个物体检测器(前面用过的 ViTDet),用其输出的框提示 SAM。这展示了 SAM 在更大系统中的组合使用。

Results. We compare the masks predicted by SAM and ViTDet on COCO and LVIS in Table 5. Looking at the mask AP metric we observe gaps on both datasets, where SAM is reasonably close, though certainly behind ViTDet. By visualizing outputs, we observed that SAM masks are often qualitatively better than those of ViTDet, with crisper boundaries (see §D.4 and Fig. 16). To investigate this observation, we conducted an additional human study asking annotators to rate the ViTDet masks and SAM masks on the 1 to 10 quality scale used before. In Fig. 11 we observe that SAM consistently outperforms ViTDet in the human study.

结果。我们在 Table 5 中比较 SAM 和 ViTDet 在 COCO 和 LVIS 上预测的掩码。从 mask AP 指标来看,我们观察到两个数据集上都有差距,SAM 相当接近,但确实落后于 ViTDet。通过可视化输出,我们观察到 SAM 的掩码通常在质量上优于 ViTDet,边界更清晰(见 §D.4 和 Fig. 16)。为了研究这一观察,我们进行了另一项人类研究,要求标注员用之前使用的 1 到 10 质量量表给 ViTDet 掩码和 SAM 掩码评分。在 Fig. 11 中,我们观察到 SAM 在人类研究中始终优于 ViTDet。

We hypothesize that on COCO, where the mask AP gap is larger and the ground truth quality is relatively low (as borne out by the human study), ViTDet learns the specific biases of COCO masks. SAM, being a zero-shot method, is unable to exploit these (generally undesirable) biases. The LVIS dataset has higher quality ground truth, but there are still specific idiosyncrasies (e.g., masks do not contain holes, they are simple polygons by construction) and biases for modal vs. amodal masks. Again, SAM is not trained to learn these biases, while ViTDet can exploit them.

我们假设,在 COCO 上,掩码 AP 差距更大且真值质量相对较低(人类研究证实了这一点),ViTDet 学习了 COCO 掩码的特定偏差。SAM 作为零样本方法,无法利用这些(通常不理想的)偏差。LVIS 数据集有更高质量的真值,但仍存在特有的癖好(例如,掩码不含孔洞,按构造是简单多边形)以及 modal 与 amodal 掩码的偏差。同样,SAM 没有被训练去学习这些偏差,而 ViTDet 可以利用它们。

零样本文本到掩码

Approach. Finally, we consider an even higher-level task: segmenting objects from free-form text. This experiment is a proof-of-concept of SAM’s ability to process text prompts. While we used the exact same SAM in all prior experiments, for this one SAM’s training procedure is modified to make it text-aware, but in a way that does not require new text annotations. Specifically, for each manually collected mask with area larger than 100² we extract the CLIP image embedding. Then, during training, we prompt SAM with the extracted CLIP image embeddings as its first interaction. The key observation here is that because CLIP’s image embeddings are trained to align with its text embeddings, we can train with image embeddings, but use text embeddings for inference. That is, at inference time we run text through CLIP’s text encoder and then give the resulting text embedding as a prompt to SAM (see §D.5 for details).

方法。最后,我们考虑一个更高层级的任务:从自由形式文本中分割物体。这个实验是对 SAM 处理文本提示能力的概念验证。虽然在之前所有实验中我们都使用完全相同的 SAM,但在本实验中,SAM 的训练过程被修改为感知文本,但以不需要新文本标注的方式进行。具体来说,对于每个面积大于 100² 的手动收集掩码,我们提取 CLIP 图像嵌入。然后在训练期间,我们用提取的 CLIP 图像嵌入作为 SAM 的第一次交互来提示它。这里的关键观察是:由于 CLIP 的图像嵌入被训练为与文本嵌入对齐,我们可以用图像嵌入训练,但用文本嵌入推理。也就是说,在推理时,我们将文本通过 CLIP 的文本编码器,然后将得到的文本嵌入作为提示提供给 SAM(详见 §D.5)。

Results. We show qualitative results in Fig. 12. SAM can segment objects based on simple text prompts like “a wheel” as well as phrases like “beaver tooth grille”. When SAM fails to pick the right object from a text prompt only, an additional point often fixes the prediction, similar to [31].

结果。我们在 Fig. 12 中展示定性结果。SAM 可以基于简单的文本提示(如「a wheel」)以及「beaver tooth grille」这样的短语分割物体。当 SAM 仅凭文本提示选错物体时,一个额外的点通常可以修正预测,与 [31] 类似。

消融实验

We perform several ablations on our 23 dataset suite with the single center point prompt protocol. Recall that a single point may be ambiguous and that ambiguity may not be represented in the ground truth, which contains only a single mask per point. Since SAM is operating in a zero-shot transfer setting there can be systematic biases between SAM’s top-ranked mask vs. the masks resulting from data annotation guidelines. We therefore additionally report the best mask with respect to the ground truth (“oracle”). Fig. 13 (left) plots SAM’s performance when trained on cumulative data from the data engine stages. We observe that each stage increases mIoU. When training with all three stages, the automatic masks vastly outnumber the manual and semi-automatic masks. To address this, we found that oversampling the manual and semi-automatic masks during training by 10× gave best results. This setup complicates training. We therefore tested a fourth setup that uses only the automatically generated masks. With this data, SAM performs only marginally lower than using all data (∼0.5 mIoU). Therefore, by default we use only the automatically generated masks to simplify the training setup. In Fig. 13 (middle) we look at the impact of data volume. The full SA-1B contains 11M images, which we uniformly subsample to 1M and 0.1M for this ablation. At 0.1M images, we observe a large mIoU decline under all settings. However, with 1M images, about 10% of the full dataset, we observe results comparable to using the full dataset. This data regime, which still includes approximately 100M masks, may be a practical setting for many use cases.

我们使用单中心点提示协议在 23 个数据集套件上进行了几项消融。回想一下,单个点可能存在歧义,且这种歧义可能不在真值中体现——真值每个点只包含一个掩码。由于 SAM 在零样本迁移设置下运行,SAM 排名第一的掩码与数据标注指南产生的掩码之间可能存在系统性偏差。因此,我们额外报告相对于真值的最佳掩码(「oracle」)。Fig. 13(左)绘制了 SAM 在数据引擎各阶段累积数据上训练的性能。我们观察到每个阶段都提高了 mIoU。当使用全部三个阶段训练时,自动掩码的数量远超手动和半自动掩码。为了解决这个问题,我们发现在训练期间对手动和半自动掩码进行 10 倍过采样效果最好。这种设置使训练复杂化。因此我们测试了第四种设置:仅使用自动生成的掩码。使用这些数据,SAM 的性能仅比使用全部数据略低(约 0.5 mIoU)。因此,默认情况下我们仅使用自动生成的掩码以简化训练设置。在 Fig. 13(中),我们考察数据量的影响。完整的 SA-1B 包含 11M 张图像,我们将其均匀子采样到 1M 和 0.1M 用于消融。在 0.1M 图像上,我们观察到所有设置下的 mIoU 都大幅下降。然而,使用 1M 图像(约占完整数据集的 10%)时,我们观察到与使用完整数据集相当的结果。这一数据规模仍包含约 100M 个掩码,对许多用例而言可能是实用的设置。

Finally, Fig. 13 (right) shows results with ViT-B, ViT-L, and ViT-H image encoders. ViT-H improves substantially over ViT-B, but has only marginal gains over ViT-L. Further image encoder scaling does not appear fruitful at this time.

最后,Fig. 13(右)展示了 ViT-B、ViT-L 和 ViT-H 图像编码器的结果。ViT-H 相比 ViT-B 有大幅提升,但相比 ViT-L 只有边际收益。进一步扩大图像编码器目前似乎没有成效。

讨论

Foundation models. Pre-trained models have been adapted to downstream tasks since the early days of machine learning [99]. This paradigm has become increasingly important in recent years with a growing emphasis on scale, and such models have recently been (re-)branded as “foundation models”: i.e. models that are “trained on broad data at scale and are adaptable to a wide range of downstream tasks” [8]. Our work correlates well with this definition, though we note that a foundation model for image segmentation is an inherently limited scope, since it represents an important, yet fractional, subset of computer vision. We also contrast one aspect of our approach with [8], which emphasizes the role of self-supervised learning in foundation models. While our model is initialized with a self-supervised technique (MAE [47]), the vast majority of its capabilities come from large-scale supervised training. In cases where data engines can scale available annotations, like ours, supervised training provides an effective solution.

基础模型。自机器学习早期以来,预训练模型就被适配到下游任务 [99]。近年来,随着对规模的日益重视,这一范式变得越来越重要,这类模型最近被(重新)命名为「基础模型」:即在广泛数据上大规模训练、可适配广泛下游任务的模型 [8]。我们的工作与该定义高度契合,不过我们注意到,图像分割的基础模型本质上范围有限,因为它只是计算机视觉中一个重要但局部的子集。我们还将我们方法的某一方面与 [8] 对比,后者强调自监督学习在基础模型中的作用。虽然我们的模型用自监督技术(MAE [47])初始化,但其绝大部分能力来自大规模监督训练。在数据引擎能够扩展可用标注的场景下(如我们的场景),监督训练提供了有效的解决方案。

Compositionality. Pre-trained models can power new capabilities even beyond ones imagined at the moment of training. One prominent example is how CLIP [82] is used as a component in larger systems, such as DALL·E [83]. Our goal is to make this kind of composition straightforward with SAM. We aim to achieve this by requiring SAM to predict a valid mask for a wide range of segmentation prompts. The effect is to create a reliable interface between SAM and other components. For example, MCC [106] can easily use SAM to segment an object of interest and achieve strong generalization to unseen objects for 3D reconstruction from a single RGB-D image. In another example, SAM can be prompted with gaze points detected by a wearable device, enabling new applications. Thanks to SAM’s ability to generalize to new domains like ego-centric images, such systems work without need for additional training.

组合性。预训练模型可以驱动训练时想象之外的新能力。一个突出的例子是 CLIP [82] 被用作更大系统(如 DALL·E [83])中的组件。我们的目标是让这种组合对 SAM 而言顺理成章。我们通过要求 SAM 对广泛的分割提示预测有效掩码来实现这一点,效果是在 SAM 与其他组件之间创建可靠的接口。例如,MCC [106] 可以轻松使用 SAM 分割感兴趣的物体,并在从单张 RGB-D 图像进行 3D 重建时对未见物体实现强泛化。在另一个例子中,SAM 可以用可穿戴设备检测到的注视点作为提示,催生新的应用。得益于 SAM 对新领域(如自我中心图像)的泛化能力,这类系统无需额外训练即可工作。

Limitations. While SAM performs well in general, it is not perfect. It can miss fine structures, hallucinates small disconnected components at times, and does not produce boundaries as crisply as more computationally intensive methods that “zoom-in”, e.g. [18]. In general, we expect dedicated interactive segmentation methods to outperform SAM when many points are provided, e.g. [67]. Unlike these methods, SAM is designed for generality and breadth of use rather than high IoU interactive segmentation. Moreover, SAM can process prompts in real-time, but nevertheless SAM’s overall performance is not real-time when using a heavy image encoder. Our foray into the text-to-mask task is exploratory and not entirely robust, although we believe it can be improved with more effort. While SAM can perform many tasks, it is unclear how to design simple prompts that implement semantic and panoptic segmentation. Finally, there are domain-specific tools, such as [7], that we expect to outperform SAM in their respective domains.

局限性。虽然 SAM 总体上表现良好,但它并不完美。它可能遗漏精细结构,有时会幻觉出小的不连通分量,且边界不如「放大」式计算密集方法(如 [18])那么清晰。总的来说,我们预期在提供许多点的情况下,专门的交互式分割方法会优于 SAM,如 [67]。与这些方法不同,SAM 是为通用性和使用广度而设计,而非高 IoU 交互式分割。此外,SAM 可以实时处理提示,但使用重型图像编码器时,SAM 的整体性能并非实时。我们对文本到掩码任务的尝试是探索性的,且不完全鲁棒,尽管我们相信通过更多努力可以改进。虽然 SAM 能执行许多任务,但如何设计简单提示来实现语义分割和全景分割尚不清楚。最后,有一些领域特定工具(如 [7]),我们预期它们在各自的领域内优于 SAM。

Conclusion. The Segment Anything project is an attempt to lift image segmentation into the era of foundation models. Our principal contributions are a new task (promptable segmentation), model (SAM), and dataset (SA-1B) that make this leap possible. Whether SAM achieves the status of a foundation model remains to be seen by how it is used in the community, but regardless we expect the perspective of this work, the release of over 1B masks, and our promptable segmentation model will help pave the path ahead.

结论。Segment Anything 项目是将图像分割带入基础模型时代的一次尝试。我们的主要贡献是实现这一跨越的新任务(可提示分割)、模型(SAM)和数据集(SA-1B)。SAM 是否能获得基础模型的地位,取决于社区如何使用它;但无论如何,我们期望这项工作的视角、超过 1B 掩码的发布以及我们的可提示分割模型,将有助于铺平前进的道路。

小结

这篇论文的目标一句话就能说清:让分割模型像 CLIP 一样,训练一次,之后用「提示」就能应付各种任务。给 SAM 一个点、一个框或者一行文字,它就输出对应的分割掩码——这就是「可提示分割」。传统分割模型每换一个任务就要重新训练,SAM 不需要。

为了实现这个目标,论文做了三件配套的事:定义任务(可提示分割)、设计模型(SAM)、构建数据集(SA-1B,11M 张图、1.1B 个掩码)。其中两个设计最有意思。

第一,把歧义当特性。一个点落在衬衫上,模型该分割衬衫还是穿衬衫的人?这就是歧义。传统交互式分割把这种问题当成缺陷,想办法避免;SAM 反过来利用它:一次输出 3 个候选掩码,每个附一个估计 IoU 分数。这样训练时只回传 3 个里损失最小的那个,模型不会把多个答案平均成四不像;自动标注时点落在物体的局部,3 个答案正好对应「局部、整体、更大的整体」,一次全标出来。没有这个设计,数据引擎的全自动阶段跑不起来,SA-1B 也就无从谈起。

第二,数据引擎自举。分割掩码在网上没有现成的大数据源,SAM 用「模型标数据 → 数据训模型」的闭环绕开了这个问题。分三阶段:先人工辅助标了 4.3M 个掩码(单掩码标注时间从 34s 降到 14s)→ 半自动再标 5.9M → 最后模型全自动标出 1.1B。最终 99.1% 的训练数据是模型自己生成的,而消融实验显示只用自动掩码、比混用三个阶段只低约 0.5 mIoU——这是论文里最反直觉、也最有说服力的数字。

评估方法上有个值得注意的细节:一个提示点可能对应多个物体,而真值里只有一个掩码,所以自动指标会系统性低估模型。论文用人类评分补充,并用 oracle(从 3 个输出里挑与真值最接近的那个)量化了这一点:SAM 默认在 23 个数据集里赢 16 个,oracle 下 23 个全赢。

不足分两类。论文自己承认的:会丢细节结构、用了重型图像编码器就不是实时、语义/全景分割没有现成的提示方案、文本到掩码只是概念验证。外部视角再补两点:一是规模,ViT-H 有 636M 参数,边缘端部署太重——MobileSAM / EfficientSAM / EdgeSAM 这些轻量化工作就是冲这个问题去的;二是数据质量,SA-1B 掩码质量是标注员用 SAM 辅助修正后比对出来的,不是完全独立的人工标注,模型自己的偏差会留在数据里。

后续演进顺着这条线走:SAM 2(2024)把可提示思路扩展到视频,加了流式记忆;SAM 3(2026)进一步统一;轻量化分支把交互能力压到移动端。组合方向上,SAM 的分割输出已经成为开放词汇检测、VLM 系统的标准组件——「被组合」这件事本身,就是这篇论文最成功的部分。