Segmentation – identifying which image pixels belong to an object – assists with tasks such as analyzing scientific imagery or editing photos. The original Segment Anything Model (SAM), released last year, inspired new AI-enabled image editing tools in applications like Backdrop and Cutouts on Instagram. SAM has also spurred diverse applications in science, medicine, and various other industries. For instance, SAM has been utilized in marine science to segment sonar images and analyze coral reefs, in satellite imagery analysis for disaster relief, and in the medical field for segmenting cellular images and aiding in detecting skin cancer.
The new Segment Anything Model 2 (SAM 2) extends these capabilities to video. SAM 2 can segment any object in an image or video and consistently follow it across all frames of a video in real-time. Existing models have struggled with this due to the complexities of segmentation in video, where objects can move quickly, change appearance, or be obscured by other objects or parts of the scene. Many of these challenges were addressed during the development of SAM 2.
"We believe this research can unlock new possibilities such as easier video editing and generation," said a spokesperson for the team behind SAM 2. They added that SAM 2 could also be used "to track a target object in a video to aid in faster annotation of visual data for training computer vision systems," including those used in autonomous vehicles. Additionally, it may enable creative ways of selecting and interacting with objects in real-time or live videos.
In line with their open science approach, the team is sharing their research on SAM 2 so others can explore new capabilities and use cases. "We’re excited to see what the AI community does with this research," they stated.
