The I-JEPA model was proposed in Image-based Joint-Embedding Predictive Architecture by Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, Nicolas Ballas. I-JEPA is a self-supervised learning method that predicts the representations of one part of an image based on other parts of the same image. This approach focuses on learning semantic features without relying on pre-defined invariances from hand-crafted data transformations, which can bias specific tasks, or on filling in pixel-level details, which often leads to less meaningful representations.
The abstract from the paper is the following:
This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations. We introduce the Image- based Joint-Embedding Predictive Architecture (I-JEPA), a non-generative approach for self-supervised learning from images. The idea behind I-JEPA is simple: from a single context block, predict the representations of various target blocks in the same image. A core design choice to guide I-JEPA towards producing semantic representations is the masking strategy; specifically, it is crucial to (a) sample tar- get blocks with sufficiently large scale (semantic), and to (b) use a sufficiently informative (spatially distributed) context block. Empirically, when combined with Vision Transform- ers, we find I-JEPA to be highly scalable. For instance, we train a ViT-Huge/14 on ImageNet using 16 A100 GPUs in under 72 hours to achieve strong downstream performance across a wide range of tasks, from linear classification to object counting and depth prediction.
I-JEPA architecture. Taken from the original paper.This model was contributed by jmtzt. The original code can be found here.
How to useHere is how to use this model for image feature extraction:
import requests
import torch
from PIL import Image
from torch.nn.functional import cosine_similarity
from transformers import AutoModel, AutoProcessor
url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"
image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)
model_id = "facebook/ijepa_vith14_1k"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)
@torch.no_grad()
def infer(image):
inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
return outputs.last_hidden_state.mean(dim=1)
embed_1 = infer(image_1)
embed_2 = infer(image_2)
similarity = cosine_similarity(embed_1, embed_2)
print(similarity) Resources
A list of official Hugging Face and community (indicated by π) resources to help you get started with I-JEPA.
Image Classification
( hidden_size = 768 num_hidden_layers = 12 num_attention_heads = 12 intermediate_size = 3072 hidden_act = 'gelu' hidden_dropout_prob = 0.0 attention_probs_dropout_prob = 0.0 initializer_range = 0.02 layer_norm_eps = 1e-12 image_size = 224 patch_size = 16 num_channels = 3 qkv_bias = True pooler_output_size = None pooler_act = 'tanh' **kwargs )
Parameters
int, optional, defaults to 768) β Dimensionality of the encoder layers and the pooler layer. int, optional, defaults to 12) β Number of hidden layers in the Transformer encoder. int, optional, defaults to 12) β Number of attention heads for each attention layer in the Transformer encoder. int, optional, defaults to 3072) β Dimensionality of the βintermediateβ (i.e., feed-forward) layer in the Transformer encoder. str or function, optional, defaults to "gelu") β The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported. float, optional, defaults to 0.0) β The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. float, optional, defaults to 0.0) β The dropout ratio for the attention probabilities. float, optional, defaults to 0.02) β The standard deviation of the truncated_normal_initializer for initializing all weight matrices. float, optional, defaults to 1e-12) β The epsilon used by the layer normalization layers. int, optional, defaults to 224) β The size (resolution) of each image. int, optional, defaults to 16) β The size (resolution) of each patch. int, optional, defaults to 3) β The number of input channels. bool, optional, defaults to True) β Whether to add a bias to the queries, keys and values. int, optional) β Dimensionality of the pooler layer. If None, defaults to hidden_size. str, optional, defaults to "tanh") β The activation function to be used by the pooler. Keys of ACT2FN are supported for Flax and Pytorch, and elements of https://www.tensorflow.org/api_docs/python/tf/keras/activations are supported for Tensorflow. This is the configuration class to store the configuration of a IJepaModel. It is used to instantiate an IJEPA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the I-JEPA facebook/ijepa_vith14_1k architecture.
Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.
Example:
>>> from transformers import IJepaConfig, IJepaModel >>> >>> configuration = IJepaConfig() >>> >>> model = IJepaModel(configuration) >>> >>> configuration = model.configIJepaModel class transformers.IJepaModel < source >
( config: IJepaConfig add_pooling_layer: bool = False use_mask_token: bool = False )
Parameters
The bare IJepa Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward < source >( pixel_values: typing.Optional[torch.Tensor] = None bool_masked_pos: typing.Optional[torch.BoolTensor] = None head_mask: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_encoding: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) β transformers.modeling_outputs.BaseModelOutputWithPooling or tuple(torch.FloatTensor)
Parameters
torch.FloatTensor of shape (batch_size, num_channels, height, width)) β Pixel values. Pixel values can be obtained using AutoImageProcessor. See IJepaImageProcessor.__call__ for details. torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) β Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
bool, optional) β Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail. bool, optional) β Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail. bool, optional) β Whether to interpolate the pre-trained position encodings. bool, optional) β Whether or not to return a ModelOutput instead of a plain tuple. torch.BoolTensor of shape (batch_size, num_patches), optional) β Boolean masked positions. Indicates which patches are masked (1) and which arenβt (0). A transformers.modeling_outputs.BaseModelOutputWithPooling or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (IJepaConfig) and inputs.
last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) β Sequence of hidden-states at the output of the last layer of the model.
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) β Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) β Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) β Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
The IJepaModel forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Example:
>>> from transformers import AutoImageProcessor, IJepaModel
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/ijepa_vith14_1k")
>>> model = IJepaModel.from_pretrained("facebook/ijepa_vith14_1k")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
... outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 256, 1280] IJepaForImageClassification class transformers.IJepaForImageClassification < source >
( config: IJepaConfig )
Parameters
IJepa Model transformer with an image classification head on top (a linear layer on top of the final hidden states) e.g. for ImageNet.
Note that itβs possible to fine-tune IJepa on higher resolution images than the ones it has been trained on, by setting interpolate_pos_encoding to True in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution.
This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.
forward < source >( pixel_values: typing.Optional[torch.Tensor] = None head_mask: typing.Optional[torch.Tensor] = None labels: typing.Optional[torch.Tensor] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None interpolate_pos_encoding: typing.Optional[bool] = None return_dict: typing.Optional[bool] = None ) β transformers.modeling_outputs.ImageClassifierOutput or tuple(torch.FloatTensor)
Parameters
torch.FloatTensor of shape (batch_size, num_channels, height, width)) β Pixel values. Pixel values can be obtained using AutoImageProcessor. See IJepaImageProcessor.__call__ for details. torch.FloatTensor of shape (num_heads,) or (num_layers, num_heads), optional) β Mask to nullify selected heads of the self-attention modules. Mask values selected in [0, 1]:
bool, optional) β Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail. bool, optional) β Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail. bool, optional) β Whether to interpolate the pre-trained position encodings. bool, optional) β Whether or not to return a ModelOutput instead of a plain tuple. torch.LongTensor of shape (batch_size,), optional) β Labels for computing the image classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy). A transformers.modeling_outputs.ImageClassifierOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (IJepaConfig) and inputs.
loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) β Classification (or regression if config.num_labels==1) loss.
logits (torch.FloatTensor of shape (batch_size, config.num_labels)) β Classification (or regression if config.num_labels==1) scores (before SoftMax).
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) β Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape (batch_size, sequence_length, hidden_size). Hidden-states (also called feature maps) of the model at the output of each stage.
attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) β Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, patch_size, sequence_length).
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
The IJepaForImageClassification forward method, overrides the __call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.
Example:
>>> from transformers import AutoImageProcessor, IJepaForImageClassification
>>> import torch
>>> from datasets import load_dataset
>>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
>>> image = dataset["test"]["image"][0]
>>> image_processor = AutoImageProcessor.from_pretrained("facebook/ijepa_vith14_1k")
>>> model = IJepaForImageClassification.from_pretrained("facebook/ijepa_vith14_1k")
>>> inputs = image_processor(image, return_tensors="pt")
>>> with torch.no_grad():
... logits = model(**inputs).logits
>>>
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
Egyptian cat< > Update on GitHub
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3