Otter¶

class mmpretrain.models.multimodal.Otter(vision_encoder, lang_encoder, tokenizer, task='caption', zeroshot_prompt='', shot_prompt_tmpl='<image>User:Please describe the image. GPT:<answer>{caption}<|endofchunk|>', final_prompt_tmpl='<image>User:Please describe the image. GPT:<answer>', generation_cfg={}, data_preprocessor=None, init_cfg=None)[source]¶

The Otter model for multiple tasks.

Parameters:

vision_encoder (dict) – The config of the vision encoder.
lang_encoder (dict) – The config of the language encoder.
tokenizer (dict) – The tokenizer to encode the text.
task (int) – The task to perform prediction.
zeroshot_prompt (str) – Prompt used for zero-shot inference. Defaults to an.
shot_prompt_tmpl (str) – Prompt used for few-shot inference. Defaults to <image>User:Please describe the image. GPT:<answer>{caption}<|endofchunk|>.
final_prompt_tmpl (str) – Final part of prompt used for inference. Defaults to ‘<image>User:Please describe the image. GPT:<answer>’.
generation_cfg (dict) – The extra generation config, accept the keyword arguments of [~`transformers.GenerationConfig`]. Defaults to an empty dict.
data_preprocessor (Optional[dict]) – The config for preprocessing input data. If None or no specified type, it will use “MutimodalDataPreprocessor” as type. See MutimodalDataPreprocessor for more details. Defaults to None.
init_cfg (dict, optional) – The initialization config. Defaults to None.

post_process(outputs, data_samples)[source]¶

Perform post process for outputs for different task.

Parameters:

outputs (torch.Tensor) – The generated outputs.
data_samples (List[DataSample], optional) – The annotation data of every samples.

Returns:

Return list of data samples.

Return type:

List[DataSample]