> ## Documentation Index
> Fetch the complete documentation index at: https://api-tools.memories.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal Speaker Recognition

> Identify named speakers by combining voice diarization with face recognition.

<Info>
  **Product**: Visual Intelligence — Audio File Transcription
  **Use case**: Transcribe an uploaded audio/video **file** to text — async batch or sync, multiple providers (Whisper, ElevenLabs, AssemblyAI) with optional speaker labels. For live streams, see Live Audio Transcription.
  **Host**: `https://mavi-backend.memories.ai/serve/api/v2`
  **Auth**: `Authorization: sk-mavi-...` (no `Bearer` prefix)
</Info>

Identifies **who said what** by combining audio speaker diarization (pyannote) with face recognition (Gemini). Returns transcription segments labeled with real names alongside face images extracted from the video.

**Different from Speaker Diarization**: [Speaker Diarization](/visual-intelligence/transcript/speaker-diarization) only assigns anonymous labels (`SPEAKER_00`, `SPEAKER_01`). Multimodal Speaker Recognition matches voices to faces to produce named speaker identification — but costs \~100× more.

<Warning>
  This is an **async endpoint**. You must configure a webhook URL in [Webhooks Settings](https://api-platform.memories.ai/webhooks) before calling this endpoint, otherwise you will not receive the processing results. See [Webhooks Configuration Guide](/visual-intelligence/getting-started/webhooks) for details.
</Warning>

<Note>
  **Pricing:**

  * \$0.1/min of video and audio
</Note>

### Request Body

| Parameter | Type   | Required | Description                                                                        |
| --------- | ------ | -------- | ---------------------------------------------------------------------------------- |
| asset\_id | string | Yes      | The unique identifier of the video or audio asset for multi-speaker identification |

### Code Example

<CodeGroup>
  ```bash cURL theme={null}
  curl --request POST \
    --url https://mavi-backend.memories.ai/serve/api/v2/transcriptions/async-generate-multi-speaker \
    --header 'Authorization: sk-mavi-...' \
    --header 'Content-Type: application/json' \
    --data '{
      "asset_id": "re_657929111888723968"
    }'
  ```

  ```javascript fetch theme={null}
  const BASE_URL = "https://mavi-backend.memories.ai/serve/api/v2/transcriptions";
  const API_KEY = "sk-mavi-...";

  const response = await fetch(`${BASE_URL}/async-generate-multi-speaker`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': API_KEY
    },
    body: JSON.stringify({
      asset_id: 're_657929111888723968'
    })
  });

  const data = await response.json();
  console.log(data);
  ```

  ```python Python theme={null}
  import requests

  BASE_URL = "https://mavi-backend.memories.ai/serve/api/v2/transcriptions"
  API_KEY = "sk-mavi-..."
  HEADERS = {
      "Authorization": f"{API_KEY}"
  }

  def async_generate_multi_speaker(asset_id: str):
      url = f"{BASE_URL}/async-generate-multi-speaker"
      data = {"asset_id": asset_id}
      resp = requests.post(url, json=data, headers=HEADERS)
      return resp.json()

  # Usage example
  result = async_generate_multi_speaker("re_657929111888723968")
  print(result)
  ```
</CodeGroup>

### Response

Returns the multi-speaker identification task information.

<ResponseExample>
  ```json Response theme={null}
  {
    "code": 200,
    "msg": "success",
    "data": {
      "task_id": "ec2449885ba84c4f943a80ff0633158e"
    },
    "failed": false,
    "success": true
  }
  ```

  ```json Callback Response theme={null}
  {
    "code": 200,
    "message": "SUCCESS",
    "data": {
      "data": {
        "audio_transcription": [
          {
            "end_time": 1.8,
            "speaker": "Kiara S Stepsister",
            "start_time": 0.0,
            "text": "You wolfless Omega! Clean that up, you jinx!"
          }
        ],
        "faces": [
          {
            "face_file_blob": "api-backend/0c5dfd30-7285-4e1e-bd20-dd31fd405365/multimodal-asr/9e545636_batch_1_video_9_person_001.jpg",
            "face_file_bucket": "memories-cache",
            "face_file_protocol": "gs",
            "face_id": "9e545636-509a-4a7d-b7c8-6359ea6a6d8b_person_001",
            "name": "Kiara S Stepsister"
          }
        ],
        "usage_metadata": [
          {
            "duration": 0.0,
            "model": "gemini-2.5-pro",
            "output_tokens": 6143,
            "prompt_tokens": 442368
          },
          {
            "duration": 0.0,
            "model": "gemini-2.5-flash",
            "output_tokens": 15643,
            "prompt_tokens": 32508
          }
        ]
      },
      "msg": "Multimodal ASR completed successfully",
      "success": true
    },
    "task_id": "29799938cfd344db8e10243a266b9990"
  }
  ```
</ResponseExample>

### Response Parameters

| Parameter     | Type    | Description                                                |
| ------------- | ------- | ---------------------------------------------------------- |
| code          | string  | Response code indicating the result status                 |
| msg           | string  | Response message describing the operation result           |
| data          | object  | Response data object containing task information           |
| data.task\_id | string  | Unique identifier of the multi-speaker identification task |
| success       | boolean | Indicates whether the operation was successful             |
| failed        | boolean | Indicates whether the operation failed                     |

### Callback Response Parameters

When the multi-speaker identification is complete, a callback will be sent to your configured webhook URL.

| Parameter                                     | Type    | Description                                                              |
| --------------------------------------------- | ------- | ------------------------------------------------------------------------ |
| code                                          | string  | Response code (200 indicates success)                                    |
| message                                       | string  | Status message (e.g., "SUCCESS")                                         |
| data                                          | object  | Response data object containing the multimodal ASR result and metadata   |
| data.data                                     | object  | Inner data object containing transcription, faces, and usage information |
| data.data.audio\_transcription                | array   | Array of transcription segments with speaker identification              |
| data.data.audio\_transcription\[].start\_time | number  | Start time of the segment in seconds                                     |
| data.data.audio\_transcription\[].end\_time   | number  | End time of the segment in seconds                                       |
| data.data.audio\_transcription\[].speaker     | string  | Identified speaker name                                                  |
| data.data.audio\_transcription\[].text        | string  | Transcription text for this segment                                      |
| data.data.faces                               | array   | Array of detected faces with metadata                                    |
| data.data.faces\[].face\_id                   | string  | Unique identifier for the detected face                                  |
| data.data.faces\[].name                       | string  | Identified name of the person                                            |
| data.data.faces\[].face\_file\_protocol       | string  | Storage protocol (e.g., "gs" for Google Cloud Storage)                   |
| data.data.faces\[].face\_file\_bucket         | string  | Storage bucket name                                                      |
| data.data.faces\[].face\_file\_blob           | string  | File path in the storage bucket                                          |
| data.data.usage\_metadata                     | array   | Array of usage statistics for different models used                      |
| data.data.usage\_metadata\[].duration         | number  | Processing duration in seconds                                           |
| data.data.usage\_metadata\[].model            | string  | The AI model used (e.g., "gemini-2.5-pro", "gemini-2.5-flash")           |
| data.data.usage\_metadata\[].output\_tokens   | integer | Number of tokens in the output                                           |
| data.data.usage\_metadata\[].prompt\_tokens   | integer | Number of tokens in the input prompt                                     |
| data.msg                                      | string  | Detailed message about the operation result                              |
| data.success                                  | boolean | Indicates whether the multimodal ASR was successful                      |
| task\_id                                      | string  | The task ID associated with this multi-speaker identification request    |


## OpenAPI

````yaml POST /transcriptions/async-generate-multi-speaker
openapi: 3.1.0
info:
  title: MAVI API Reference
  description: REST APIs for memory management, search, and entity operations
  version: v1.0.1
servers:
  - url: https://mavi-backend.memories.ai/serve/api/v2
security:
  - ApiKeyAuth: []
paths:
  /transcriptions/async-generate-multi-speaker:
    post:
      summary: Async Generate Multi Speaker
      description: Generate multi-speaker identification asynchronously.
      operationId: async_generate_multi_speaker
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                asset_id:
                  type: string
                  description: The asset ID to identify multiple speakers for
                  example: re_657929111888723968
              required:
                - asset_id
      responses:
        '200':
          description: Multi-speaker identification task information
          content:
            application/json:
              schema:
                type: object
                properties:
                  code:
                    type: string
                    example: 200
                    description: Response code indicating the result status
                  msg:
                    type: string
                    example: success
                    description: Response message describing the operation result
                  data:
                    type: object
                    properties:
                      task_id:
                        type: string
                        example: ec2449885ba84c4f943a80ff0633158e
                        description: >-
                          Unique identifier of the multi-speaker identification
                          task
                    description: Response data object containing task information
                  success:
                    type: boolean
                    example: true
                    description: Indicates whether the operation was successful
                  failed:
                    type: boolean
                    example: false
                    description: Indicates whether the operation failed
components:
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: Authorization

````