> ## Documentation Index
> Fetch the complete documentation index at: https://api-tools.memories.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Start Audio Stream Transcription
> Start real-time audio stream transcription with ElevenLabs or AssemblyAI, controlled by a provider parameter.
**Product**: Visual Intelligence — Live Audio Transcription
**Use case**: Real-time speech-to-text on a live audio stream. For one-off file transcription, see Audio File Transcription.
**Mode**: Server-Pull (you give us an RTMP/RTSP stream URL → we pull → results to your callback). For client-controlled streaming, see the WebSocket variant.
**Host**: `https://mavi-backend.memories.ai/serve/api/v2`
**Auth**: `Authorization: sk-mavi-...` (no `Bearer` prefix)
**Access Required**: To use this API endpoint, please contact us at [contact@memories.ai](mailto:contact@memories.ai) to enable stream processing features for your account.
This endpoint starts real-time audio stream transcription. The server pulls audio from the provided stream URL, decodes it to PCM via FFmpeg, and streams it to the selected provider (ElevenLabs or AssemblyAI) over WebSocket. Every message returned by the provider is forwarded **verbatim** to your webhook callback. Billing occurs every 5 seconds of audio streamed.
**When to use this vs the WebSocket endpoint?**
* Use **this HTTP endpoint** when you have a stream URL (RTMP/RTSP/HLS) and want the server to handle audio decoding and streaming.
* Use the **[WebSocket endpoint](/visual-intelligence/stream/audio-stream-websocket)** when your client can send audio directly (e.g., browser microphone).
**Pricing (varies by provider):**
| Provider | Rate | Per 5s billing cycle | Per minute |
| -------------- | --------------- | -------------------- | ---------- |
| **AssemblyAI** | **\$0.15/hour** | \$0.000208 | \$0.0025 |
| **ElevenLabs** | **\$0.39/hour** | \$0.000542 | \$0.0065 |
```
Cost (USD) = duration × rate / 3600
```
Where:
* `duration`: Audio duration in seconds
* `rate`: $0.15 (AssemblyAI) or $0.39 (ElevenLabs)
* Charges: pre-check at start, then billed every 5 seconds of audio streamed
## Supported Protocols
* RTMP (Recommended)
* RTSP
* HLS (.m3u8)
* HTTP/HTTPS (direct audio URLs)
## Key Features
* Real-time transcription via ElevenLabs or AssemblyAI (controlled by `provider` parameter)
* Server-side audio decoding (FFmpeg) — no client-side processing needed
* Verbatim callback: every upstream message forwarded as-is to your webhook
* Real-time billing every 5 seconds of audio
* Auto-stop on insufficient balance (status 402)
* All provider-specific parameters transparently forwarded
## Architecture
```
Your Server
|
POST /audio-stream/start
{ audio_url, provider, ... }
|
v
+------------------+
| Memories.ai |
| |
audio_url -------> | FFmpeg (decode) |
| | |
| v |
| PCM 16kHz mono |
| | |
| v |
| WebSocket -------+-------> ElevenLabs / AssemblyAI
| |
| <-- messages ---|<------- Provider responses
| | |
| v |
| Webhook callback |-------> Your callback URL
+------------------+
```
### Code Example
```python Python (ElevenLabs) theme={null}
import requests
BASE_URL = "https://mavi-backend.memories.ai/serve/api/v2"
API_KEY = "sk-mavi-..."
HEADERS = {
"Authorization": f"{API_KEY}",
"Content-Type": "application/json"
}
def start_audio_stream(audio_url: str):
url = f"{BASE_URL}/audio-stream/start"
data = {
"audio_url": audio_url,
"provider": "elevenlabs",
"language_code": "en",
"model_id": "scribe_v2_realtime",
"diarize": True,
"num_speakers": 2
}
resp = requests.post(url, json=data, headers=HEADERS)
return resp.json()
result = start_audio_stream("rtmp://example.com/live/audio")
print(result)
print(f"Task ID: {result['data']['task_id']}")
```
```python Python (AssemblyAI) theme={null}
import requests
BASE_URL = "https://mavi-backend.memories.ai/serve/api/v2"
API_KEY = "sk-mavi-..."
HEADERS = {
"Authorization": f"{API_KEY}",
"Content-Type": "application/json"
}
def start_audio_stream(audio_url: str):
url = f"{BASE_URL}/audio-stream/start"
data = {
"audio_url": audio_url,
"provider": "assemblyai",
"language_code": "en",
"speaker_labels": True,
"speakers_expected": 2,
"punctuate": True,
"format_text": True
}
resp = requests.post(url, json=data, headers=HEADERS)
return resp.json()
result = start_audio_stream("rtmp://example.com/live/audio")
print(result)
print(f"Task ID: {result['data']['task_id']}")
```
```javascript JavaScript theme={null}
const axios = require('axios');
const BASE_URL = 'https://mavi-backend.memories.ai/serve/api/v2';
const API_KEY = 'sk-mavi-...';
const headers = {
'Authorization': API_KEY,
'Content-Type': 'application/json'
};
async function startAudioStream(audioUrl, provider = 'elevenlabs') {
const body = {
audio_url: audioUrl,
provider: provider,
language_code: 'en',
// ElevenLabs params
...(provider === 'elevenlabs' && {
model_id: 'scribe_v2_realtime',
diarize: true,
num_speakers: 2
}),
// AssemblyAI params
...(provider === 'assemblyai' && {
speaker_labels: true,
speakers_expected: 2,
punctuate: true,
format_text: true
})
};
const response = await axios.post(
`${BASE_URL}/audio-stream/start`,
body,
{ headers }
);
return response.data;
}
startAudioStream('rtmp://example.com/live/audio', 'elevenlabs')
.then(result => {
console.log(result);
console.log(`Task ID: ${result.data.task_id}`);
});
```
```bash cURL (ElevenLabs) theme={null}
curl -X POST "https://mavi-backend.memories.ai/serve/api/v2/audio-stream/start" \
-H "Authorization: sk-mavi-..." \
-H "Content-Type: application/json" \
-d '{
"audio_url": "rtmp://example.com/live/audio",
"provider": "elevenlabs",
"language_code": "en",
"model_id": "scribe_v2_realtime",
"diarize": true,
"num_speakers": 2
}'
```
```bash cURL (AssemblyAI) theme={null}
curl -X POST "https://mavi-backend.memories.ai/serve/api/v2/audio-stream/start" \
-H "Authorization: sk-mavi-..." \
-H "Content-Type: application/json" \
-d '{
"audio_url": "rtmp://example.com/live/audio",
"provider": "assemblyai",
"language_code": "en",
"speaker_labels": true,
"speakers_expected": 2,
"punctuate": true,
"format_text": true
}'
```
### Response
Returns the task information for the started audio stream.
```json Response theme={null}
{
"code": 200,
"message": "success",
"data": {
"task_id": "660e8400-e29b-41d4-a716-446655440001",
"message": "Audio stream transcription started"
}
}
```
```json Callback - ElevenLabs Message theme={null}
{
"code": 200,
"message": "SUCCESS",
"data": {
"status": 0,
"message": null,
"transcript": {
"message_type": "transcript",
"language_code": "en",
"language_probability": 0.98,
"text": "Hello, how are you doing today?",
"words": [
{
"text": "Hello",
"start": 0.0,
"end": 0.5,
"type": "word"
},
{
"text": "how",
"start": 0.6,
"end": 0.8,
"type": "word"
}
]
}
},
"task_id": "660e8400-e29b-41d4-a716-446655440001"
}
```
```json Callback - AssemblyAI Message theme={null}
{
"code": 200,
"message": "SUCCESS",
"data": {
"status": 0,
"message": null,
"transcript": {
"type": "final_transcript",
"text": "Smoke from hundreds of wildfires in Canada is triggering air quality alerts.",
"words": [
{
"text": "Smoke",
"start": 250,
"end": 650,
"confidence": 0.97503
}
],
"created": "2025-01-01T00:00:00.000Z"
}
},
"task_id": "660e8400-e29b-41d4-a716-446655440001"
}
```
```json Callback - Insufficient Balance theme={null}
{
"code": 200,
"message": "SUCCESS",
"data": {
"status": 402,
"message": "You don't have enough points.",
"transcript": null
},
"task_id": "660e8400-e29b-41d4-a716-446655440001"
}
```
```json Callback - User Stopped theme={null}
{
"code": 200,
"message": "SUCCESS",
"data": {
"status": 14,
"message": "Task stopped by user",
"transcript": null
},
"task_id": "660e8400-e29b-41d4-a716-446655440001"
}
```
### Request Parameters
### Required
| Parameter | Type | Description |
| ---------- | ------ | ---------------------------------------------------- |
| audio\_url | string | Audio stream URL (RTMP, RTSP, HLS, or HTTP) |
| provider | string | Transcription provider: `elevenlabs` or `assemblyai` |
### Common Parameters
| Parameter | Type | Default | Description |
| -------------- | ------ | ------- | -------------------------------------------------------------- |
| language\_code | string | - | Language code for transcription (e.g., `en`, `zh`, `es`, `fr`) |
### ElevenLabs Parameters
These parameters are forwarded when `provider=elevenlabs`.
| Parameter | Type | Default | Description |
| ------------------- | ------- | -------------------- | --------------------------------------------------- |
| model\_id | string | scribe\_v2\_realtime | Model to use for transcription |
| tag\_audio\_events | boolean | false | Tag audio events (music, laughter, etc.) |
| num\_speakers | integer | - | Expected number of speakers for diarization |
| diarize | boolean | false | Enable speaker diarization |
| enable\_logging | boolean | false | Enable server-side logging |
| inactivity\_timeout | integer | - | Session timeout (seconds) when no audio is received |
### AssemblyAI Parameters
These parameters are forwarded when `provider=assemblyai`.
| Parameter | Type | Default | Description |
| ------------------------------- | ------- | ------- | ----------------------------------------------------------- |
| punctuate | boolean | false | Add punctuation to the transcript |
| format\_text | boolean | false | Format text in the transcript (numbers, dates) |
| language\_detection | boolean | false | Enable automatic language detection |
| language\_confidence\_threshold | number | 0.0 | Confidence threshold for language detection (0.0-1.0) |
| audio\_start\_from | integer | 0 | Start transcription from this timestamp (ms) |
| audio\_end\_at | integer | 0 | End transcription at this timestamp (ms) |
| multichannel | boolean | false | Enable multi-channel audio processing |
| speech\_models | array | - | Array of speech models to use |
| speech\_threshold | number | 0.0 | Speech detection threshold (0.0-1.0) |
| disfluencies | boolean | false | Include disfluencies (um, uh) in transcript |
| speaker\_labels | boolean | false | Enable speaker diarization |
| speakers\_expected | integer | 0 | Expected number of speakers |
| sentiment\_analysis | boolean | false | Enable sentiment analysis |
| entity\_detection | boolean | false | Enable entity detection |
| auto\_highlights | boolean | false | Enable automatic highlights extraction |
| content\_safety | boolean | false | Enable content safety detection |
| iab\_categories | boolean | false | Enable IAB category classification |
| auto\_chapters | boolean | false | Enable automatic chapter generation |
| summarization | boolean | false | Enable automatic summarization |
| summary\_model | string | - | Model for summarization (`informative` or `conversational`) |
| summary\_type | string | - | Summary type (`bullets` or `paragraph`) |
| custom\_topics | boolean | false | Enable custom topic detection |
| topics | array | - | Array of custom topics to detect |
| redact\_pii | boolean | false | Redact personally identifiable information |
| redact\_pii\_sub | string | - | PII redaction substitution method |
| redact\_pii\_policies | array | - | Array of PII policies to apply |
| redact\_pii\_audio | boolean | false | Redact PII from audio |
| redact\_pii\_audio\_quality | string | - | Quality for PII audio redaction |
| filter\_profanity | boolean | false | Filter profanity from transcript |
| custom\_spelling | array | - | Array of custom spelling corrections |
| speech\_understanding | object | - | Speech understanding configuration |
Any parameters not listed above can still be passed in the request body. They will be captured and forwarded to the upstream provider via the URL query string. This ensures forward compatibility with new provider features.
### Response Parameters
| Parameter | Type | Description |
| ------------- | ------ | ------------------------------------------- |
| code | string | Response code (200 indicates success) |
| message | string | Response message |
| data.task\_id | string | Unique identifier of the transcription task |
| data.message | string | Status message about the stream start |
### Callback Response Parameters
Callbacks are sent continuously — one for each message received from the upstream provider.
| Parameter | Type | Description |
| --------------- | ------- | ------------------------------------------------------------------------------ |
| code | string | Response code (200 indicates callback delivery success) |
| message | string | Response message ("SUCCESS") |
| task\_id | string | The task ID associated with this stream |
| data.status | integer | Status code (0 for normal message, see Status Codes below) |
| data.message | string | Status message (null for normal messages) |
| data.transcript | object | **Verbatim JSON from the upstream provider** (null for error/control statuses) |
The `data.transcript` field contains the **raw, unmodified response** from the selected provider. The structure differs between ElevenLabs and AssemblyAI. Refer to each provider's documentation for detailed field descriptions.
### Status Codes
| Status | Name | Description | Stream Continues |
| ------ | -------------------- | ------------------------------------------ | ---------------- |
| 0 | Message | Normal transcription message from provider | Yes |
| -1 | Error | Processing or connection error | No |
| 14 | User Stopped | User called `/audio-stream/stop` | No |
| 16 | Capacity Reached | Server capacity limit reached | No |
| 402 | Insufficient Balance | User balance insufficient | No |
### Important Notes
* **`provider` is required**: You must specify `elevenlabs` or `assemblyai`. Without it the request will fail.
* **Webhook required**: Configure your webhook URL in user settings before using this API.
* **Verbatim callbacks**: Each callback contains the exact JSON message from the provider — the server does not transform or aggregate the data.
* **Real-time billing**: Billing occurs every 5 seconds of audio data streamed to the provider. Auto-stops when balance is insufficient.
* **Pre-charge**: Balance is checked at start (one 5-second unit). If insufficient, the task is not started (status 402).
* **Parameter transparency**: Parameters specific to a provider are forwarded as-is. Parameters not relevant to the selected provider are silently ignored by the provider.
### Supported Languages
Common language codes (supported by both providers):
* `en` - English
* `zh` - Chinese
* `es` - Spanish
* `fr` - French
* `de` - German
* `ja` - Japanese
* `ko` - Korean
* And many more...
### Rate Limiting
* **Maximum concurrent streams**: Each user can run N concurrent stream tasks (video + audio combined)
* **Capacity check**: Returns status 16 if server capacity is reached
* **Balance check**: Returns status 402 if insufficient balance at start
## OpenAPI
````yaml visual-intelligence/stream/openapi.json POST /audio-stream/start
openapi: 3.0.0
info:
title: Stream Processing API
version: 2.0.0
description: Real-time video and audio stream processing APIs
servers:
- url: https://mavi-backend.memories.ai/serve/api/v2
description: Production server
security: []
paths:
/audio-stream/start:
post:
tags:
- Audio Stream
summary: Start Audio Stream Transcription
description: Start real-time audio stream transcription with AssemblyAI
operationId: startAudioStream
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- audio_url
properties:
audio_url:
type: string
description: Audio stream URL (RTMP or RTSP protocol)
example: rtmp://example.com/live/audio
language_code:
type: string
description: Language code for transcription
example: en
punctuate:
type: boolean
description: Add punctuation to the transcript
default: false
format_text:
type: boolean
description: Format text in the transcript
default: false
language_detection:
type: boolean
description: Enable automatic language detection
default: false
language_confidence_threshold:
type: number
description: Confidence threshold for language detection
default: 0
audio_start_from:
type: integer
description: Start transcription from this timestamp (milliseconds)
default: 0
audio_end_at:
type: integer
description: End transcription at this timestamp (milliseconds)
default: 0
multichannel:
type: boolean
description: Enable multi-channel audio processing
default: false
speech_models:
type: array
description: Array of speech models to use
items:
type: string
speech_threshold:
type: number
description: Speech detection threshold (0.0-1.0)
default: 0
disfluencies:
type: boolean
description: Include disfluencies in the transcript
default: false
speaker_labels:
type: boolean
description: Enable speaker diarization
default: false
speakers_expected:
type: integer
description: Expected number of speakers
default: 0
sentiment_analysis:
type: boolean
description: Enable sentiment analysis
default: false
entity_detection:
type: boolean
description: Enable entity detection
default: false
auto_highlights:
type: boolean
description: Enable automatic highlights extraction
default: false
content_safety:
type: boolean
description: Enable content safety detection
default: false
iab_categories:
type: boolean
description: Enable IAB category classification
default: false
auto_chapters:
type: boolean
description: Enable automatic chapter generation
default: false
summarization:
type: boolean
description: Enable automatic summarization
default: false
summary_model:
type: string
description: Model to use for summarization
summary_type:
type: string
description: Type of summary to generate
custom_topics:
type: boolean
description: Enable custom topic detection
default: false
topics:
type: array
description: Array of custom topics to detect
items:
type: string
redact_pii:
type: boolean
description: Redact personally identifiable information
default: false
redact_pii_sub:
type: string
description: PII redaction substitution method
redact_pii_policies:
type: array
description: Array of PII policies to apply
items:
type: string
redact_pii_audio:
type: boolean
description: Redact PII from audio
default: false
redact_pii_audio_quality:
type: string
description: Quality setting for PII audio redaction
filter_profanity:
type: boolean
description: Filter profanity from transcript
default: false
custom_spelling:
type: array
description: Array of custom spelling corrections
items:
type: object
speech_understanding:
type: object
description: Speech understanding configuration
responses:
'200':
description: Audio stream started successfully
content:
application/json:
schema:
type: object
properties:
code:
type: string
example: 200
message:
type: string
example: success
data:
type: object
properties:
task_id:
type: string
example: 660e8400-e29b-41d4-a716-446655440001
message:
type: string
example: Audio stream transcription started successfully
security:
- ApiKeyAuth: []
components:
securitySchemes:
ApiKeyAuth:
type: apiKey
in: header
name: Authorization
````