Note:
To use voice in watsonx Orchestrate Developer Edition, enable the voice feature by adding the
--with-voice flag to the orchestrate server start command. For more information, see Installing watsonx Orchestrate Developer Edition: watsonx Orchestrate server.Creating voice configurations
To create a voice configuration, first create a YAML file that defines your voice settings. This file includes the voice name, speech-to-text and text-to-speech provider settings, the primary language, and optional configurations for advanced settings.Supported Providers
Speech-to-Text (STT) Providers:watson_stt: IBM Watson Speech to Text - Docsdeepgram_stt: Deepgram Speech to Text - Docselevenlabs_stt: ElevenLabs Speech to Text - Docsgoogle_stt: Google Cloud Speech-to-Text - Docsazure_stt: Azure Cognitive Services Speech to Text - Docs
watson_tts: IBM Watson Text to Speech - Docsdeepgram_tts: Deepgram Text to Speech - Docselevenlabs_tts: ElevenLabs Text to Speech - Docsgoogle_tts: Google Cloud Text-to-Speech - Docsazure_tts: Azure Cognitive Services Text to Speech - Docs
Note:
Please refer to the official documentations of each providers for more information on models and parameters.
Configuration Examples
Watson STT/TTS Configuration
Basic Example:YAML
name: watson_voice_config
speech_to_text:
provider: watson_stt
watson_stt_config:
api_url: "watson_stt_url"
api_key: "your_watson_stt_api_key"
model: "en-US"
text_to_speech:
provider: watson_tts
watson_tts_config:
api_url: "watson_tts_url"
api_key: "your_watson_tts_api_key"
voice: "en-US_AllisonV3Voice"
language: "en-US"
YAML
name: watson_voice_config
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: watson_stt
watson_stt_config:
api_url: "https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/your-instance-id"
api_key: "your_watson_stt_api_key"
model: "en-US_Telephony"
background_audio_suppression: 0.5
language_customization_id: null
inactivity_timeout: 30
profanity_filter: true
smart_formatting: true
speaker_labels: false
redaction: false
low_latency: true
learning_opt_out: false
watson_metadata: null
smart_formatting_version: null
customization_weight: null
character_insertion_bias: null
end_of_phrase_silence_time: 0.8
text_to_speech:
provider: watson_tts
watson_tts_config:
api_url: "https://api.us-south.text-to-speech.watson.cloud.ibm.com/instances/your-instance-id"
api_key: "your_watson_tts_api_key"
voice: "en-US_AllisonV3Voice"
language: "en-US"
rate_percentage: 0
pitch_percentage: 0
customization_id: null
meta_id: null
learning_opt_out: false
language: "en-US"
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Watson Speech to Text service URL
string
required
Watson Speech to Text API key
string
required
Speech recognition model (e.g., “en-US_Telephony”)
float
Background noise suppression level (0.0-1.0)
string
Language customization ID for custom language models
integer
Seconds of inactivity before stopping (-1 for no timeout, default: 30)
boolean
Filter profanity in transcript
boolean
Enable smart formatting (converts dates, currency, etc. to readable format)
boolean
Enable speaker labels (beta feature)
boolean
Redact numbers with three or more consecutive digits
boolean
Enable low latency mode. Available only for some next-generation models. See Watson STT Models.
boolean
Opt out of data collection for learning purposes
string
Value for x-watson-metadata header (max 512 characters)
integer
Version of smart formatting to use
float
Weight for custom language model (0.0 to 1.0, default: 0.5)
float
Bias for character insertion (-1.0 to 1.0, default: 0.0)
float
Seconds of silence indicating end of phrase (0.0-120.0)
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Watson Text to Speech service URL
string
required
Watson Text to Speech API key
string
required
Voice model (e.g., “en-US_AllisonV3Voice”)
integer
Speech rate adjustment percentage (default: 0)
integer
Speech pitch adjustment percentage (default: 0)
string
Language code for the voice (e.g., “en-US”)
string
Custom ID for the Watson TTS service
string
Meta ID for the Watson TTS service
boolean
Opt out of data collection for learning purposes
Deepgram STT/TTS Configuration
Basic Example:YAML
name: deepgram_voice_config
speech_to_text:
provider: deepgram_stt
deepgram_stt_config:
api_url: "wss://api.deepgram.com/v1/listen"
api_key: "your_deepgram_api_key"
model: "nova-2"
text_to_speech:
provider: deepgram_tts
deepgram_tts_config:
api_key: "your_deepgram_api_key"
YAML
name: deepgram_voice_config
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: deepgram_stt
deepgram_stt_config:
api_url: "wss://api.deepgram.com/v1/listen"
api_key: "your_deepgram_api_key"
model: "nova-3"
keyterm: ["help", "search", "Mr. Smith"]
mip_opt_out: false
channels: 1
diarize: true
dictation: false
endpointing: 10
extra: null
interim_results: true
keywords: null
language: "en-US"
multichannel: false
numerals: true
profanity_filter: false
punctuate: true
redact: null
replace: ["apple:orange", "orange:apple"]
search: ["action item", "follow up"]
smart_format: true
tag: ["customer_service", "sales_call"]
utterance_end_ms: 1000
vad_events: true
version: "latest"
eager_eot_threshold: null
eot_threshold: null
eot_timeout_ms: null
text_to_speech:
provider: deepgram_tts
deepgram_tts_config:
api_key: "your_deepgram_api_key"
api_url: "https://api.deepgram.com/v1/speak"
language: "en"
model: "aura-2-thalia-en"
mip_opt_out: false
language: "en-US"
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Deepgram WebSocket API URL (e.g., “wss://api.deepgram.com/v1/listen”)
string
required
Deepgram API key
string
required
Speech recognition model (e.g., “nova-2”, “nova-3”). Only
nova-2 and nova-3 models (and their variations like nova-2-finance, nova-3-medical) are officially supported.string
Language code (e.g., “en-US”)
string
Language hint for Flux multilingual model to improve transcription accuracy. See Flux Multilingual & Language Prompting for supported languages and usage. Do not set this parameter when using the flux-general-en model.
boolean
Convert numbers to numerals
array
List of keywords to boost recognition accuracy (e.g., [“help”, “support”, “Mr. Smith”]). Only supported for nova-3 models and their variations. Using this parameter with nova-2 models will result in a validation error.
array
List of keywords to detect with optional boost/suppression values (e.g., [“snuffleupagus:5”, “monster”, “cookie:-2”]). Only supported for nova-2 models and their variations. Using this parameter with nova-3 models will result in a validation error.
boolean
Opt out of model improvement program
integer
Number of audio channels
boolean
Enable speaker diarization
boolean
Enable dictation mode
integer
Endpointing silence duration in seconds
array
Extra parameters to pass to Deepgram (e.g., [“custom_param1:value1”])
boolean
Enable interim results
boolean
Transcribe each audio channel independently
boolean
Filter profanity
boolean
Add punctuation and capitalization
array
Redact sensitive information (e.g., [“pci”, “pii”])
array
Replace specified terms (e.g., [“apple:orange”, “orange:apple”])
array
Search for specific terms (e.g., [“action item”, “follow up”])
boolean
Apply smart formatting to the transcript
array
Tags for the request (e.g., [“customer_service”, “sales_call”])
integer
How long Deepgram will wait to send UtteranceEnd message after word has been transcribed
boolean
Enable Deepgram’s voice activity detection events
string
API version (e.g., “latest”)
float
Only for v2 endpoint. End-of-turn confidence required to fire an eager EOT event (0.3-0.9)
float
Only for v2 endpoint. End-of-turn confidence required to finish a turn (0.5-0.9)
integer
Only for v2 endpoint. Time (ms) after speech to finish turn regardless of EOT confidence
Show detailed parameter descriptions
Show detailed parameter descriptions
ElevenLabs STT/TTS Configuration
Basic Example:YAML
name: elevenlabs_voice_config
speech_to_text:
provider: elevenlabs_stt
elevenlabs_stt_config:
api_key: "your_elevenlabs_api_key"
model_id: "scribe_v2_realtime"
text_to_speech:
provider: elevenlabs_tts
elevenlabs_tts_config:
api_key: "your_elevenlabs_api_key"
model_id: "eleven_turbo_v2_5"
voice_id: "CwhRBWXzGAHq8TQ4Fs17"
YAML
name: elevenlabs_voice_config
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: elevenlabs_stt
elevenlabs_stt_config:
api_key: "your_elevenlabs_api_key"
model_id: "scribe_v2_realtime"
language_code: "en"
commit_strategy: "manual"
include_timestamps: false
enable_logging: false
include_language_detection: false
min_speech_duration_ms: null
min_silence_duration_ms: null
vad_silence_threshold_secs: null
vad_threshold: null
text_to_speech:
provider: elevenlabs_tts
elevenlabs_tts_config:
api_url: "https://api.elevenlabs.io/v1/text-to-speech"
api_key: "your_elevenlabs_api_key"
model_id: "eleven_turbo_v2_5"
voice_id: "CwhRBWXzGAHq8TQ4Fs17"
language_code: "en"
apply_text_normalization: "auto"
optimize_streaming_latency: 3
apply_language_text_normalization: false
pronunciation_dictionary_locators: null
seed: 42
voice_settings:
speed: 1.0
stability: 0.5
style: 0.0
similarity_boost: 0.75
use_speaker_boost: true
language: "en-US"
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
ElevenLabs API key. The WebSocket endpoint is fixed at
wss://api.elevenlabs.io/v1/speech-to-text/realtime and is not configurable.string
ElevenLabs transcription model to use (default:
"scribe_v2_realtime")string
ISO-639-1 or ISO-639-3 language code (e.g.,
"en", "fra"). When absent, ElevenLabs auto-detects the language.string
How to segment speech.
"manual" uses Pipecat’s VAD to commit transcript segments; "vad" delegates commit decisions to ElevenLabs’ own VAD. (default: "manual")boolean
Include word-level timestamps in transcripts. (default: false)
boolean
Enable logging on ElevenLabs’ side. (default: false)
boolean
Include per-utterance language detection metadata in transcripts. (default: false)
float
Seconds of silence before ElevenLabs VAD commits a segment (0.3–3.0). Only applies when
commit_strategy is "vad".float
VAD sensitivity (0.1–0.9), where lower is more sensitive. Only applies when
commit_strategy is "vad".integer
Minimum speech duration for VAD, in milliseconds (50–2000). Only applies when
commit_strategy is "vad".integer
Minimum silence duration for VAD, in milliseconds (50–2000). Only applies when
commit_strategy is "vad".Show detailed parameter descriptions
Show detailed parameter descriptions
string
ElevenLabs API URL (e.g., “https://api.elevenlabs.io/v1/text-to-speech”)
string
required
ElevenLabs API key
string
required
TTS model ID (e.g., “eleven_turbo_v2_5”)
string
required
Voice ID from ElevenLabs (e.g., “CwhRBWXzGAHq8TQ4Fs17”)
string
Language code (e.g., “en”, “ja” for Japanese)
string
Text normalization mode (“auto”, “on”, “off”)
integer
Optimize streaming latency (0-4)
boolean
Apply language-specific text normalization. Currently only supported for Japanese (
ja). Using this parameter with other languages will result in a validation error.array
List of pronunciation dictionary locators with
pronunciation_dictionary_id and version_idinteger
Seed for deterministic audio generation
object
Advanced voice customization settings
float
Speech speed (default: 1.0)
float
Voice stability - how stable the voice is and the randomness between each generation (0.0-1.0, default: 0.5)
float
Style exaggeration - the higher the value, the more computational resources are used (0.0-1.0, default: 0.0)
float
Similarity boost - how closely the AI should adhere to the original voice (0.0-1.0, default: 0.75)
boolean
Enable speaker boost (default: true)
Google STT/TTS Configuration
Basic Example:YAML
name: google_voice_config
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: google_stt
google_stt_config:
project_id: "project-id"
credentials_json: "service-account-json"
language_code: "en-US"
model: "latest_long"
enable_automatic_punctuation: true
enable_interim_results: true
text_to_speech:
provider: google_tts
google_tts_config:
credentials_json: "service-account-json"
voice: "en-US-Neural2-C"
language: "en-US"
speaking_rate: 1.0
language: "en-US"
YAML
name: google_voice_config_advanced
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: google_stt
google_stt_config:
project_id: "your-gcp-project-id"
credentials_json: "service-account-json"
language_code: "en-US"
alternative_language_codes:
- "es-US"
- "fr-CA"
model: "latest_long"
use_enhanced: true
max_alternatives: 3
profanity_filter: true
enable_automatic_punctuation: true
enable_spoken_punctuation: true
enable_interim_results: true
enable_word_time_offsets: true
enable_word_confidence: true
text_to_speech:
provider: google_tts
google_tts_config:
credentials_json: "service-account-json"
voice: "en-US-Neural2-F"
language: "en-US"
ssml_gender: "FEMALE"
speaking_rate: 1.0
pitch: 0.0
volume_gain_db: 0.0
effects_profile_id:
- "telephony-class-application"
language: "en-US"
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Google Cloud project ID
string
required
Service Account credentials JSON string
string
required
Primary recognition language as a BCP-47 tag (e.g., “en-US”)
array
Up to 3 additional BCP-47 language tags for multi-language recognition. Only supported for Voice Command and Voice Search use cases (e.g., [“es-US”, “fr-CA”])
string
Speech recognition model variant. Options:
latest_long | latest_short | command_and_search | phone_call | video | default | medical_conversation | medical_dictationboolean
Use the enhanced model variant when available. Falls back to standard model if an enhanced variant does not exist
integer
Maximum number of recognition hypotheses returned (0–30). Defaults to 1
boolean
Replace profanities with
*** style masking. Defaults to falseboolean
Add punctuation to recognition results. Defaults to false
boolean
Replace spoken punctuation commands with the corresponding symbols (e.g., “question mark” → ”?”). Defaults to model-specific behavior
boolean
Enable streaming partial (interim) results before the final transcript is ready
boolean
Include start and end timestamps for each word in the top result. Defaults to false
boolean
Include per-word confidence scores in the top result. Defaults to false
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Service Account credentials JSON string. Can be substituted with
api_keystring
Google Cloud API key. Can be substituted with
credentials_jsonstring
required
Full voice name (e.g., “en-US-Neural2-F”). Use Neural2, WaveNet, Studio, or Standard voices to enable
speaking_rate, pitch, and volume_gain_db. Chirp 3 HD voices do not support these parameters.string
required
BCP-47 language tag (e.g., “en-US”). The service may substitute a nearby dialect if an exact match is unavailable
string
Preferred voice gender —
MALE | FEMALE | NEUTRAL. Not enforced if a voice with a matching gender is unavailablefloat
Speech speed (0.25–4.0). 1.0 = normal speed. Not supported for Chirp 3 HD voices.
float
Pitch adjustment in semitones (−20.0 to 20.0). 0.0 = default pitch. Not supported for Chirp 3 HD voices.
float
Volume gain in dB (−96.0 to 16.0). 0.0 = no change. Recommended not to exceed +10 to avoid clipping. Not supported for Chirp 3 HD voices.
array
Optimises audio for a specific playback device, applied on top of speaking_rate/pitch/volume (e.g., [“telephony-class-application”])
Azure STT/TTS Configuration
Basic Example:YAML
name: azure_voice_config
speech_to_text:
provider: azure_stt
azure_stt_config:
subscription_key: "your-azure-subscription-key"
region: "eastus"
text_to_speech:
provider: azure_tts
azure_tts_config:
subscription_key: "your-azure-subscription-key"
region: "eastus"
voice: "en-US-JennyNeural"
YAML
name: azure_voice_config
llm_aggregation_timeout_seconds: 0.8
speech_to_text:
provider: azure_stt
azure_stt_config:
subscription_key: "your-azure-subscription-key"
region: "eastus"
language: "en-US"
endpoint_id: null
profanity_filter: "masked"
phrase_list: ["watsonx Orchestrate", "IBM Cloud"]
text_to_speech:
provider: azure_tts
azure_tts_config:
subscription_key: "your-azure-subscription-key"
region: "eastus"
voice: "en-US-JennyNeural"
language: "en-US"
rate: 1.1
pitch: 5.0
volume: 90
style: "cheerful"
style_degree: 1.5
role: "YoungAdultFemale"
language: "en-US"
Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Azure Speech subscription key
string
required
Azure datacenter region (e.g., “eastus”, “westus”). Falls back to “eastus” at runtime if omitted.
string
BCP-47 language code (e.g., “en-US”, “fr-FR”). Default: “en-US”. Falls back to the top-level voice config
language field, then “en-US”.string
Custom Speech model endpoint ID. Use when pointing at a fine-tuned or on-premises model. Default: null.
string
Controls how profanity is handled in transcripts. Options:
"masked" | "removed" | "raw". Default: null (Azure SDK default is "masked").array
Domain-specific vocabulary hints fed to Azure’s PhraseListGrammar to boost recognition of specialized terms (up to 500 entries). Default: null.
integer
Milliseconds of silence after speech before Azure emits a final transcript (100–5000). Default: 800. Mirrors Watson’s
end_of_phrase_silence_time so one spoken turn produces one final event.Show detailed parameter descriptions
Show detailed parameter descriptions
string
required
Azure Speech subscription key
string
required
Azure datacenter region (e.g., “eastus”, “westus”). Falls back to “eastus” at runtime.
string
required
Azure Neural voice name (e.g., “en-US-JennyNeural”). Falls back to “en-US-JennyNeural” at runtime.
string
BCP-47 language code. If omitted, extracted from the voice name at runtime (e.g., “en-US-JennyNeural” → “en-US”). Default: null.
float
Speech rate multiplier mapped to an SSML prosody relative percent (0.5–2.0). Default: 1.0. 1.0 = normal speed; 0.5 = half speed; 2.0 = double speed.
float
Pitch as an SSML percentage offset (−20.0–20.0). Default: 0.0. Positive values increase pitch; negative values decrease pitch.
integer
Volume as an SSML absolute level (0–100). Default: 100. 100 = Azure’s neutral default; 0 = silent.
string
Speaking style name (e.g., “cheerful”, “sad”, “friendly”). The voice must support the requested style. Default: null.
float
Intensity of the speaking style (0.01–2.0). Default: null. 1.0 = normal intensity; 2.0 = maximum. Only used when
style is also set.string
Speaking role (e.g., “YoungAdultFemale”, “OlderAdultMale”). The voice must support the requested role. Default: null.
boolean
Strip markdown syntax from LLM output before sending text to Azure. Default: true.
Advanced Voice Settings
Voice configurations support advanced features for enhanced call handling:Voice Activity Detection (VAD)
Automatically detect when the user is speaking:YAML
vad:
enabled: true
provider: "silero_vad"
silero_vad_config:
confidence: 0.7
start_seconds: 0.2
stop_seconds: 0.8
min_volume: 0.6
DTMF Input (Dual-Tone Multi-Frequency)
Enable keypad input during voice calls:YAML
dtmf_input:
inter_digit_timeout_ms: 2500
termination_key: "#"
maximum_count: 10
ignore_speech: true
User Idle Handler
Handle situations when the user stops responding:YAML
user_idle_handler:
enabled: true
idle_timeout: 7
idle_max_reprompts: 2
idle_timeout_message: "Are you still there?"
idle_hangup_message: ""
Agent Idle Handler
Provide feedback when the agent is processing:YAML
agent_idle_handler:
typing_enabled: true
typing_duration_seconds: 5
audio_clip_id: "guitar_1"
hold_audio_seconds: 15
pre_hold_message: "Please wait a moment..."
hold_message: "Still processing your request..."
Complete Advanced Example
YAML
name: advanced_voice_config
speech_to_text:
provider: deepgram_stt
deepgram_stt_config:
api_url: "wss://api.deepgram.com/v1/listen"
api_key: "your_deepgram_api_key"
model: "nova-2"
language: "en-US"
numerals: true
keywords: ["urgent", "important", "priority"]
mip_opt_out: false
text_to_speech:
provider: deepgram_tts
deepgram_tts_config:
api_key: "your_deepgram_api_key"
api_url: null
language: "en"
model: "aura-2-thalia-en"
mip_opt_out: false
language: "en-US"
vad:
enabled: true
provider: "silero_vad"
silero_vad_config:
confidence: 0.7
start_seconds: 0.2
stop_seconds: 0.8
min_volume: 0.6
dtmf_input:
inter_digit_timeout_ms: 2500
termination_key: "#"
maximum_count: 10
ignore_speech: true
user_idle_handler:
enabled: true
idle_timeout: 7
idle_max_reprompts: 2
idle_timeout_message: "Are you still there?"
idle_hangup_message: ""
agent_idle_handler:
typing_enabled: true
typing_duration_seconds: 5
audio_clip_id: "guitar_1"
hold_audio_seconds: 15
pre_hold_message: "Please wait a moment..."
hold_message: "Still processing your request..."
Importing voice configurations
After creating your YAML file, import it using the following command:BASH
orchestrate voice-configs import --file <path-to-your-voice-file>
Show command flags
Show command flags
string
required
Path to the YAML file with the voice configuration.
Note:
The import command creates a new voice configuration or updates an existing one based on the name. If a voice configuration with the same name exists, the command updates it using the new configuration.
Listing voice configurations
To list all available voice configurations:BASH
orchestrate voice-configs list
Show command flags
Show command flags
Show full details of all voice configurations in JSON format.
Getting voice configuration details
To retrieve details of a specific voice configuration:BASH
orchestrate voice-configs get --name <config_name>
Show command flags
Show command flags
Exporting voice configurations
To export a voice configuration to a YAML file:BASH
orchestrate voice-configs export --name <config_name> --output <output_path>
Show command flags
Show command flags
Removing voice configurations
To remove a voice configuration by name or ID:BASH
orchestrate voice-configs remove --name <config_name>
# OR
orchestrate voice-configs remove --id <config_id>
Show command flags
Show command flags
Note:
If both
--id and --name are provided, the ID takes precedence.Important:
When you remove a voice configuration, agents using this configuration will no longer have voice capabilities. Make sure you want to remove the configuration before you proceed.