Skip to content

feat: generic audio types and voice listing in place of ElevenLabs tools - #438

Merged
ProfSynapse merged 1 commit into
mainfrom
feat/generate-audio-types
Nov 5, 2026
Merged

ProfSynapse merged 1 commit into
mainfrom
feat/generate-audio-types

Conversation

@ProfSynapse

@ProfSynapse ProfSynapse commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #437, so this should merge after it. The diff here is only this layer.

Summary

One audio tool, one type. generateAudio takes type: speech | music | sfx and adapts it to the model:

ElevenLabs Lyria 3.5 / Pro Lyria Clip
durationSeconds Real parameter, usually close Written into the prompt, can be far off Always 30s, and the result says so
instrumental Real parameter Written into the prompt Written into the prompt
loop (SFX only) Supported Not available Not available
  • Music comes from ElevenLabs, Google (lyria-3.5, lyria-3-clip-preview) and OpenRouter (Lyria Pro and Clip). Sound effects are ElevenLabs only.
  • The schema follows the configured keys. With only OpenAI, an agent sees no type option or music wording.
  • Leaving type out means speech, and JSON callers sending mode: 'voice' still get speech. The CLI's --mode flag is gone.
  • Songs with words return lyrics, with Lyria's structural markers removed.
  • A wrong file extension, a too-long sound effect or sound effects on Google fail before any request.

One voice list. New listVoices replaces the ElevenLabs one.

  • It reads VoiceCatalogService, which also fills the Voice settings picker, and gets its default from SpeechSynthesisService.resolveRequest, which generateAudio also uses. The default it reports is the one generation will use.
  • generateAudio's voice description now states the current default and points to listVoices, instead of listing voices per model.
  • Provider refusals are reported rather than appearing as an empty list. The settings picker still falls back to the catalog.

Removed. The ElevenLabs soundEffects, generateMusic and listVoices tools. The app keeps its key and validation, and adds itself as a provider to the shared tools. This is a breaking change for anything calling those tools by name: it gets Unknown tool, which points it to getTools.

Testing

  • Jest: 7,126 passed, 0 failed, 118 skipped. Lint, mobile check, npm run build and schemas:check pass.
  • Guards for the music and SFX path were mutation-checked. listVoices and the tool removal were not.
  • In a test vault in the running plugin, through real requests:
    • Music and SFX: ElevenLabs music and SFX, Lyria 3.5, Lyria Clip with lyrics, and Clip through OpenRouter.
    • Voices: listVoices on the settings default, OpenAI, Google, ElevenLabs (70 voices from a real account) and OpenRouter.
    • Refusals: a provider with no key and an unknown model.
    • nexus CLI: discovery, listing, generating a sound effect, and the removed tool failing to resolve.
  • Lyria Pro through OpenRouter worked through the shipped tool outside the app.
  • dev:errors reported nothing on each run, but console capture was only enabled on the first app session, so this is weaker evidence on later runs. npm run verify:obsidian was not run.

Not tested live:

  • The settings UI: the Voice settings picker and the Apps tab were never opened. Keys were injected in memory, and tools were called directly or through the CLI.
  • A real chat turn generating audio.
  • Music and SFX on the final build: they last ran in the app before the ElevenLabs tools were removed, with no change to that code since.
  • An ElevenLabs install saved across a restart.
  • A real scope failure through listVoices, because the key has that scope. The test uses ElevenLabs' real response wording.
  • Mistral voices, because no key is available.
  • Removing the ElevenLabs tools in a vault that already had them installed and saved.

Worth checking in review

  • Lengths aren't exact. ElevenLabs music asked for 4s made 6s, while 3, 8 and 15s landed. Lyria 3.5 asked for 25s made 61s. The schema says so.
  • Lyria Pro through OpenRouter once returned a 200 with no audio, then worked on six later tries. It isn't retried, because that could bill twice, and the error says to try again.
  • No settings default for music or SFX. They use the first configured model in catalog order.
  • The ElevenLabs agent now has no tools. It's kept for its key and validation, and this has not been a shape apps had before.

🤖 Generated with Claude Code

Base automatically changed from fix/provider-scope-errors-openrouter-audio to main November 5, 2026 14:23
generateAudio takes one type, speech, music or sfx, and adapts it per model.
ElevenLabs takes real parameters, Lyria reads requests from the prompt, Lyria
Clip is always 30 seconds, and OpenRouter only sends audio over a stream. The
schema follows the configured keys.

Add a generic listVoices tool, backed by the catalog that fills the Voice
settings picker and the resolver generateAudio uses, so all three agree on a
model's voices and its default.

Remove the ElevenLabs soundEffects, generateMusic and listVoices tools, which
these replace. The ElevenLabs app keeps its key and adds itself as a provider.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@ProfSynapse
ProfSynapse force-pushed the feat/generate-audio-types branch from 20ec668 to d304506 Compare November 5, 2026 14:23
@ProfSynapse
ProfSynapse merged commit 5275894 into main Nov 5, 2026
1 check passed
@ProfSynapse
ProfSynapse deleted the feat/generate-audio-types branch November 5, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant