Repository navigation
Feat/gemini eval - #226
Merged
Merged
Feat/gemini eval#226
Conversation
(cherry picked from commit 6bb56c7)
Cherry-picked from fae5426 (rfan-debug/eva feat/gemini-eval). Adaptations for our current main: - Dropped the global TextJudgeMetric max_tokens 100000->65536 change (Vertex-specific workaround; kept our 100000 + service_tier=flex). - Made s2s endpoint optional (.get) so non-Vertex Gemini S2S configs without an explicit endpoint keep working. Co-authored-by: Ruixi Fan <[email protected]>
Cherry-picked from 8e982cc (rfan-debug/eva feat/gemini-eval). Adapted to our refactored system-prompt path (_build_system_prompt) and kept the optional-endpoint handling from the prior commit. Co-authored-by: Ruixi Fan <[email protected]>
…92fc81) Hand-ported from rfan-debug/eva feat/gemini-eval 392fc81, whose target audio_interface.py was renamed to audio_bridge.py (~500 commits diverged). Adapted for our current main: - Only the ORIGINAL half is ported: the trailing white-noise block that gives S2S assistant VADs (e.g. Gemini Live) an unambiguous end-of-speech. The commit's other half — fallback end-of-turn detection — is already covered on our side by the _user_turn_complete exact-boundary path. - Per review decision: the wire gets white noise, but the saved user_clean track records plain silence, so it stays aligned in time yet unpolluted for speech-fidelity / audio metrics. Co-authored-by: Ruixi Fan <[email protected]>
gemini-*-live-preview models exist only on Vertex AI (aiplatform.googleapis.com). The client factory checked api_key first, so a Developer key (AIza...) in s2s_params sent them to generativelanguage.googleapis.com/v1beta -> '1008 ... not found for API version v1beta'. - Prefer Vertex whenever a project is resolvable (GOOGLE_CLOUD_PROJECT / VERTEXAI_PROJECT / s2s_params['project']) or GOOGLE_GENAI_USE_VERTEXAI is set; ignore the Developer key in that mode. - Never use location='global' for Live (unsupported for bidiGenerateContent); fall back to a region with a warning. - Support optional s2s_params 'api_version' and 'endpoint' via HttpOptions. - Add routing unit tests. Co-authored-by: Ruixi Fan <[email protected]>
The trailing white-noise block only exists to nudge a slow assistant VAD into detecting end-of-speech. Once the assistant actually starts responding it has served its purpose, so cancel any remaining frames in _on_assistant_audio_start. Without this, noise kept streaming into the assistant's mic during its own turn (the drain is gated on user-inactive, not assistant-active), which normal silence never does — risking false barge-in on models with server-side interruption. This makes the trailing-noise behavior safe to keep enabled for all assistant models, not just Gemini.
raghavm243512
force-pushed
the
feat/gemini-eval
branch
from
September 15, 2026 18:16
2b14b51 to
c9ab074
Compare
raghavm243512
force-pushed
the
feat/gemini-eval
branch
from
September 15, 2026 18:33
7e4740d to
ae7e124
Compare
fanny-riols
reviewed
Sep 15, 2026
fanny-riols
reviewed
Sep 15, 2026
Co-authored-by: Fanny Riols <[email protected]>
Comment on lines
+251
to
+270
| # Thinking config: controls Gemini's internal reasoning budget. | ||
| # Accepts a dict with optional keys: | ||
| # "thinking_budget": int (0=disabled, -1=auto, or token count) | ||
| # "include_thoughts": bool | ||
| # "thinking_level": str ("MINIMAL", "LOW", "MEDIUM", "HIGH") | ||
| # Example: {"thinking_budget": 1024, "include_thoughts": false} | ||
| # If not set, defaults to ThinkingConfig() (model-dependent defaults). | ||
| thinking_raw = s2s_params.get("thinking_config", {}) | ||
| if isinstance(thinking_raw, dict) and thinking_raw: | ||
| tc_kwargs: dict[str, Any] = {} | ||
| if "thinking_budget" in thinking_raw: | ||
| tc_kwargs["thinking_budget"] = int(thinking_raw["thinking_budget"]) | ||
| if "include_thoughts" in thinking_raw: | ||
| tc_kwargs["include_thoughts"] = bool(thinking_raw["include_thoughts"]) | ||
| if "thinking_level" in thinking_raw: | ||
| tc_kwargs["thinking_level"] = thinking_raw["thinking_level"] | ||
| self._thinking_config = types.ThinkingConfig(**tc_kwargs) | ||
| logger.info(f"Thinking config: {tc_kwargs}") | ||
| else: | ||
| self._thinking_config = types.ThinkingConfig() |
Collaborator
There was a problem hiding this comment.
i think thinking_budget is deprecated so we should remove that option.
also, if include_thoughts is True, I don't think we are saving/capturing the thoughts anywhere. Could you check if can do that?
Collaborator
Author
There was a problem hiding this comment.
Removed thinking_budget support (kept include_thoughts + thinking_level). Also wired up capturing thoughts: when include_thoughts is set, parts with part.thought=True are now logged via a new FrameworkLogWriter.thought() call (framework_logs.jsonl, type="thought") instead of being silently dropped.
| description=f"{tool.name}: {tool.description}", | ||
| parameters=params_schema, | ||
| behavior=types.Behavior.BLOCKING, | ||
| # behavior=types.Behavior.BLOCKING, |
Collaborator
There was a problem hiding this comment.
the blocking behaviour is supported by gemini 3.8 live but not 3.8 live extended thinking, so can we either enable for the or remove this line if it is not needed anymore
Collaborator
Author
There was a problem hiding this comment.
Checked — Behavior.BLOCKING/NON_BLOCKING is explicitly documented as unsupported in Vertex AI in the google-genai SDK, and this server always creates its client with vertexai=True (see _create_genai_client, Vertex-only Live-preview models require it). Since Vertex is our only path here, I removed the line rather than conditionally enabling it, and left a comment explaining why.
katstankiewicz
approved these changes
Sep 22, 2026
katstankiewicz
left a comment
katstankiewicz
left a comment
Collaborator
There was a problem hiding this comment.
thank you for addressing all of the comments!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring over changes from google for new Gemini Live model. Original branch was conflicting and pretty old, so this has all the changes integrated
Some changes related to tool scheduling for new gemini live models, a couple additional diagnostic metrics, and a white noise addition that I disabled since we don't have global testing with this change (but is easy to turn on).
I also updated the Dockerfile and this small reorder should have better reuse of built content, so smaller updates when we make changes