Skip to content

fix(modelsdev): skip models.dev fetch for custom providers (#3165)#3169

Merged
Sayt-0 merged 1 commit into
mainfrom
fix/3165-custom-provider-models-dev
Jun 19, 2026
Merged

fix(modelsdev): skip models.dev fetch for custom providers (#3165)#3169
Sayt-0 merged 1 commit into
mainfrom
fix/3165-custom-provider-models-dev

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Jun 19, 2026

Copy link
Copy Markdown
Member

Problem

A custom provider — a name that is not a built-in/alias (e.g. mistral_gateway) defined with base_url + token_key — triggered a blocking GET https://models.dev/api.json (full ~3.4 MB catalog) during model resolution. In internet-restricted environments this hangs ~30s on every turn, making a self-contained custom-provider config unusable. Renaming to a known provider (e.g. openai) bypassed it.

Closes #3165.

Fix

modelsdev.Store gains WithKnownProvider: a provider the predicate rejects is served cache-only and never fetches (missing cache → empty catalog → clean "provider not found"). The runtime's default store (lazyModelStore) and RuntimeConfig.ModelsDevStore() are wired with provider.IsKnownProvider, so only catalog providers (built-ins + aliases) can reach the network.

Path Before After
loop.go GetModel (every turn) fetch catalog for any provider custom → no fetch
session_compaction.go resolveContextLimit ~30s hang air-gapped resolves instantly
known provider (openai/anthropic/bedrock…) fetch unchanged

Verification

Real binary, cold cache, custom-provider config from the issue:

Binary models.dev catalog fetched
origin/main yes (~3.4 MB download)
this PR no
  • TestLazyModelStore_GatesCustomProvider (runtime) reproduces the exact issue error without the fix, passes with it.
  • modelsdev gate unit tests; go build / vet / golangci-lint / -race green.

Known limitations (follow-ups, out of scope)

  • TUI model picker / models CLI still fetch the full catalog (not the request-loop path).
  • A custom provider without provider_opts.context_size has no context limit, so compaction stays disabled for it (inference unaffected — it routes via base_url).

@Sayt-0 Sayt-0 requested a review from a team as a code owner June 19, 2026 09:06

@docker-agent docker-agent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟢 APPROVE

The fix is well-scoped, correct, and thoroughly tested.

What the PR does: Adds a WithKnownProvider predicate to modelsdev.Store so that custom (non-catalog) providers are served from cache only and never trigger an outbound GET models.dev/api.json. The lazyModelStore and RuntimeConfig.ModelsDevStore() are both wired with provider.IsKnownProvider.

Correctness review:

  • The mutex protocol is sound: getDatabase is called while holding s.mu only from within GetDatabase/getProvider, both of which acquire the lock before calling into getDatabase. No re-entrant locking issue.
  • The allowFetch=false path deliberately skips memoization of s.db to avoid poisoning future fetch-eligible lookups for known providers — this is correct and clearly commented.
  • NewDatabaseStore leaves knownProvider nil (fetch-eligible for all providers), which is intentional: it is used in tests with an in-memory database, not a live fetch path.
  • The loadDatabase stale-cache logic change (!allowFetch || time.Since(...) < refreshInterval) correctly short-circuits on the first err == nil check when fetching is disallowed, so the secondary if !allowFetch block only handles the cache-miss case — logically consistent.
  • GetDatabase (public API) always passes allowFetch=true, preserving backward compatibility for direct callers (e.g., TUI model picker, models CLI) that explicitly want the full catalog.
  • WithKnownProvider(nil) stores nil, which is handled correctly: s.knownProvider == nilallowFetch = true (same as no predicate).

Tests: TestKnownProviderGatesFetch, TestNoPredicateAlwaysAllowsFetch, and TestLazyModelStore_GatesCustomProvider together cover the regression, backward compatibility, and production store path. The use of an already-expired context deadline as a stand-in for "network unreachable" is a clean, deterministic approach.

No issues found in the changed code.

Resolving a model for a user-defined custom provider (a name that is not a
built-in or alias, e.g. mistral_gateway) loaded the full models.dev catalog,
which performs a blocking 30s GET to https://models.dev/api.json. In
internet-restricted environments this hung the request loop (loop.go) and
session compaction on every turn, making a self-contained custom-provider
config (base_url + token_key) unusable.

modelsdev.Store gains WithKnownProvider: a provider the predicate rejects is
resolved from the cache only and never triggers an outbound fetch (a missing
cache yields an empty catalog, so the lookup returns a clean "provider not
found"). The cache-only snapshot is memoized separately from the authoritative
catalog, keeping the per-turn hot path cheap (no repeated catalog re-parse)
without ever satisfying a fetch-eligible lookup for a known provider.

The runtime's default store (lazyModelStore) and RuntimeConfig.ModelsDevStore()
are wired with provider.IsKnownProvider, so only catalog providers (built-ins
and aliases) can reach the network; custom providers resolve locally and the
agent stays usable air-gapped.
@Sayt-0 Sayt-0 force-pushed the fix/3165-custom-provider-models-dev branch from 3e1abd1 to 7acef6c Compare June 19, 2026 09:18
@Sayt-0 Sayt-0 merged commit cce107c into main Jun 19, 2026
9 checks passed
@Sayt-0 Sayt-0 deleted the fix/3165-custom-provider-models-dev branch June 19, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom provider names trigger unexpected outbound lookup to models.dev, causing failures in internet-restricted environments

3 participants