fix(modelsdev): skip models.dev fetch for custom providers (#3165)#3169
Conversation
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟢 APPROVE
The fix is well-scoped, correct, and thoroughly tested.
What the PR does: Adds a WithKnownProvider predicate to modelsdev.Store so that custom (non-catalog) providers are served from cache only and never trigger an outbound GET models.dev/api.json. The lazyModelStore and RuntimeConfig.ModelsDevStore() are both wired with provider.IsKnownProvider.
Correctness review:
- The mutex protocol is sound:
getDatabaseis called while holdings.muonly from withinGetDatabase/getProvider, both of which acquire the lock before calling intogetDatabase. No re-entrant locking issue. - The
allowFetch=falsepath deliberately skips memoization ofs.dbto avoid poisoning future fetch-eligible lookups for known providers — this is correct and clearly commented. NewDatabaseStoreleavesknownProvidernil (fetch-eligible for all providers), which is intentional: it is used in tests with an in-memory database, not a live fetch path.- The
loadDatabasestale-cache logic change (!allowFetch || time.Since(...) < refreshInterval) correctly short-circuits on the firsterr == nilcheck when fetching is disallowed, so the secondaryif !allowFetchblock only handles the cache-miss case — logically consistent. GetDatabase(public API) always passesallowFetch=true, preserving backward compatibility for direct callers (e.g., TUI model picker,modelsCLI) that explicitly want the full catalog.WithKnownProvider(nil)storesnil, which is handled correctly:s.knownProvider == nil→allowFetch = true(same as no predicate).
Tests: TestKnownProviderGatesFetch, TestNoPredicateAlwaysAllowsFetch, and TestLazyModelStore_GatesCustomProvider together cover the regression, backward compatibility, and production store path. The use of an already-expired context deadline as a stand-in for "network unreachable" is a clean, deterministic approach.
No issues found in the changed code.
Resolving a model for a user-defined custom provider (a name that is not a built-in or alias, e.g. mistral_gateway) loaded the full models.dev catalog, which performs a blocking 30s GET to https://models.dev/api.json. In internet-restricted environments this hung the request loop (loop.go) and session compaction on every turn, making a self-contained custom-provider config (base_url + token_key) unusable. modelsdev.Store gains WithKnownProvider: a provider the predicate rejects is resolved from the cache only and never triggers an outbound fetch (a missing cache yields an empty catalog, so the lookup returns a clean "provider not found"). The cache-only snapshot is memoized separately from the authoritative catalog, keeping the per-turn hot path cheap (no repeated catalog re-parse) without ever satisfying a fetch-eligible lookup for a known provider. The runtime's default store (lazyModelStore) and RuntimeConfig.ModelsDevStore() are wired with provider.IsKnownProvider, so only catalog providers (built-ins and aliases) can reach the network; custom providers resolve locally and the agent stays usable air-gapped.
3e1abd1 to
7acef6c
Compare
Problem
A custom provider — a name that is not a built-in/alias (e.g.
mistral_gateway) defined withbase_url+token_key— triggered a blockingGET https://models.dev/api.json(full ~3.4 MB catalog) during model resolution. In internet-restricted environments this hangs ~30s on every turn, making a self-contained custom-provider config unusable. Renaming to a known provider (e.g.openai) bypassed it.Closes #3165.
Fix
modelsdev.StoregainsWithKnownProvider: a provider the predicate rejects is served cache-only and never fetches (missing cache → empty catalog → clean "provider not found"). The runtime's default store (lazyModelStore) andRuntimeConfig.ModelsDevStore()are wired withprovider.IsKnownProvider, so only catalog providers (built-ins + aliases) can reach the network.loop.goGetModel(every turn)session_compaction.goresolveContextLimitVerification
Real binary, cold cache, custom-provider config from the issue:
origin/mainTestLazyModelStore_GatesCustomProvider(runtime) reproduces the exact issue error without the fix, passes with it.modelsdevgate unit tests;go build/vet/golangci-lint/-racegreen.Known limitations (follow-ups, out of scope)
modelsCLI still fetch the full catalog (not the request-loop path).provider_opts.context_sizehas no context limit, so compaction stays disabled for it (inference unaffected — it routes viabase_url).