How to evaluate a new model API before you move real work to it

A lab opens an API, prices it under the incumbents, and for a week your feed fills with screenshots of it doing something clever. Meta did this on July 9, 2026, launching a public preview of the Meta Model API with Muse Spark 1.1 and a context window of 1 million tokens [4]. It will happen again within a few months, from a different lab, with a different name, and the question it puts in front of you does not change.

That question is not whether the model is good. It is whether the model is good at your work, at your volumes, at a price and under a policy that will still be there once you have built on it. Answering it takes about two weeks and a small amount of money, and most of the work is reading rather than coding. This guide is the procedure. It is written for someone who calls a model API directly, or points a coding agent at one, and pays the bill personally. If you only use chat subscriptions such as Claude, ChatGPT or Gemini, switching costs you an afternoon and this is more machinery than you need. If you have a procurement process and a security questionnaire, this is too small for you.

The launch price has a date on it

Read the price table as a snapshot, not a rate card. Meta lists Muse Spark on its Standard tier at $1.25 per million input tokens, $4.25 per million output tokens, and $0.15 per million cached input tokens, with no minimum and no upfront commitment [2]. That is cheap against the field, and it is also the number the provider chose for the launch. Google publishes Gemini 3.8 Flash at $0.75 per million input tokens and $3.75 per million output tokens and marks those rates as running through December 31, 2026 [8]. The expiry date is printed on the pricing page, in public, and almost nobody reads that far.

So do the budget twice. Once at today’s price, and once at double it. Doubling is not a pessimistic multiplier here, it is the published successor rate: the same page lists Gemini 3.8 Flash at $1.50 per million input tokens and $7.50 per million output tokens from January 1, 2027 [8]. If the second number kills the project, you are not evaluating a tool, you are speculating on a promotion.

Then find the prices that are not in the headline table, because those are where a working system actually spends money. On the Meta API, web search grounding costs $2.50 per 1,000 search queries on top of the token cost of the request [2]. Image generation is billed at $0.01 per generated image, and speech to text at $0.18 per hour of audio processed [2]. There is no long-context premium, so filling the 1,048,576-token context window [1] costs the same rate per token as leaving it nearly empty [2]. That is worth knowing, because other providers charge more above a threshold: Google prices Gemini 3.1 Pro Preview at $2.00 per million input tokens for prompts up to 200,000 tokens and $4.00 above that [8].

Also check what the provider gives back for work that is not urgent. Google lists batch and flex tiers at 50% off standard pricing [8], and a job you can run overnight belongs there. A provider with no batch tier is not disqualified, but it is more expensive than its headline suggests for anything you were going to queue.

The cheapest tier is usually a data deal

When a price looks impossible, find out what is being bought with the discount. Meta runs two tiers of the same model. On Standard, your prompts and completions are not used to train Meta models [2]. On the Contributor tier, the same model is $0.10 per million input tokens and $0.20 per million output tokens, roughly a twelfth of the Standard input price and a twentieth of the output price, in exchange for permission to use your prompts and completions to train future Meta models [2]. Meta describes it as lowering the barrier to entry for prototyping, testing integrations and scaling experiments where training on your data is acceptable [2].

That is a clear and honestly labelled trade, which makes it a good example to reason with. If the material you send is your own notes, public documents or throwaway test prompts, the trade is fine and the saving is real. If it is client work under an NDA, a candidate’s CV, patient information or anything you promised someone else you would not spread, the tier is not available to you at any price, and no amount of saved money changes that. Write down which of your workloads can sit on a training-eligible tier before you look at the discount, not after.

The discount usually buys the provider something else as well. Meta’s Standard tier allows 3,000 requests per minute and 4,000,000 tokens per minute, while the Contributor tier allows 100 requests per minute against 3,000,000 tokens per minute [2]. So the cheap tier is also a throughput cap, and a batch job that fans out to hundreds of small parallel requests will hit a wall on it that never appears in a demo. One more line worth reading in any provider’s limits page: Meta applies limits per team rather than per API key, so extra keys do not buy extra quota [2].

Drop-in compatible describes the base URL, not the behaviour

New providers make migration sound free, and the claim is usually half true. Meta says the Model API is drop-in compatible with the OpenAI SDK, the Anthropic SDK and OpenAI-compatible agent CLIs, so you set the base URL to https://api.meta.ai/v1, supply a key and keep the rest of your code [1]. That is genuinely useful. It is also a statement about wire format, not about what the model does with your request.

The most honest description of what compatibility layers cost comes from a vendor documenting its own. Anthropic states that its OpenAI SDK compatibility layer is “primarily intended to test and compare model capabilities, and is not considered a long-term or production-ready solution for most use cases” [5]. The specifics are the useful part. The strict parameter on function calling is ignored, so tool use JSON is not guaranteed to follow the schema you supplied. Audio input is ignored and stripped. Prompt caching is not supported through the layer. Fields including response_format, logprobs, seed, logit_bias and reasoning_effort are ignored [5]. None of that returns an error. Your code keeps running and quietly does something slightly different.

So spend the first hour of a trial on the four things a compatibility layer breaks silently: structured output against a real schema, a tool call with arguments you validate, streaming under a slow connection, and caching if your prompts share a long prefix. Check modality support against the exact model version too, rather than the family. Meta notes that audio understanding in Muse Spark 1.3 is not fully supported and that responses including audio may be degraded, and points you at 1.2 or its dedicated transcription model instead [3]. The newest version of a model is not automatically the best one for the thing you do.

Test on work you have already finished

The only evaluation that answers your question uses work you have already done and already judged. Pull 20 to 40 real tasks from the last month, the ones where you know what a good answer looks like because you shipped it. Run them through the new provider with your existing prompts unchanged, then grade them against the output you accepted at the time. Do not write a fresh prompt for the new model in this pass; you are measuring the model, not your enthusiasm.

Score two things per task and nothing else. Whether the output was usable without rework, and what the call cost end to end. Cost per finished task is the number that decides this, and it is not proportional to the advertised input price. Output tokens are priced several times higher than input on both providers quoted here, $4.25 against $1.25 on Meta’s Standard tier [2] and $3.75 against $0.75 on Gemini 3.8 Flash [8], so a model that reasons at length can be dearer per task while looking cheaper per million tokens. Reasoning settings compound this, and they are not always available everywhere: Meta offers reasoning effort levels including a max level for extended reasoning, on the Standard tier only [3]. Tool use compounds it again, since every search grounding call adds to the bill separately [2].

Read the retirement policy before the pricing page

The model you evaluate is not the model you will be running in a year, and the notice you get before it disappears is a published policy you can check in two minutes. Anthropic notifies customers with active deployments and gives at least 60 days’ notice before retiring a publicly released model, after which requests to a retired model fail [6]. OpenAI commits to at least 6 months for generally available models and at least 3 months for specialised variants, and states that preview models may be retired with much shorter notice, such as 2 weeks [7]. That last clause is the one to hold on to, because a new provider’s first offering is usually a preview.

A young API also moves faster than a mature one. Meta launched the Model API with Muse Spark 1.1 on July 9, 2026 [4]; two months later the docs list three versions, 1.1, 1.2 and 1.3, with 1.3 recommended for new work and only 1.3 and 1.2 offered on the discounted tier [3]. That pace is a good sign about the lab and a cost to you, because every version change is a re-run of your evaluation set.

Two habits absorb most of it. Pin the exact dated model ID in your code rather than an alias that silently moves under you, so a change is something you choose. And keep the evaluation set from the previous section in a file you can re-run in an afternoon, because that file is the thing that turns a migration from a week of anxiety into a morning of grading.

Keep the exit cheap while the tokens are cheap

The first month is when lock-in gets installed, usually by accident. Keep prompts and evaluation cases in text files in your own repository rather than in a provider’s playground. Keep the model name, the base URL and the key in configuration, so switching provider is an edit to one file and not a search across a codebase. If you use a framework or an agent CLI that already speaks several providers, that is worth more during an evaluation than any single feature.

Then be deliberate about provider-specific features. Prompt caching, server-side search tools, reasoning continuity across turns and per-provider structured output all improve results and all cost you something to unpick later. Use them where they earn their keep, and note in a comment that they are provider-specific. The distinction that matters is between features that make your code better and features that make your code unmovable.

It also helps to know the floor you can fall back to. Meta publishes Muse Glimmer, an open-weight multimodal model distilled from Muse Spark, under a permissive Apache 2.0 licence, to be downloaded and served on your own hardware through runtimes such as vLLM, SGLang, llama.cpp or ExecuTorch [1]. An open-weight option in the same family is not a substitute for a frontier hosted model, and running it yourself is real work. But its existence changes the shape of the risk you are taking, and a provider that offers nothing similar is a provider whose price rises you can only accept or leave.

What still goes wrong

Every number here is a snapshot taken on the date in the header and cited to the page it came from, because these pages change without ceremony. Tiers get renamed, promotional rates expire, rate limits are revised, and a model that is recommended for new work today is a legacy version next quarter. Check the source link before you commit money, and treat any price quoted in an article, including this one, as a starting point.

A 30-task evaluation is also a blunt instrument. It will tell you whether a model is broadly competent at your work and roughly what it costs. It will not surface the failure that happens once in 500 calls, and that is usually the one that hurts, because it arrives in front of a client rather than in front of you. Nor does two weeks of testing tell you anything reliable about latency under load, error rates at three in the morning, or how the provider behaves during an outage. Those take months, and a new API has not existed for months. Where a mistake would be expensive, run the new provider in parallel on real traffic for a while and keep the old one deciding, which costs you double tokens on a subset of requests and buys the only evidence that counts.

The last thing that goes wrong is subtler. Switching provider costs attention, and attention is the resource you actually have least of. A saving of $30 a month is not worth a week of your time, no matter how good the arithmetic looks in isolation. Run this procedure when the new option is either substantially better at something you do often or substantially cheaper at volume you actually have. Otherwise, note the launch, keep the tab open, and go back to work.