What an AI Feature Costs After Launch

Shipping an AI feature is the easy part. Keeping it working, accurate, and cost-efficient over the following 12–24 months is where most teams get surprised. Here is a breakdown of what that actually costs and why.

The Costs Nobody Puts in the Initial Estimate

Most engineering estimates for AI features cover model API calls, GPU compute for fine-tuning (if any), and some developer time. That covers roughly the first sprint. What it misses is everything that kicks in once real users hit the system.

The main ongoing cost buckets are:

• Inference compute — every prediction or generation costs money, and volume compounds
• Monitoring and observability — you need to know when the model drifts or fails silently
• Data pipeline maintenance — the feeds that keep your model current do not run themselves
• Human-in-the-loop review — some percentage of outputs will always need a human check
• Retraining or prompt revision cycles — models go stale; the world changes
• Compliance and audit logging — especially relevant in fintech and healthtech

These are not optional line items you can defer. Skipping monitoring means you find out about model degradation from a customer complaint rather than an alert.

What Does Inference Actually Cost at Scale?

This depends heavily on which model you are calling and how you are calling it.

For GPT-4o as of mid-2025, OpenAI charges approximately $2.50 per million input tokens and $10.00 per million output tokens. A feature that processes 50,000 user queries per month, each averaging 800 input tokens and 400 output tokens, costs roughly $280 per month in pure API fees at low volume. Scale that to 500,000 queries and you are at $2,800/month before you have paid for anything else.

Switch to a self-hosted open-weight model like Llama 3.1 70B on a single A100 80 GB GPU and you are looking at roughly $2.50–$3.50/hour on most cloud providers, or around $1,800–$2,500/month for a dedicated instance. At the 500,000-query volume, that starts to look cheaper per query, but only if your utilisation is high enough to justify the reserved capacity. Below about 60–70% utilisation, the API route wins on cost.

The hidden multiplier: retries and context window bloat

Most applications do not send clean, minimal prompts. They send conversation history, retrieved documents from a RAG pipeline, system instructions, and structured output schemas. A prompt that is 800 tokens in your test environment is often 3,000–4,000 tokens in production once you add retrieval context. That 4x multiplier is not uncommon and it is rarely in the initial estimate.

Retries add further cost. If you are calling a model with a fallback strategy (e.g., retry on timeout or low-confidence output), you may be paying for 1.2–1.4x the calls you think you are making.

How Much Does Monitoring an AI Feature Actually Cost?

More than most teams expect, and less than the cost of not doing it.

A minimal observability setup for an AI feature includes:

• Logging inputs and outputs for a sample of traffic (LangSmith, Weights & Biases, or a custom pipeline into BigQuery or Snowflake)
• Latency and error rate tracking (standard APM tools like Datadog or Grafana handle this)
• Output quality metrics — this is the hard part

Output quality is where AI monitoring diverges from standard software monitoring. You cannot just check for HTTP 200. You need to track whether the outputs are actually good. That means either an LLM-as-judge pattern (which adds another API call per evaluation, roughly $0.01–$0.05 per evaluated output), a human review queue, or both.

A realistic monitoring cost for a medium-traffic AI feature is $300–$800/month in tooling, plus 4–8 hours of engineer time per month to act on what the monitoring surfaces. That figure goes up if you are in a regulated industry that requires immutable audit logs.

When Does the Model Go Stale, and What Does Fixing It Cost?

This varies by domain. A model answering questions about a static knowledge base may stay accurate for 12–18 months. A model summarising news, processing financial data, or working in a domain where regulations change (tax rules, compliance requirements, medical guidelines) may need meaningful revision every 3–6 months.

"Revision" can mean several things:

Type	Trigger	Approximate cost
Prompt engineering update	Output quality degrades on new input patterns	8–20 engineer hours
RAG index refresh	Source documents are outdated	Compute cost of re-embedding + storage; often $50–$400/refresh
Fine-tune on new data	Domain shift significant enough that prompting alone cannot fix it	$500–$5,000+ depending on model size and data volume
Model swap (e.g., GPT-4 to GPT-4o or a newer Llama)	Provider deprecates version or better model available	20–60 engineer hours for regression testing and integration

The fine-tune cost is the one that surprises teams most. Fine-tuning GPT-4o mini via the OpenAI API costs $25 per million training tokens. A modest fine-tuning job on 10 million tokens is $250 in compute, but the data preparation, evaluation, and deployment work around it typically runs to 40–80 hours of engineering time.

Data pipeline costs are often the largest hidden expense

If your AI feature depends on fresh data (most do), someone has to maintain the ingestion pipeline. That means handling schema changes from upstream APIs, dealing with data quality issues, re-embedding documents when your chunking strategy changes, and managing vector store costs (Pinecone, Weaviate, or pgvector on Postgres each have their own cost profile at scale).

A vector store holding 10 million 1536-dimension embeddings on Pinecone's serverless tier costs roughly $0.033 per million vectors queried. At 500,000 queries/month each querying 20 vectors, that is $330/month in retrieval alone, before you count storage.

What Does a Realistic Monthly Budget Look Like?

The table below is based on a mid-scale B2B SaaS feature handling around 200,000 AI-assisted actions per month, using a hosted LLM API (not self-hosted), with a basic RAG pipeline and standard monitoring.

Cost category	Monthly estimate
LLM API calls (inference)	$600–$1,200
Vector store (storage + queries)	$150–$350
Monitoring tooling	$200–$500
Human review (4 hrs/month @ £80/hr)	$400
Engineering maintenance (6 hrs/month)	$600–$900
Total	$1,950–$3,350

This is before retraining cycles or model swaps, which are infrequent but cost $2,000–$10,000 when they happen. Annualised, plan for $25,000–$50,000 for a feature at this scale, including one retraining or integration cycle per year.

Conclusion

The build cost of an AI feature is typically 20–30% of the total two-year cost. The rest is inference, maintenance, monitoring, and the periodic work of keeping the model accurate as the world changes.

If you are scoping an AI feature now, build a two-year cost model before you commit the first sprint. Identify the retraining trigger (what metric tells you the model needs work), decide early between hosted API and self-hosted inference based on your volume projections, and make sure your monitoring covers output quality rather than just uptime.

If you want help building that cost model or reviewing your current AI feature's operational architecture, the team at Sodio is happy to look at it with you.

FAQ

What is the biggest unexpected cost of running an AI feature in production? Usually the data pipeline. Teams budget for inference but underestimate the engineering time required to keep retrieval indexes fresh, handle upstream schema changes, and re-embed documents when chunking strategies evolve. This work is ongoing, not a one-time setup, and it compounds as your data volume grows.

How often do AI models need to be retrained or updated? In stable domains, a well-promped RAG-based feature can stay accurate for 12–18 months with only minor prompt tweaks. In fast-moving domains like finance, legal, or anything news-adjacent, meaningful revision is typically needed every 3–6 months. The trigger should be a measurable output quality metric, not a calendar date.

Is it cheaper to use a hosted API like OpenAI or to self-host an open-weight model? At low to medium volume (under roughly 300,000–500,000 queries/month), hosted APIs are almost always cheaper when you factor in GPU instance costs, maintenance, and engineering overhead. Above that threshold, self-hosting a model like Llama 3.1 70B starts to make economic sense, provided your team can manage the infrastructure.

What monitoring does an AI feature actually need? At minimum: latency, error rates, and output quality sampling. Output quality is the hard part. An LLM-as-judge pattern (using a cheaper model to evaluate outputs automatically) covers most cases but adds cost per evaluated call. Human review queues are necessary for high-stakes outputs. Standard APM tools handle the infrastructure side; you need something separate for semantic quality.

How should I budget for an AI feature over two years? A useful rule of thumb: multiply your initial build cost by 3–4 to get a two-year total cost of ownership. The largest variables are query volume growth, retraining frequency, and whether you stay on a hosted API or migrate to self-hosted infrastructure. Build your two-year model before committing to architecture decisions that are expensive to reverse.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.