The Six Skills Behind Production-Ready AI Applications

Building an AI demo has never been easier. A few lines of code can call a model, generate an answer and create the impression that the hard work is finished. Production tells a different story. Real users ask unexpected questions. Source documents change. Tools fail. Latency rises. Costs drift. A model produces an answer that is fluent, plausible and wrong.

This is why skilled AI application developers need more than prompt-writing ability. A useful map of the field, informed by patterns across job postings, structured expert interviews and practitioner surveys, contains six connected areas:

1. LLM foundations
2. Grounding models with data
3. Building agentic systems
4. Evaluation-driven development
5. Operating in production
6. Machine learning foundations

The value is not in mastering these as six isolated subjects. It is in understanding how they work together when an uncertain AI component becomes part of a dependable software system.

Why AI application development is different

Traditional software is usually deterministic. Give a function the same valid input and you expect the same output. Tests can compare an actual value with an exact expected value. Plans can be decomposed into features with relatively stable acceptance criteria.

AI components change that contract. You cannot know every sentence an LLM will generate or every prediction a model will make. Even a small change to a prompt, context document, model version or sampling setting can alter the result. The system may pass ten examples and fail the eleventh in a way nobody anticipated.

That uncertainty makes AI engineering more iterative:

1. Build a small, observable version of the behaviour.
2. Inspect traces, outputs, errors, latency and cost.
3. Decide which failure matters most and what experiment could address it.
4. Improve one part of the system, then run the evaluation again.

The critical skill is deciding what to try next. Reliable AI software emerges from a disciplined sequence of evidence-based decisions, not from expecting an unreliable component to become predictable through a longer prompt.

The three-stage delivery loop

The iterative mindset becomes practical when the work is organised into three connected stages. These stages are not a one-way waterfall. Evidence from execution or production often sends the team back to revise the plan.

Planning

Planning begins with brainstorming and discovery. The team researches the domain, runs small experiments, studies the users and examines the existing codebase and data before committing to a design. This is where assumptions should be made visible rather than hidden inside a prompt.

The discovery work then becomes a specification that captures the user requirements, acceptance criteria, technical design, data flows, permissions and architecture. From that specification, the team creates an execution plan with small, verifiable checkpoints. A separate plan review is valuable: challenge the security model, identify unnecessary complexity, test whether the proposed tools have enough context and permission, and remove features that do not serve the core use case.

Execution

Execution is the build-test-verify cycle. The AI agent can read the codebase, implement changes, run commands and repair failures, but its level of autonomy should match the risk. A reversible prototype can allow broad exploration. A change involving personal data, payments or external communication needs narrower permissions and stronger human gates.

Verification combines automated and human checks. Automated checks can cover schemas, unit tests, integration tests, security rules and evaluation datasets. Human review remains important for intent, usability, domain correctness and high-impact decisions. The goal is neither maximum autonomy nor constant manual control; it is a calibrated division of work in which the agent moves quickly and the human retains judgment over consequential choices.

Deployment and monitoring

Deployment moves a verified build into a real environment through controlled gates. A CI/CD pipeline can run tests, evaluations, security scans and migration checks before release, while a human approval gate can remain for sensitive systems. Progressive rollout, feature flags and rollback procedures limit the impact of an unexpected result.

After release, monitoring closes the loop. Agents can watch logs and traces, group recurring failures, surface unusual cost or latency, and propose improvements. They may even implement low-risk fixes inside a controlled workflow, but production changes should still pass the same tests, reviews and deployment gates. Monitoring is not the final step; it generates the evidence that drives the next planning cycle.

1. LLM foundations: know the component you are directing

Large language models process tokens and generate a probability distribution over possible next tokens. That simple mechanism has practical consequences. It explains why models can be strong at synthesis but weak at exact counting, why context placement matters, and why a confident answer is not proof that the answer is correct.

An AI engineer needs to reason about:

• Tokenisation and context windows: what fits, what gets truncated and what should be summarised or retrieved later.
• Model selection: when a fast, inexpensive model is enough and when the task needs stronger reasoning, multimodal input or a specialist model.
• Sampling and reasoning controls: how temperature, output limits and reasoning effort affect consistency, latency and cost.
• Knowledge boundaries: why training knowledge has a cutoff and why current or private facts must come from tools or supplied context.
• Tool calling and structured output: when the model should produce prose, return validated data or request an external action.
• Adaptation: when prompt design is sufficient and when fine-tuning or self-hosting is justified.

Foundations prevent category errors. If the problem is missing information, changing the temperature will not solve it. If the problem is an ambiguous instruction, adding a vector database will not solve it. Understanding the model helps you choose the right intervention.

2. Grounding models with data: give the model the right evidence

Useful AI output depends on useful input context. Retrieval-augmented generation, or RAG, popularised vector search as a way to supply relevant passages, but modern grounding is a broader design problem.

You must decide what belongs directly in the prompt and what the model should retrieve on demand. You must also choose a representation that matches the question:

The ingestion pipeline matters as much as retrieval. PDFs, web pages, spreadsheets and images must be parsed, cleaned, segmented and enriched with metadata. Access controls must survive indexing. Updates and deletions must reach the index. Retrieval quality must be measured instead of assumed.

Good grounding does not force every available fact into the context window. It gives the model the smallest trustworthy evidence set needed for the current decision.

3. Building agentic systems: move from answering to acting

An agentic system does more than produce a response. It selects tools, performs multi-step work, observes results and adapts its next action. A useful agent might search an approved knowledge base, query a database, create a draft, run a check and request human approval before an external action.

This requires several kinds of design:

• Tools: narrow, well-described interfaces with validated inputs and clear errors.
• State and memory: enough continuity to complete the task without accumulating irrelevant or sensitive context.
• Control flow: explicit limits on retries, recursion, time, cost and permitted actions.
• Permissions: least-privilege access enforced outside the model.
• Human oversight: approval points for high-impact, irreversible or external actions.
• Multi-agent coordination: focused delegation only when separate roles improve quality, speed or review.

Framework features do not make a workflow safe. Reliability comes from the system around the model: schemas, permissions, deterministic checks, audit trails and clear stopping conditions.

4. Evaluation-driven development: replace impressions with evidence

AI teams often begin by reading a handful of outputs and deciding that the system “looks good.” That is a useful exploratory step, but it is not a release criterion.

Evaluation-driven development turns desired behaviour into a repeatable test set. Depending on the application, useful measures may include:

• Answer correctness, completeness and citation support
• Retrieval relevance and coverage
• Tool selection and argument accuracy
• Task completion rate and human escalation rate
• Safety, privacy and policy compliance
• Latency, token consumption and cost per successful task

Some checks can be deterministic: a JSON object matches its schema, a cited identifier exists, or a tool was not called without approval. Others need a scoring rubric, model-based grader or human review. Strong teams combine them.

The evaluation set should contain ordinary cases, difficult edge cases and known failures from production. Every important bug becomes a regression case. This is how the application improves without quietly breaking behaviour that already worked.

5. Operating in production: engineer the whole service

A model call is only one dependency in a production AI application. The complete service also needs authentication, secrets management, data protection, rate limits, queues, timeouts, fallbacks, monitoring and incident response.

Production observability should show more than whether the endpoint returned HTTP 200. Teams need to see the user request, retrieved context, tool calls, model response, validation result, latency and cost - with sensitive data redacted. Without traces, a failure becomes an anecdote. With traces, it becomes a diagnosable case and a candidate for the evaluation suite.

Models and data also change. Providers release new versions. Documents are updated. User behaviour shifts. A system that was reliable at launch can drift later, so evaluation, monitoring and rollback plans must continue after deployment.

6. Machine learning foundations: understand prediction and measurement

Not every AI problem needs an LLM. Forecasting demand, ranking leads, detecting fraud or classifying images may be better served by supervised learning, specialised models or a hybrid system.

Machine learning foundations help engineers reason about data quality, labels, train-validation-test splits, overfitting, class imbalance, leakage, calibration and distribution shift. They also help teams choose the metric that reflects the business cost of an error. Accuracy alone may conceal a system that misses most rare but important events.

This knowledge also sharpens LLM work. Many of the same habits apply: establish a baseline, separate development examples from final tests, measure failure modes, monitor drift and avoid optimising against a benchmark until it stops representing real use.

How the six skills connect

Imagine an internal support agent. LLM foundations help you choose the model and control its output. Grounding retrieves the right policy and account data. Agentic design lets it diagnose the issue and use approved support tools. Evaluation measures whether the resolution is correct and safe. Production engineering protects credentials and monitors the service. Machine learning foundations help you interpret the metrics and detect drift.

Remove any one layer and the weakness appears quickly. A capable model without grounding invents facts. A grounded chatbot without agent design cannot complete work. An agent without evaluation may fail silently. A tested prototype without production controls cannot be trusted with real users.

A practical learning path

1. Start with one bounded use case. Define the user, the decision or task, and what a successful result looks like.
2. Build the smallest end-to-end version. Include the real data and at least one real tool, not only a conversational mock-up.
3. Create evaluations early. Capture representative examples and write acceptance checks before polishing the interface.
4. Add autonomy gradually. Begin with read-only tools, explicit approvals and strict budgets.
5. Instrument every step. Record enough evidence to explain what happened without exposing secrets or personal data.
6. Iterate from failures. Treat each failure as information about the next improvement, not as a reason to hide the result.

Build these skills with Codex

The WSQ Agentic AI Applications with Codex course turns this mindset into hands-on practice. Learners plan an agentic application, build reusable workflows with Codex Skills and MCP tools, develop RAG and multi-agent applications, and test, evaluate and optimise their solutions.

The course also covers interfaces, APIs, deployment, secure coding, permission control, data privacy and human oversight. Eligible Singapore Citizens and Permanent Residents can receive WSQ funding, with baseline and enhanced funding options shown on the current course page. SkillsFuture Credit and PSEA may be used by eligible learners to offset the payable balance.

View the course outline, funding details and upcoming class dates.

Frequently asked questions