Course Outline
Introduction to LLMOps
- LLMOps vs MLOps: unique challenges of operating LLMs
- The LLM application lifecycle: prompt, evaluate, deploy, monitor
- Production readiness checklist for GenAI applications
Prompt Management and Versioning
- Prompt templating systems and variable injection
- Semantic versioning for prompts with automated regression testing
- Prompt registries and collaboration workflows
LLM Evaluation at Scale
- Evaluation dimensions: accuracy, relevance, safety, groundedness
- LLM-as-judge metrics and human evaluation pipelines
- Automated eval frameworks: RAGAS, DeepEval, and custom evaluators
- Quality gates in CI/CD for LLM deployments
Safety Guardrails and Content Governance
- Input and output guardrails: NeMo Guardrails and Guardrails AI
- PII detection, toxicity filtering, and topic boundaries
- Jailbreak and prompt injection defense strategies
- Red-teaming LLM applications for safety assurance
LLM Observability and Monitoring
- Telemetry: token usage, latency, cost, and quality metrics
- Drift detection in LLM outputs and embedding spaces
- Session-level tracing for multi-turn agent conversations
- Dashboards and alerting with LangSmith, Arize, and OpenTelemetry
AI Gateway and Model Orchestration
- Multi-provider routing with LiteLLM and Portkey
- Fallback strategies, retry logic, and circuit breakers
- Cost-aware model selection and load balancing
- Rate limiting, quota management, and API key governance
Performance Optimisation
- Semantic caching with vector stores and exact-match strategies
- Structured output enforcement with constrained decoding
- Batching, streaming, and concurrency patterns
- Latency optimization across model providers
Governance, Compliance, and Audit
- LLM audit trails: prompt logs, response logs, and decision provenance
- Data residency and privacy considerations for LLM APIs
- Policy-as-code for LLM usage within organizations
- Building an internal LLM operations playbook
Requirements
- Experience building or integrating LLM-powered applications.
- Familiarity with Python and REST APIs.
- Basic understanding of prompt engineering concepts.
Audience
- ML engineers and MLOps practitioners transitioning to LLM operations.
- Platform engineers responsible for LLM infrastructure.
- Technical leads managing production GenAI deployments.
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customised Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from £3200 + VAT*
Contact us for an exact quote and to hear our latest promotions
Testimonials (3)
Correct use of prompts
Marses Pacaldo
Course - Generative AI and Prompt Engineering for Corporate Professionals
The interactive style, the exercises
Tamas Tutuntzisz
Course - Introduction to Prompt Engineering
A great repository of resources for future use, instructor's style (full of good sense of humor, great level of detail)
