Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 35 hours
Course Outline
Introduction and Diagnostic Foundations
- Overview of common failure modes in LLM systems and Ollama-specific challenges.
- Establishing reproducible experiments and controlled environments.
- Essential debugging tools: local logs, request/response capture, and sandboxing techniques.
Reproducing and Isolating Failures
- Techniques for creating minimal failing examples and test seeds.
- Stateful versus stateless interactions: strategies for isolating context-related bugs.
- Managing determinism, randomness, and controlling non-deterministic behaviour.
Behavioural Evaluation and Metrics
- Quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies.
- Qualitative evaluations: human-in-the-loop scoring and rubric design.
- Task-specific fidelity checks and defining acceptance criteria.
Automated Testing and Regression
- Unit tests for prompts and components, alongside scenario and end-to-end testing.
- Creating regression suites and establishing golden example baselines.
- CI/CD integration for Ollama model updates and automated validation gates.
Observability and Monitoring
- Structured logging, distributed tracing, and the use of correlation IDs.
- Key operational metrics: latency, token usage, error rates, and quality signals.
- Alerting strategies, dashboard creation, and SLIs/SLOs for model-backed services.
Advanced Root Cause Analysis
- Tracing through graphed prompts, tool calls, and multi-turn flows.
- Comparative A/B diagnosis and ablation studies.
- Data provenance, dataset debugging, and addressing failures induced by data quality.
Safety, Robustness, and Remediation Strategies
- Mitigation techniques: filtering, grounding, retrieval augmentation, and prompt scaffolding.
- Rollback, canary, and phased rollout patterns for model updates.
- Post-mortem analysis, lessons learned, and continuous improvement loops.
Summary and Next Steps
Requirements
- Substantial experience in building and deploying Large Language Model (LLM) applications.
- Proficiency with Ollama workflows and model hosting processes.
- Working knowledge of Python, Docker, and fundamental observability tools.
Target Audience
- AI Engineers
- MLOps Professionals
- QA Teams managing production LLM systems
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customised Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from £8000 + VAT*
Contact us for an exact quote and to hear our latest promotions