Docs
Core Concepts
Three Pillars

Three Pillars of LLMOps

AgenticAnts implements LLMOps (Large Language Model Operations) through three foundational pillars that work together to provide comprehensive AI operations management.

Overview

LLMOps is the overarching discipline that encompasses the entire lifecycle of LLM operations from development to production. Our three pillars approach ensures complete coverage of AI operational needs:

┌─────────────────────────────────────┐
│                                     │
│           LLMOps Framework          │
│     Large Language Model Operations │
│                                     │
├─────────────────────────────────────┤
│                                     │
│      AI Cost (FinOps)               │
│     Cost Visibility & Optimization  │
│                                     │
├─────────────────────────────────────┤
│                                     │
│      AI Resilient (SRE)             │
│     Latency & Performance           │
│                                     │
├─────────────────────────────────────┤
│                                     │
│      AI Governance & Security       │
│     Compliance & Risk               │
│                                     │
└─────────────────────────────────────┘

LLMOps Framework

LLMOps provides the comprehensive framework for managing LLM operations:

  • Model Lifecycle Management - Selection, versioning, deployment, and retirement
  • Prompt Operations - Prompt engineering, versioning, and optimization
  • Performance Optimization - Latency, throughput, and cost optimization
  • Model Governance - Policies, compliance, and risk management
  • Versioning & Deployment - CI/CD pipelines and rollback strategies

Learn more about LLMOps →

Pillar 1: AI Cost (FinOps)

Cost Visibility, Allocation, Optimization & Accountability

What is AI Cost (FinOps)?

Cost (FinOps) for AI helps organizations understand, control, and optimize AI spending. Cost is the primary indicator and measurable outcome of FinOps - providing:

  • Cost Visibility: See where every dollar is spent in real-time
  • Cost Attribution: Track costs by customer, team, or product
  • Cost Optimization: Identify and eliminate waste
  • Cost Accountability: Budget allocation and forecasting

Key Capabilities

Token Usage Monitoring

Track every token consumed by your AI systems:

// Automatically tracked
const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [{ role: 'user', content: query }]
})
 
// AgenticAnts records:
// - Model used: gpt-4
// - Tokens: prompt=150, completion=200, total=350
// - Cost: $0.0105 (based on current pricing)

Cost Per Customer Query

Understand the economics of your AI operations:

# View cost breakdown
customer_costs = ants.metrics.get_customer_costs(
    start_date='2025-10-01',
    end_date='2025-10-31',
    group_by='customer'
)
 
# Results:
# customer_123: $45.50 (450 queries)
# customer_456: $89.20 (920 queries)
# customer_789: $12.30 (95 queries)

Budget Management

Set budgets and receive alerts:

await ants.budgets.create({
  name: 'Q4 AI Spending',
  amount: 10000,
  period: 'quarterly',
  alerts: [
    { threshold: 0.7, type: 'warning' },  // 70%
    { threshold: 0.9, type: 'critical' }  // 90%
  ]
})

ROI Analytics

Measure the business impact of AI investments:

const roi = await ants.analytics.calculateROI({
  costs: 5000,        // AI costs
  revenue: 25000,     // Revenue attributed to AI
  timePeriod: 'month'
})
 
// ROI: 400% (5x return on investment)
// Cost per conversion: $2.50
// Customer lifetime value: $500

AI Cost Best Practices

  1. Tag Everything: Use consistent tagging for cost attribution
  2. Set Budgets: Define spending limits for teams and projects
  3. Monitor Regularly: Review costs weekly, not monthly
  4. Optimize Models: Use smaller models where appropriate
  5. Cache Responses: Reduce redundant LLM calls

Learn more about AI Cost (FinOps) →

Pillar 2: AI Resilient (SRE)

Latency, Performance, Availability & Reliability

What is AI Resilient (SRE)?

Resilient (SRE) for AI systems emphasizes latency, performance, and operational health through SRE principles:

  • Low Latency: Fast response times and optimized performance
  • High Performance: Maximum throughput and efficiency
  • Reliable: High availability and fault tolerance with error budgets
  • Observable: Complete visibility into system behavior
  • SLAs/SLOs: Service level objectives and compliance

Key Capabilities

End-to-End Tracing

Follow requests through your entire AI stack:

// Trace shows complete execution path
Trace: customer-support-query (2.3s)
├─ Span: input-validation (10ms)
├─ Span: retrieve-customer-context (150ms)
│  └─ Span: database-query (145ms)
├─ Span: vector-search (200ms)
│  ├─ Span: embedding-generation (50ms)
│  └─ Span: similarity-search (150ms)
├─ Span: llm-inference (1.8s)
│  ├─ Span: prompt-construction (5ms)
│  ├─ Span: api-call (1.78s)
│  └─ Span: response-parsing (15ms)
└─ Span: response-formatting (140ms)

Performance Monitoring

Track key performance metrics:

# View performance metrics
metrics = ants.metrics.get_performance({
    agent: 'customer-support',
    period: 'last_24h'
})
 
print(f"Latency p50: {metrics.latency.p50}ms")  # 1,200ms
print(f"Latency p95: {metrics.latency.p95}ms")  # 3,500ms
print(f"Latency p99: {metrics.latency.p99}ms")  # 5,200ms
print(f"Error rate: {metrics.error_rate}%")     # 0.5%
print(f"Throughput: {metrics.throughput}/s")    # 45 req/s

Automated Alerting

Get notified when things go wrong:

await ants.alerts.create({
  name: 'High Error Rate',
  condition: 'error_rate > 5%',
  window: '5m',
  channels: ['slack', 'pagerduty'],
  severity: 'critical'
})
 
await ants.alerts.create({
  name: 'Slow Response Time',
  condition: 'p95_latency > 5000ms',
  window: '10m',
  channels: ['email'],
  severity: 'warning'
})

Incident Response

Quickly diagnose and resolve issues:

# Get incident details
incident = ants.incidents.get('inc-123')
 
# View timeline
for event in incident.timeline:
    print(f"{event.time}: {event.description}")
 
# Identify root cause
root_cause = ants.incidents.analyze_root_cause('inc-123')
print(f"Root cause: {root_cause.description}")
 
# View similar incidents
similar = ants.incidents.find_similar('inc-123')

AI Resilient Best Practices

  1. Set SLOs: Define Service Level Objectives for latency and availability
  2. Monitor Proactively: Don't wait for users to report performance issues
  3. Automate Responses: Auto-remediate common issues
  4. Learn from Incidents: Conduct post-mortems
  5. Test Resilience: Implement chaos engineering

Learn more about AI Resilient (SRE) →

Pillar 3: AI Governance and Security Posture

Compliance, Risk Management & Policy Enforcement

What is AI Governance and Security Posture?

AI Governance and Security Posture provides comprehensive oversight of AI/LLM operations through governance, compliance, risk management, and security:

  • AI Governance: Policy enforcement and model usage controls
  • Compliance: Meet regulatory requirements (SOC2, GDPR, HIPAA, AI Act)
  • Risk Management: Continuous assessment and mitigation
  • Data Protection: Prevent sensitive data leaks
  • Access Control: Manage who can access what with RBAC
  • Audit Trails: Complete logs for compliance and forensics

Key Capabilities

PII Detection & Protection

Automatically identify and protect sensitive data:

// AgenticAnts automatically detects PII
const trace = await ants.trace.create({
  name: 'customer-query',
  input: 'My SSN is 123-45-6789 and email is john@example.com'
})
 
// Dashboard shows:
// - PII detected: SSN, Email
// - Automatically redacted in storage
// - Alert sent to security team
// - Audit log created

Security Guardrails

Prevent harmful or policy-violating outputs:

# Configure guardrails
ants.guardrails.create({
    'name': 'content-policy',
    'rules': [
        {'type': 'no_pii', 'action': 'redact'},
        {'type': 'no_toxic_content', 'action': 'block'},
        {'type': 'no_financial_advice', 'action': 'warn'}
    ]
})
 
# Automatically enforced on all outputs
response = agent.run(query)  # Checked against guardrails

Compliance Reporting

Generate compliance reports automatically:

// Generate SOC2 compliance report
const report = await ants.compliance.generate({
  framework: 'SOC2',
  period: 'Q4-2025',
  controls: [
    'access-control',
    'data-encryption',
    'audit-logging',
    'incident-response'
  ]
})
 
// Download GDPR data export
const gdprExport = await ants.compliance.exportData({
  userId: 'user-123',
  format: 'json'
})

RBAC & Access Control

Fine-grained permissions management:

# Create role
ants.roles.create({
    'name': 'data-scientist',
    'permissions': [
        'traces.read',
        'metrics.read',
        'dashboards.read',
        'projects.list'
    ],
    'resources': ['project-123', 'project-456']
})
 
# Assign to user
ants.users.assign_role('user-789', 'data-scientist')

Audit Trails

Complete logging of all activities:

// Query audit logs
const logs = await ants.audit.query({
  action: 'data.export',
  startDate: '2025-10-01',
  endDate: '2025-10-31'
})
 
for (const log of logs) {
  console.log(`${log.timestamp}: ${log.user} ${log.action}`)
  console.log(`  Resource: ${log.resource}`)
  console.log(`  IP: ${log.ip}`)
  console.log(`  Status: ${log.status}`)
}

AI Governance and Security Posture Best Practices

  1. Principle of Least Privilege: Give minimum necessary access
  2. Policy Enforcement: Define and enforce AI/LLM usage policies
  3. Regular Audits: Review access, compliance, and activities regularly
  4. Risk Assessment: Continuously assess and mitigate AI risks
  5. Incident Response Plan: Have a plan before incidents occur

Learn more about AI Governance and Security Posture →

Integration of the Three Pillars

The pillars work together to provide comprehensive coverage:

Example: Production Incident

1. AI Resilient (SRE): Detects high error rate
   └─> Alert sent to on-call engineer

2. AI Cost (FinOps): Identifies cost spike
   └─> Related to failed retries

3. AI Governance: Reviews audit logs
   └─> No security breach detected, compliance maintained

4. Resolution:
   - AI Resilient fixes the performance issue
   - AI Cost tracks cost impact
   - AI Governance documents for compliance

Example: Cost Optimization

1. AI Cost (FinOps): Identifies expensive queries
   └─> Customer X costs $500/day

2. AI Resilient (SRE): Analyzes latency
   └─> Slow queries causing retries

3. AI Governance: Checks data access
   └─> Proper authorization and policy compliance confirmed

4. Optimization:
   - AI Resilient optimizes query performance
   - AI Cost validates cost reduction
   - AI Governance audits changes for compliance

Getting Started with Each Pillar

Understanding AI AgentsCredit System