Docs
Batch Inference
AWS Bedrock Setup

AWS Bedrock Batch Inference Setup

Set up AWS Bedrock for batch inference with the ANTS Platform AI Optimizer. Batch inference can be up to 50% cheaper when comparing large numbers of traces.

Batch Inference allows the AI Optimizer to evaluate model comparisons in bulk at reduced cost (up to 50% cheaper). See Provider Limits for batch size constraints.

Prerequisites

  • An AWS account with access to Amazon Bedrock
  • AWS CLI configured locally (for IAM setup)
  • Access to the models you want to use for comparison

Setup

Step 1: Create an IAM User

Create a dedicated IAM user for batch inference:

aws iam create-user --user-name bedrock-batch-inference-user

Step 2: Create the Bedrock Policy

Create a policy that grants access to Bedrock batch inference operations:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BedrockBatchInference",
      "Effect": "Allow",
      "Action": [
        "bedrock:CreateModelInvocationJob",
        "bedrock:GetModelInvocationJob",
        "bedrock:ListModelInvocationJobs",
        "bedrock:StopModelInvocationJob"
      ],
      "Resource": "*"
    }
  ]
}

Attach the policy to the user:

aws iam create-policy \
  --policy-name BedrockBatchInferencePolicy \
  --policy-document file://bedrock-policy.json
 
aws iam attach-user-policy \
  --user-name bedrock-batch-inference-user \
  --policy-arn "arn:aws:iam::ACCOUNT_ID:policy/BedrockBatchInferencePolicy"

Step 3: Create the S3 Policy

Batch inference requires S3 access for input/output data:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3BatchInference",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket",
        "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bedrock-batch-inference-*",
        "arn:aws:s3:::bedrock-batch-inference-*/*"
      ]
    }
  ]
}

Step 4: Create the Bedrock Service Role

Bedrock needs a service role to access S3 during batch jobs:

aws iam create-role \
  --role-name bedrock-batch-inference-service-role \
  --assume-role-policy-document file://trust-policy.json

Trust policy (trust-policy.json):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "bedrock.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attach S3 permissions to the service role so Bedrock can read/write batch data.

Step 5: Grant PassRole Permission

Your IAM user needs permission to pass the service role to Bedrock:

{
  "Sid": "PassRoleForBedrock",
  "Effect": "Allow",
  "Action": "iam:PassRole",
  "Resource": "arn:aws:iam::ACCOUNT_ID:role/bedrock-batch-inference-service-role",
  "Condition": {
    "StringEquals": {
      "iam:PassedToService": "bedrock.amazonaws.com"
    }
  }
}

Step 6: Configure in ANTS Platform

  1. Go to Project Settings > LLM API Keys
  2. Add a new key with AWS Bedrock adapter
  3. Enter your AWS credentials:
    • AWS Access Key ID
    • AWS Secret Access Key
    • AWS Region (e.g., us-east-1)
    • Service Role ARN (e.g., arn:aws:iam::123456789012:role/bedrock-batch-inference-service-role)
  4. Check "I intend to use Batch Inference for LLM optimizer"
⚠️

Replace ACCOUNT_ID with your actual AWS account ID in all ARN references above.

Self-Hosted Deployments

For self-hosted ANTS Platform instances, AWS credentials are optional in the LLM API key form. When omitted, authentication falls back to the AWS SDK default credential provider chain:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  2. Shared credentials file (~/.aws/credentials)
  3. EC2 instance profile / ECS task role
  4. EKS Pod Identity / IRSA

Troubleshooting

"Access Denied" on batch job creation

Verify your IAM user has the bedrock:CreateModelInvocationJob permission and the iam:PassRole permission for the service role.

"S3 bucket not found" errors

The S3 bucket is created automatically. Ensure your S3 policy allows s3:CreateBucket on the bedrock-batch-inference-* pattern.

Batch job stuck in "Submitted" state

Bedrock batch jobs may take time to start processing. Check your account's Service Quotas for batch size limits — see Batch Inference Limits for provider-specific requirements.

Rag SystemVertex AI Setup