I Built a TypeScript SDK for Batch Processing LLM Calls Across Model Providers

Inspired by Vercel’s AI SDK (But for Batch Processing)

The Vercel AI SDK makes switching models really easy. Just swap the model name, and everything else stays the same. However, the AI SDK doesn't support batching, so I built one for that.

batch-ai gives you a single SDK that works across providers so you can focus on your app, not writing code for different batch APIs.

Why Batch API Calls Matter (Hint: They’re 50% Cheaper)

If you’re processing a high volume of AI requests and don’t need real-time responses, batch APIs can cut your costs in half.

For example, at Filtyr (my AI-powered content moderation SaaS), we process thousands of moderation events daily. By using OpenAI’s and Anthropic’s batch APIs instead of real-time calls, we save 50% on API costs while handling the same workload.

If your use case involves large-scale AI processing, sentiment analysis, classification, content moderation, or research, you should consider using batch APIs.

How to Use batch-ai

Here’s how batch-ai simplifies batch processing while letting you switch between providers effortlessly:

import { z } from 'zod';
import { openai, createObjectBatch, getObjectBatch } from 'batch-ai';

// Define output schema using Zod
const responseSchema = z.object({
  sentiment: z.enum(['positive', 'negative', 'neutral']),
  confidence: z.number().min(0).max(1),
});

// Initialize OpenAI model
const model = openai('gpt-4o');

// Batch requests
const requests = [
  { customId: 'review-1', input: 'I absolutely love this product! Best purchase ever.' },
  { customId: 'review-2', input: 'This is terrible, would not recommend.' },
];

// Create batch
const { batchId } = await createObjectBatch({
  model,
  requests,
  outputSchema: responseSchema,
});

// Retrieve batch results at some later point
const { batch, results } = await getObjectBatch({
  model,
  batchId,
});

if (batch.status === 'completed' && results) {
  console.log('Results:', results);
}

Want to switch to Anthropic? Just replace:

const model = anthropic('claude-3-5-sonnet-20241022');

That’s it. No need to rewrite anything else.

Who Should Use batch-ai?

If you’re dealing with high-volume AI processing, this SDK can help. Ideal users include:

• AI Moderation Platforms (like Filtyr) processing thousands of content moderation events daily.
• Marketing Teams analyzing customer sentiment at scale.
• Enterprises running classification, summarization, or AI-driven automation.
• Researchers working with massive datasets who need structured AI output efficiently.

If you process a lot of AI requests and want to cut costs while simplifying API interactions, batch processing is the way to go.

Future Plans & How You Can Get Involved

I built batch-ai to solve my own batch processing headaches, but there’s more to come:

• More provider support: Google Gemini and xAI Grok are next on the roadmap.
• Expanding batch capabilities: Adding generateTextBatch for text-based responses.
• Better error handling & retries: Making batch requests more robust.

I’d love your feedback! If you have feature ideas or run into issues, open an issue on GitHub.

---

What’s your experience with batch AI processing? Have you used batch APIs before? Let’s discuss in the comments!