How to Implement Dynamic Multilingual Sitemap with 40,000 URLs in Next.js App Router

Introduction

This article explains implementing a dynamic sitemap for a large-scale website using Next.js App Router. I'll share the implementation method for a multilingual site containing over 40,000 content items.

TL;DR

By combining Next.js sitemap API with 24-hour caching, we implemented a practical dynamic sitemap solution for large-scale sites. This approach minimizes the impact on frontend build time while enabling daily data updates.

Development Environment

• Frontend: Next.js (App Router)
• Backend: Express + MongoDB
• Infrastructure: Vercel (Frontend), Heroku (Backend)
• Languages: 2 (bilingual)

Context

• Over 20,000 unique pages
• Each page has 2 language variations (total: 40,000+ URLs)
• Daily data updates
• Sitemap required for SEO requirements

Understanding Sitemap Generation

There are three main approaches to sitemap generation:

1. Static Generation

   • Fast response time
   • Low server load
   • Longer build times
   • Difficult to handle immediate data updates

2. Dynamic Generation

   • Easy reflection of latest data
   • No impact on build time
   • Higher server load
   • Potential slower response times

3. ISR (Incremental Static Regeneration)

   • Balanced approach between static and dynamic
   • Easy cache control
   • Slightly more complex implementation

I had written about basic sitemap generation in the following document:
Implementing Multilingual Sitemap with next-intl in Next.js App Router

Implementation

We adopted an implementation combining Next.js sitemap API with 24-hour caching:

// app/sitemap.ts
import { MetadataRoute } from "next";

const BASE_URL = process.env.NODE_ENV === "production" 
  ? "https://example.com"
  : "localhost:3000";

const API_BASE_URL = process.env.NODE_ENV === "production"
  ? `${process.env.NEXT_PUBLIC_API_BASE_URL}/api/util`
  : "http://localhost:3500/api/util";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const locales = ["en", "ja"];
  const routes = ["/", "/about", "/products", "/contact"];

  // Generate static routes
  const staticRoutes = routes.flatMap((route) =>
    locales.map((locale) => ({
      url: `${BASE_URL}/${locale}${route === "/" ? "" : route}`,
      lastModified: new Date(),
      changeFrequency: "monthly" as const,
      priority: route === "/" ? 1 : 0.8,
    }))
  );

  try {
    // Fetch product data with caching
    const response = await fetch(`${API_BASE_URL}/products`, {
      next: { revalidate: 86400 }, // Cache for 24 hours
      headers: { "Cache-Control": "public, max-age=86400" },
    });

    if (!response.ok) {
      throw new Error(`Failed to fetch products: ${response.statusText}`);
    }

    const products = await response.json();

    // Generate product URLs
    const productRoutes = products.flatMap((product: any) =>
      locales.map((locale) => ({
        url: `${BASE_URL}/${locale}/product/${product.id}`,
        lastModified: new Date(product.updatedAt),
        changeFrequency: "daily" as const,
        priority: 0.7,
      }))
    );

    return [...staticRoutes, ...productRoutes];
  } catch (error) {
    console.error("Error generating sitemap:", error);
    return staticRoutes; // Fallback to static routes on error
  }
}

Backend API implementation:

// Express route handler
router.get("/products", async (req, res) => {
  try {
    // Fetch minimal required data from MongoDB
    const products = await Product.find(
      {},
      {
        id: 1,
        lastUpdateDate: 1,
      }
    ).lean();
    res.json(products);
  } catch (error) {
    console.error("Failed to fetch products:", error);
    res.status(500).json({
      message: "Error fetching products",
      error: error instanceof Error ? error.message : String(error),
    });
  }
});

Implementation Results

• Total URLs: 42,000 (21,000 for en, ja)
• File size: 4.6MB
• Generation time: Under 5 seconds
• Impact on build time: Minimal
• Cache duration: 24 hours

Checking results

You can check the generated sitemap in dev environment, localhost:3000/sitemap.xml.

Key Implementation Considerations

1. Caching Strategy

   • Use both Next.js revalidation and HTTP caching
   • 24-hour cache balances freshness and performance
   • Implement fallback to static routes on errors

2. Error Handling

   • Graceful degradation to static routes
   • Structured error logging
   • Monitoring integration recommended

3. SEO Optimization

   • Proper lastModified dates
   • Appropriate priority settings
   • Consistent language codes

Conclusion

Next.js Sitemap API proves capable of handling large-scale multilingual sites effectively. The key to success lies in implementing proper caching strategies and error handling, enabling a balance between performance and maintainability.

Resources

• Implementing Multilingual Sitemap with next-intl in Next.js App Router
• Next.js Sitemap API Documentation
• Google Search Central - Build and submit a sitemap
• Understanding Cache-Control Headers