Auto-Creating Kafka Topics in Spring Boot Microservices

Key Configuration: Partitions: 3 | Replicas: 3 | min.insync.replicas: 2

When building event-driven microservices with Spring Boot and Kafka, it's crucial to ensure topics exist before producers attempt to send messages. Programmatic topic creation during startup provides a robust solution that works across all environments. This approach guarantees your topics have the right configuration while avoiding manual setup steps.

Topic Configuration Class

KafkaConfig.java

package com.example.productservice.config;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.Map;

@Configuration
public class KafkaConfig {

    public static final String PRODUCT_CREATED_TOPIC = "product-created-events";
    private static final int PARTITIONS = 3;
    private static final short REPLICATION_FACTOR = 3;

    @Bean
    public NewTopic productCreatedEventsTopic() {
        return TopicBuilder.name(PRODUCT_CREATED_TOPIC)
                .partitions(PARTITIONS)
                .replicas(REPLICATION_FACTOR)
                .configs(Map.of(
                    "min.insync.replicas", "2",
                    "retention.ms", "604800000" // 7 days
                ))
                .build();
    }
}

Parameter	Recommended Value	Purpose
Partitions	3-6	Enables parallel processing
Replication Factor	3 (production)	Ensures fault tolerance

Environment-Specific Configurations

Environment-Specific Configuration

// For local development (single broker)
@Profile("!prod")
@Bean
public NewTopic localProductTopic() {
    return TopicBuilder.name(PRODUCT_CREATED_TOPIC)
            .partitions(1)
            .replicas((short) 1)
            .build();
}

// For production environment
@Profile("prod")
@Bean
public NewTopic productionProductTopic() {
    return TopicBuilder.name(PRODUCT_CREATED_TOPIC)
            .partitions(6)
            .replicas((short) 3)
            .configs(Map.of(
                "min.insync.replicas", "2",
                "retention.ms", "2592000000" // 30 days
            ))
            .build();
}

Important: When running locally with a single broker, set replication factor to 1 to avoid startup errors

Complete Configuration Example

Advanced Topic Configuration

@Bean
public KafkaAdmin.NewTopics topics() {
    return new KafkaAdmin.NewTopics(
        TopicBuilder.name("product-created-events")
            .partitions(3)
            .replicas(3)
            .config(Map.of(
                "min.insync.replicas", "2",
                "retention.ms", "604800000",
                "cleanup.policy", "compact,delete"
            ))
            .build(),
            
        TopicBuilder.name("inventory-updates")
            .partitions(6)
            .replicas(3)
            .build()
    );
}

Production Tip: Always set min.insync.replicas to (replication_factor - 1) for durability. For RF=3, set min.insync.replicas=2

Implementation Steps

1. Create a new @Configuration class in your Spring Boot application
2. Add the spring-kafka dependency to your pom.xml or build.gradle
3. Define topics as @Bean using TopicBuilder
4. Configure partitions based on expected throughput
5. Set appropriate replication factor for your environment

Next Steps: Learn about Kafka consumer groups | Sample code available on GitHub

Frequently Asked Questions

Q1. What happens if the topic already exists?
A: Kafka will ignore the creation request if the topic exists with identical configuration. If configurations differ, you'll need to either delete the topic first or update its configuration manually.

Q2. How to handle different environments?
A: Use Spring profiles to provide environment-specific configurations. The code examples above show how to configure different settings for local development vs production.

Q3. What are the most important topic configurations?
A: Key configurations include: partitions (for parallelism), replication factor (for durability), retention (how long to keep messages), and cleanup policy (compact or delete).