Flink & Kafka: Building A Basic Real-Time Script

Dec 3, 2025 by Alex Johnson 49 views

Are you looking to dive into the world of real-time data processing with Flink and Kafka? This guide will walk you through creating a basic Flink script that connects to Kafka, reads messages, and prints them to the console. This is a fundamental step in understanding how to build more complex data pipelines, so let's get started!

Understanding Flink and Kafka

Before we jump into the code, let's take a moment to understand the technologies we'll be using.

Apache Flink: Flink is a powerful, open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. It excels at processing continuous data streams in real-time, making it ideal for applications like fraud detection, real-time analytics, and data pipeline orchestration.
Apache Kafka: Kafka is a distributed, fault-tolerant streaming platform that enables you to build real-time data pipelines and streaming applications. It acts as a central nervous system for your data, allowing you to ingest, store, and process data streams from various sources.

Together, Flink and Kafka form a potent combination for building robust and scalable real-time data processing solutions. Flink provides the processing engine, while Kafka acts as the data backbone, ensuring reliable data ingestion and delivery.

Prerequisites

Before we start coding, make sure you have the following set up:

Java Development Kit (JDK): Flink is a Java-based framework, so you'll need a JDK installed (Java 8 or later is recommended).
Apache Maven: Maven is a build automation tool used for building and managing Java projects. You'll need it to manage Flink dependencies.
Apache Kafka: You'll need a running Kafka cluster to connect your Flink application to. You can download and set up Kafka locally or use a managed Kafka service.
An IDE (Integrated Development Environment): While not strictly required, an IDE like IntelliJ IDEA or Eclipse will make development much easier.

Once you have these prerequisites in place, you're ready to start building your Flink Kafka script!

Setting up Your Flink Project

The first step is to create a new Maven project for your Flink application. You can do this using your IDE or by using the command line. Here's an example Maven command to create a new project:

mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-quickstart-java \
    -DarchetypeVersion=1.17.1 \
    -DgroupId=com.example \
    -DartifactId=flink-kafka-example \
    -Dversion=1.0-SNAPSHOT \
    -Dpackage=com.example.flink

Replace 1.17.1 with the version of Flink you want to use. This command will generate a basic Flink project structure with a pom.xml file for managing dependencies and a src directory for your code.

Adding Flink and Kafka Dependencies

Next, you need to add the necessary Flink and Kafka dependencies to your pom.xml file. This will allow your project to use the Flink Kafka connector and other Flink libraries. Open your pom.xml file and add the following dependencies within the <dependencies> tags:

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>1.17.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java</artifactId>
        <version>1.17.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients</artifactId>
        <version>1.17.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-kafka</artifactId>
        <version>1.17.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>3.6.1</version>
    </dependency>

    <!-- Add logging framework, to produce console output when running in the IDE. -->
    <!-- These dependencies are excluded from the application JAR to avoid conflicts with Flink's own dependencies. -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.36</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.36</version>
        <scope>runtime</scope>
    </dependency>
</dependencies>

Make sure to replace 1.17.1 with the Flink version you're using and 3.6.1 with the Kafka Client version. Maven will automatically download these dependencies when you build your project.

Writing the Flink Kafka Script

Now, let's write the core Flink script that connects to Kafka, reads messages, and prints them to the console. Create a new Java class (e.g., KafkaFlinkConsumer.java) in your src/main/java/com/example/flink directory and add the following code:

package com.example.flink;

import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;

import java.util.Properties;

public class KafkaFlinkConsumer {

    public static void main(String[] args) throws Exception {
        // Set up the streaming execution environment
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // Configure Kafka consumer properties
        Properties properties = new Properties();
        properties.setProperty("bootstrap.servers", "localhost:9092"); // Replace with your Kafka brokers
        properties.setProperty("group.id", "flink-consumer-group");

        // Specify the Kafka topic
        String topic = "your-topic"; // Replace with your Kafka topic

        // Create a Flink Kafka consumer
        FlinkKafkaConsumer<String> kafkaConsumer = new FlinkKafkaConsumer<>(topic, new SimpleStringSchema(), properties);

        // Create a data stream from the Kafka consumer
        DataStream<String> stream = env.addSource(kafkaConsumer);

        // Print the messages to the console
        stream.map(new MapFunction<String, String>() {
            @Override
            public String map(String value) throws Exception {
                System.out.println("Received message: " + value);
                return value;
            }
        }).print();

        // Execute the Flink job
        env.execute("Flink Kafka Consumer");
    }
}

Let's break down this code step by step:

Set up the Streaming Execution Environment:
```
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
```
This line creates a StreamExecutionEnvironment, which is the entry point for any Flink streaming application. It allows you to configure the execution environment and define your data processing pipeline.
Configure Kafka Consumer Properties:
```
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "localhost:9092"); // Replace with your Kafka brokers
properties.setProperty("group.id", "flink-consumer-group");
```
This section configures the properties for the Kafka consumer. You need to specify the bootstrap.servers, which is a comma-separated list of Kafka brokers. Replace localhost:9092 with the actual address of your Kafka brokers. The group.id property defines the consumer group, which allows multiple consumers to read from the same topic in a load-balanced manner.
Specify the Kafka Topic:
```
String topic = "your-topic"; // Replace with your Kafka topic
```
This line specifies the Kafka topic you want to consume messages from. Replace `