The first time I had to secure a Kafka cluster, I felt like I was drowning in acronyms: SASL, SSL, SCRAM, Kerberos… The more I delved into the documentation, the more confused I became.
If you’ve felt the same, don’t worry! You’re not alone. Kafka security sounds more complicated than it is. Once you separate protocols (the road your data travels on) from mechanisms (how you prove who you are), it starts making sense.
At each step, I'll break down the jargon and provide practical configuration examples.
Why Kafka Security Even Matters
Kafka often sits at the heart of an organization’s data pipeline. It’s where clickstream data, financial transactions, and system logs all flow.
If it’s left unsecured:
Anyone could publish garbage messages.
Attackers could silently read sensitive data.
A bad actor could impersonate a legitimate client.
That’s why Kafka bakes in a flexible security model. You just have to piece together the building blocks.
Protocols vs. Mechanisms: A Simple Analogy
Here’s the mental model that helped me:
Protocol = the road. Is it safe to drive on? Is it fenced off? Or is it a wide-open dirt path anyone can walk onto?
Mechanism = the toll booth. How do you prove you belong there? Show an ID, enter a password, flash a badge?
Kafka gives you both:
SSL/TLS: The Secure Tunnel
SSL/TLS makes sure no one can peek into or tamper with the messages flowing between your clients and brokers.
How it works in practice:
The Kafka broker shows a certificate, proving it’s legit.
The client checks if the broker’s certificate can be trusted.
Optionally, the client shows its own certificate back (mutual TLS).
Here’s what that looks like in Kafka broker config:
listeners=SSL://:9092
ssl.keystore.location=/etc/kafka/secrets/kafka.server.keystore.jks
ssl.keystore.password=YOUR_PASSWORD
ssl.key.password=YOUR_PASSWORD
ssl.truststore.location=/etc/kafka/secrets/kafka.server.truststore.jks
ssl.truststore.password=YOUR_PASSWORD
From then on, all traffic over the port 9092 is encrypted.
What are the Keystore and Truststore?
Keystore: Think of it as your ID wallet. It holds your private key and certificate that prove who you are (the broker in this case).
Truststore: This is your list of trusted IDs. It contains certificates from other parties you trust (like clients or certificate authorities).
In the example above, the broker uses its keystore to prove its identity to clients. The truststore is used to verify client certificates if mutual TLS is enabled. So, the truststore is what the broker uses to check others' identities.
What are those JKS files?
.jks files are Java KeyStores, which are file formats used to store cryptographic keys and certificates. In Kafka, they are used for SSL/TLS encryption to ensure secure communication between clients and brokers. In other environments, other formats like PEM or PFX might be used, but Kafka primarily relies on JKS due to its Java foundation.
You can generate JKS files using Java's keytool utility. For a step-by-step guide, check out this resource.
Understanding SSL vs. TLS: What's the Difference?
SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are like the older and younger siblings in the world of secure communication protocols. While they often get mentioned together, they have some important differences:
Evolution: Think of SSL as the trailblazer. It paved the way, but TLS took over with more advanced versions. SSL 3.0 was the last of its kind, and then TLS 1.0 came along, evolving through versions 1.1, 1.2, and now the latest, 1.3, each bringing stronger security features.
Security Enhancements: TLS is like a fortified castle compared to SSL's wooden fort. It uses stronger encryption algorithms and more secure hash functions, making it a tough nut to crack against modern cyber threats.
Deprecation of SSL: SSL has had its day in the sun, but due to vulnerabilities, it's now considered outdated and insecure. Most systems have moved on to TLS to keep data safe and sound.
Handshake Process: Both protocols use a handshake to establish a secure connection, but TLS does it with more finesse. Its handshakes are more efficient and secure, especially in the latest versions.
Backward Compatibility: TLS is designed to be backward compatible with SSL, allowing systems to support both during transitions. However, it's wise to disable SSL support to avoid potential security risks.
In a nutshell, while SSL laid the groundwork, TLS is the modern standard that offers enhanced security and performance. When securing Kafka or any other system, using the latest version of TLS is crucial to fend off vulnerabilities.
SASL: The Authentication Framework
SASL (Simple Authentication and Security Layer) isn’t an encryption, but a framework for checking identities. You can run SASL on:
PLAINTEXT (not secure, don’t use in production).
SSL (secure, because SSL wraps the authentication exchange in encryption).
So in real deployments, you’ll almost always see SASL_SSL.
SASL Mechanisms in Kafka
Here are the main authentication options you can plug into SASL:
Example: SASL_PLAINTEXT with PLAIN
Let's say you want to set up a quick test environment using SASL_PLAINTEXT with the PLAIN mechanism. Here's how you can configure both the Kafka broker and client.
Broker config:
listeners=SASL_PLAINTEXT://:9092
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
Wait! inter.broker.protocol? What is that?
This setting defines how Kafka brokers authenticate with each other. In a multi-broker setup, brokers need to communicate securely. By setting sasl.mechanism.inter.broker.protocol=PLAIN, you're specifying that brokers will use the PLAIN mechanism for their internal communication. Of course, you can choose the same mechanism as clients or a different one, depending on your security requirements.
Example: SASL_PLAINTEXT with SCRAM
This is a step up from PLAIN, adding better password security:
Broker config:
listeners=SASL_PLAINTEXT://:9092
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
How is SCRAM more secure than PLAIN?
SCRAM enhances security over PLAIN in several key ways:
Password Storage: In PLAIN, passwords are stored in clear text, making them vulnerable if the storage is compromised. SCRAM stores passwords in a salted and hashed format, which means even if someone gains access to the stored credentials, they can't easily retrieve the original passwords.
Challenge-Response Mechanism: SCRAM uses a challenge-response mechanism during authentication. This means that the password is never sent over the network in clear text. Instead, a challenge is issued, and the client responds with a hashed version of the password combined with the challenge, making it much harder for attackers to intercept and misuse the password.
Salting: SCRAM adds a unique salt to each password before hashing it. This means that even if two users have the same password, their stored hashes will be different, protecting against rainbow table attacks.
Iterative Hashing: SCRAM allows for multiple iterations of hashing, which increases the time it takes to compute the hash. This makes brute-force attacks significantly more difficult, as attackers would need to spend more time and resources to guess passwords.
Put them all together: SASL_SSL
This combo is one of the most common in production because it’s secure without being overcomplicated.
Broker config:
listeners=SASL_SSL://:9094
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
ssl.keystore.location=/etc/kafka/secrets/kafka.server.keystore.jks
ssl.keystore.password=YOUR_PASSWORD
ssl.key.password=YOUR_PASSWORD
ssl.truststore.location=/etc/kafka/secrets/kafka.server.truststore.jks
ssl.truststore.password=YOUR_PASSWORD
Client config:
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="alice" \
password="alice-secret"
Result:
Docker Compose Example
Here is a full example using Docker Compose with Bitnami's Kafka image, which supports three listeners: Plaintext, SASL_PLAINTEXT with SCRAM, and SASL_SSL with SCRAM out of the box. This setup includes SSL certificates and user credentials. For more information, check out this GitHub repo. There I have explained how to generate the certificates and run test clients to see how different mechanisms and protocols work.
version: '3.8'
services:
kafka:
image: bitnami/kafka:3.7
container_name: kafka
environment:
- KAFKA_KRAFT_MODE=true
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_NODE_ID=1
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka:9093
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,SASL_PLAINTEXT://:29092,SASL_SSL://:39092,CONTROLLER://:9093
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,SASL_PLAINTEXT://kafka:29092,SASL_SSL://kafka:39092
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=SASL_PLAINTEXT:SASL_PLAINTEXT,PLAINTEXT:PLAINTEXT,SASL_SSL:SASL_SSL,CONTROLLER:PLAINTEXT
- KAFKA_CFG_SSL_CLIENT_AUTH=required
- KAFKA_CFG_SSL_KEYSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.keystore.jks
- KAFKA_CFG_SSL_KEYSTORE_PASSWORD=bitnami123
- KAFKA_CFG_SSL_KEY_PASSWORD=bitnami123
- KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/bitnami/kafka/config/certs/kafka.truststore.jks
- KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=bitnami123
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_SASL_ENABLED_MECHANISMS=SCRAM-SHA-512
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=SCRAM-SHA-512
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=SASL_PLAINTEXT
- KAFKA_CLIENT_USERS=testuser
- KAFKA_CLIENT_PASSWORDS=testpass
- KAFKA_INTER_BROKER_USER=admin
- KAFKA_INTER_BROKER_PASSWORD=adminpass
- KAFKA_CFG_SUPER_USERS=User:admin
- KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_CFG_CLUSTER_ID=VkGbvjMzQNKtC-P_RMzqgg
ports:
- "9092:9092"
- "29092:29092"
- "39092:39092"
volumes:
- ./certs:/bitnami/kafka/config/certs:ro
What are those certs?
The certs directory contains the necessary SSL certificates and keystores for secure communication. You can generate these using tools like OpenSSL or Java's keytool.
Final Thoughts
Common Real-World Patterns
SSL only → Secure channel, optional client certs.
SASL_SSL + SCRAM → The sweet spot for many teams.
SASL_SSL + Kerberos/OAUTHBEARER → Enterprises with Kerberos in place.
Best Practices (Learned the Hard Way)
Avoid using SASL_PLAINTEXT outside of development environments unless you are certain the network channel is secure.
Rotate credentials and certificates regularly.
Prefer SCRAM or OAUTHBEARER over PLAIN for authentication.
Use Kafka ACLs to enforce the least privilege.
Conclusion
Securing Kafka might initially seem daunting due to the myriad of acronyms and configurations, but breaking it down into its core components—protocols and mechanisms—simplifies the process. By understanding SSL/TLS as the secure channel and SASL as the authentication framework, you can effectively protect your Kafka cluster. Implementing best practices such as using SCRAM or OAUTHBEARER for authentication, regularly rotating credentials, and enforcing Kafka ACLs ensures robust security. With these insights, securing Kafka becomes a manageable task, transforming what once felt like a complex challenge into a straightforward process.
The trick to understanding Kafka security is simple:
SSL/TLS = the secure pipe (encryption).
SASL = the framework for authentication.
Mechanisms = the actual way you prove identity.
Once you see it this way, the acronym soup clears up, and securing Kafka stops feeling like dark magic.
