Kafka 3.8 Brings Faster Startups to Java Developers
Java developers who wrangle with Kafka are getting some love from the new release of the open source data streaming platform.
The project has packaged the latest version, Kafka 3.8, in a GraalVM, a container for enterprise Java environments.
While not jaw-dropping, for Java developers this is some good news, noted Confluent Developer Advocate Sandon Jacobs, who, prior to joining Confluent, worked as a Java developer on Kafka.
Most developers are probably writing their apps for Kafka on local machines, and, as with any good development process, this involves lots of testing. Installing Kafka on a local machine with the Java Virtual Machine, and then re-installing it after things “blow up,” is a time-consuming process, he said.
The GraalVM, which works natively with the JVM, will streamline this process quite a bit, Jacobs said. Muck up your copy of Kafka, and you can just swap in a new GraalVM copy. Kafka itself is written in Java.
“The start-up times on these GraalVM images are phenomenal,” Jacobs said, in an interview with TNS. “It’s going to greatly increase the efficiency of developers in that restart cycle.”
Kafka has offered a Docker container of Kafka as well, though, with this release, the Kafka container will now be officially supported by Docker.
Performance Tuning in Kafka
Beyond the new packaging, many of the improvements with this release help users tune Kafka to be more performant.
For instance, users can now get a lot more control over which compression scheme to use. To date, Kafka offered a select number of compression schemes: gzip, lz4 and zstd, with default settings for each.
“These defaults give you a generally good performance. But now Apache Kafka with 3.8 allows you to be able to set that level, using the values for whatever compression codec that you’re using,” Jacobs said.
With compression, he explained, there is always a trade-off between throughput and CPU usage. The more compressed data is, the faster it goes through the system and the less storage it requires, but at the same time more computational power it requires to compress and decompress.
The new controls allow admins to put into effect their own preferences, allowing them to choose different levels, depending on their own needs.
“You got to find that trade-off for your use case,” Jacobs said.
2/9
KIP-390: Support Compression Level allows you to configure the compression level of each supported algorithm.
Used for both broker-side & producer-side compression.
It can increase performance substantially!https://t.co/oWY0alg2Zp pic.twitter.com/NyTOWKgZLu
— Stanislav Kozlovski (@BdKozlovski) July 30, 2024
In some tests, Confluent found that tweaking the compression level can lead to performance improvements of up to 156%, Jacobs said. And the default settings are still there, for those who don’t want to fiddle with these settings at all.
Likewise, more options are now available for storage, such as support for tiered storage, which now supports clusters configured with JBOD (“Just a Bunch of Disks”), allowing admins to store old data in slower, but more inexpensive object storage, and more readily-consulted data on faster SSDs on the servers.
The Consumer Rebalance Protocol has also gotten some performance adjustments as well, now in preview. The protocol was designed to spread workloads evenly across all members of a workgroup, or consumers of that group. The initial implementation took up too much computational overhead on the consumer side, and this release moves the work back to group coordinators, Jacobs explained.
Confluent and Kafka
Confluent is one of the chief contributors to the Kafka project, and offers a commercially-supported enterprise distribution as well as a cloud-based Kafka service.
Confluent will be holding its annual user Current conference Sept. 17-18 in Austin. TNS readers can get a 30% discount with the promo code NEWSTACK24.
Check the release notes for a full rundown of features of Kafka 3.8.
