Apache Kafka includes new java clients (in the org.apache.kafka.clients package). These are meant to supplant the older Scala clients, but for compatibility they will co-exist for some time. These clients are available in a separate jar with minimal dependencies, while the old Scala clients remain packaged with the server.
We recommend the new Java producer for all new development. The old Scala producers have been deprecated and will be removed in a future major release.
The new Java producer is production tested and generally faster and more fully featured than the previous Scala clients. You can use this client by adding a dependency on
the client jar using the following example maven co-ordinates (you can change the version numbers with new releases):
In the following sections we introduce new Java consumer API and the old Scala consumer APIs (both high-level ConsumerConnector and low-level SimpleConsumer).
<h4><aid="newconsumerapi"href="#newconsumerapi">2.2.1 New Consumer API</a></h4>
This new unified consumer API removes the distinction between the 0.8 high-level and low-level consumer APIs. You can use this client by adding a dependency on the client jar using the following example maven co-ordinates (you can change the version numbers with new releases):
* Create a list of message streams of type T for each topic, using the default decoder.
*/
public Map<String, List<KafkaStream<byte[], byte[]>>> createMessageStreams(Map<String, Integer> topicCountMap);
/**
* Create a list of message streams for topics matching a wildcard.
*
* @param topicFilter a TopicFilter that specifies which topics to
* subscribe to (encapsulates a whitelist or a blacklist).
* @param numStreams the number of message streams to return.
* @param keyDecoder a decoder that decodes the message key
* @param valueDecoder a decoder that decodes the message itself
* @return a list of KafkaStream. Each stream supports an
* iterator over its MessageAndMetadata elements.
*/
public <K,V> List<KafkaStream<K,V>>
createMessageStreamsByFilter(TopicFilter topicFilter, int numStreams, Decoder<K> keyDecoder, Decoder<V> valueDecoder);
/**
* Create a list of message streams for topics matching a wildcard, using the default decoder.
*/
public List<KafkaStream<byte[], byte[]>> createMessageStreamsByFilter(TopicFilter topicFilter, int numStreams);
/**
* Create a list of message streams for topics matching a wildcard, using the default decoder, with one stream.
*/
public List<KafkaStream<byte[], byte[]>> createMessageStreamsByFilter(TopicFilter topicFilter);
/**
* Commit the offsets of all topic/partitions connected by this connector.
*/
public void commitOffsets();
/**
* Shut down the connector
*/
public void shutdown();
}
</pre>
You can follow
<ahref="https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example"title="Kafka 0.8 consumer example">this example</a> to learn how to use the high level consumer api.
For most applications, the new Java Consumer API is the best option and it's the API we intend to support going forward. However, if you need to use the SimpleConsumer API, the logic will be a bit more complicated and you can follow the example <ahref="https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example"title="Kafka 0.8 SimpleConsumer example">here</a>.
As of the 0.10.0 release we have added a stream processing engine to Apache Kafka called Kafka Streams, which is a client library that lets users implement their own stream processing applications for data stored in Kafka topics.
You can use Kafka Streams from within your Java applications by adding a dependency on the kafka-streams jar using the following maven co-ordinates:
