Programate algo - Programming Blog/Blog de Programación

Spanish version / Versión en Español A basic Kafka consumer application, using the same library used for producers in part 3 , could be: public class KafkaConsumerApp{ public static void main([]String args) { Properties props = new Properties(); props.put("bootstrap.servers", "BROKER-1:9092, BROKER-2:9093"); props.put("key.deserializer", "org.apache.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.common.serialization.StringDeserializer"); KafkaConsumer myConsumer = new KafkaConsumer(props); myConsumer.subscribe(Arrays.asList("my-topic")); try{ while (true) { ConsumerRecords<String, String> records = myConsumer.poll(100); processRecords(records); } } catch (Exception ex) { ex.printStackTrace(); } finally { myCon...

Spanish version / Versión en Español All code examples in this post will be in Java , for simplicity; however, keep in mind that Kafka offers client libraries in a myriad of languages. More specifically, in a Maven project, the Kafka dependency can be defined by: groupId: org.apache.kafka artifactId: kafka-clients version: 0.10.0.1 (or newer if available) So, to create a producer, some basic properties are required (although there are many, many more optional properties to tweak): Properties props = new Properties(); props.put("bootstrap.servers", "BROKER-1:9092, BROKER-2:9093"); props.put("key.serializer", "org.apache.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.common.serialization.StringSerializer"); The bootstrap.servers property defines a list of brokers in the cluster the producer can connect to. It doesn't need to be a full list of the cluster...

Topics In Kafka, a topic is a named feed, or a category of messages. Producers send messages to a specific topic, addressable by its name, and consumers can read messages from that topic, also addressing it by name (all of this in the context of a particular Kafka cluster). This is how a topic is understood as a logical entity: the way it's stored and handled inside the cluster doesn't matter to producers and consumers; they just want to send and read data. Kafka topics have the following essential characteristics: Order : Kafka topics are a time-ordered sequence of messages. Messages are saved in the topic in the order in which they are received. Immutability : Once a message has been saved to a topic, it cannot be modified, or deleted: topics are append-only structures (because of the time order). If a producer sends invalid data in a message, it will be up to him to notify the consumer and send the corrected data in a new message, and the consumer will have...