Programate algo - Programming Blog/Blog de Programación

Spanish version / Versión en Español A single consumer, having to receive and process messages in a single thread, can soon become a bottleneck, especially as producers, topics and partitions grow in number. Kafka overcomes this limitation by allowing consumers to share their load as evenly as possible by setting the group.id option on creation, and thus belonging to a consumer group. When a bunch of consumers set the same group id, a broker inside the cluster is automatically assigned as the group coordinator: his goal will be to monitor the consumers and keep track of their membership. The group coordinator also communicates with Zookeeper and the cluster controller/leader node in order to assign partitions to each consumer in the group. The ideal situation is to have consumer and partitions evenly matched, i.e. each consumer will read only one partition from the topic. Each consumer regularly sends a heartbeat to the cluster (with a frequency defined by t...

Spanish version / Versión en Español A basic Kafka consumer application, using the same library used for producers in part 3 , could be: public class KafkaConsumerApp{ public static void main([]String args) { Properties props = new Properties(); props.put("bootstrap.servers", "BROKER-1:9092, BROKER-2:9093"); props.put("key.deserializer", "org.apache.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.common.serialization.StringDeserializer"); KafkaConsumer myConsumer = new KafkaConsumer(props); myConsumer.subscribe(Arrays.asList("my-topic")); try{ while (true) { ConsumerRecords<String, String> records = myConsumer.poll(100); processRecords(records); } } catch (Exception ex) { ex.printStackTrace(); } finally { myCon...

Spanish version / Versión en Español All code examples in this post will be in Java , for simplicity; however, keep in mind that Kafka offers client libraries in a myriad of languages. More specifically, in a Maven project, the Kafka dependency can be defined by: groupId: org.apache.kafka artifactId: kafka-clients version: 0.10.0.1 (or newer if available) So, to create a producer, some basic properties are required (although there are many, many more optional properties to tweak): Properties props = new Properties(); props.put("bootstrap.servers", "BROKER-1:9092, BROKER-2:9093"); props.put("key.serializer", "org.apache.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.common.serialization.StringSerializer"); The bootstrap.servers property defines a list of brokers in the cluster the producer can connect to. It doesn't need to be a full list of the cluster...

Versión en inglés / English version Por simplicidad, todos los ejemplos de código del post serán en Java . Sin embargo, es importante considerar que Kafka ofrece bibliotecas cliente en varios otros lenguajes. Volviendo a Java, en un proyecto gestionado por Maven , la dependencia con Kafka puede estar dada por: groupId: org.apache.kafka artifactId: kafka-clients version: 0.10.0.1 (o más reciente) Con la dependencia agregada, para crear un productor, será necesario agregar algunas propiedades básicas al proyecto. Hay muchas, muchas más; se verán las básicas por ahora: Properties props = new Properties(); props.put("bootstrap.servers", "BROKER-1:9092, BROKER-2:9093"); props.put("key.serializer", "org.apache.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.common.serialization.StringSerializer"); La propiedad bootstrap.servers define una lista de brokers del cluste...

Spanish version / Versión en Español First of all, what is Apache Kafka ? An initial definition for it is "a high-throughput distributed messaging system". Breaking this down a bit: High throughput: One of the main goals of Kafka is to process as much messages per unit of time as possible, in a scalable and fault-tolerant way. For example, LinkedIn , which is where Kafka originated, needs to achieve throughputs in the order of 20 million messages per second and 3 GB of data per second. Distributed messaging system: In a generic distributed system , networked computers work together for a common goal. In Kafka's case, that goal is to send data from points A 1 , A 2 , ... A N , to B 1 , B 2 , ... B M . A common benefit of this kind of system, which is crucial to Kafka's success, is horizontal scalability : the ability to handle higher and higher volumes of data by adding nodes to the system (not to be confused with  vertical scalability , whi...