Questions tagged [confluent-kafka]

A family of Apache Kafka clients for different languages by Confluent, all based on libkafka. Add a language tag to specify the exact client.

0
votes
0answers
7 views

Confluent Kafka Backup and Recovery

Is there a procedure in Confluent-Kafka to take backup of Kafka broker data ? How does backup and restore work in Confluent- Kafka ? Note- The one method is to create another DC and configure inter ...
0
votes
2answers
34 views

confluent_kafka consumer offset count reset problem

Description I've been trying to test the correctness of a chunk of data I send to Kafka. When I was trying to use multiprocessing in fabric, I messed up the process as well as the message consumer. ...
0
votes
1answer
55 views

Create multiple consumers for same topic in kafka

I am newbie and can see one example with one consumer in below github repository, but any ideas how to create multiple consumers for same topic in go lang? https://github.com/confluentinc/confluent-...
1
vote
1answer
46 views

Don't read next Kafka message until current message is processed?

When consuming messages using the Confluent Kafka C# library, I only want to see a new message when I've finished processing the current one (I need to read the same message again if something fails). ...
0
votes
1answer
28 views

How to push notify consumers from kafka message bus

I am writing microservice application using C# asp.net core 2.0, Kafka, CQRS and Confluent Kafka .net library. Just wonder if kafka can notify the consumers rather than the consumers poll/consume ...
0
votes
1answer
35 views

GenericRecord to SpecificRecord object using Avro

I use Confluent.Kafka.Avro for serialization between producer and consumer for Apache Kafka. From producer I use specific record, but on consumer side I want consume all records as generic and then ...
0
votes
1answer
121 views

how to install confluent-Kafka (python) on windows 10

how to install confluent-Kafka Python in windows 10 system along with librdkafka. I am not able to perform pip install confluent-Kafka in windows it is throwing error that it requires Ms c++ build ...
0
votes
1answer
80 views

Does Kafka python API support stream processing?

I have used Kafka Streams in Java. I could not find similar API in python. Do Apache Kafka support stream processing in python?
0
votes
1answer
102 views

Apache Kafka Consumer not receiving messages in Order

I am trying a POC for Kafka in my project and created two console apps in .net core 2.1 using Confluent.kafka library. I have installed Kafka on Ubuntu box and it is running properly. When i am ...
1
vote
0answers
27 views

Duplicate symbol errors when running go run main.go when using Kafka with librdkafka

The problem When I issue a go run main.go I get many duplicate symbol errors like duplicate symbol __cgo_c9759b266284_Cfunc_rd_kafka_produce_batch in: /var/folders/9x/...
2
votes
0answers
40 views

Confluent Monitoring Interceptors inside Docker container for confluent-kafka-dotnet

I try to run example app using confluent-kafka-dotnet inside Docker container, and it works fine with producer and consumer. But to get stream processor in confluent control center, need setup ...
0
votes
1answer
34 views

Apache Kafka: Programmatic monitoring of consumer groups

I would like to monitor the consumer groups (Kafka based groups not ZK) in my Kafka cluster from within a Python application (I am open to using either of the Kafka-python library and the confluent ...
1
vote
0answers
17 views

python Testing confluent_kafka Consumer [duplicate]

I'm trying to test following piece of code using pytest: self.consumer = Consumer({ 'group.id': config['KAFKA_GROUP_ID'], 'bootstrap.servers': config['KAFKA_BROKERS'], ...
3
votes
1answer
88 views

How to consume from Kafka Spring Cloud Stream by default and also consume a Kafka message generated by the confluent API?

I am building a microservice component which will consume by default Spring Cloud Stream (SCS) Kafka messages generated by other (SCS) components. But I also have a requirement to consume Kafka ...
0
votes
0answers
28 views

Kafka connector is not pulling the data from PostgreSQL when mode is timestamp which is not in UTC format

By using Kafka connector, I'm unable to pull the data from PostgreSQL when mode = timestamp and the timestamp column in psql is in default timestamp. But when i change my timestamp field to 'UTC' ...
0
votes
1answer
81 views

Caused by: java.lang.ClassNotFoundException: io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor

Trying to publish message to kafka topic using rest proxy by Confluent platform using this command and responds with an error as mentioned below Request: $ curl -X POST -H "Content-Type: application/...
0
votes
1answer
41 views

How to create Transform in Memsql when source is Kafka Avro Format

I am able to push data from Kafka to Memsql. I am trying to push using Transform. I have created Kafka Consumer in Python which is consuming data from Kafka Topic and converting to Json Format. I ...
2
votes
1answer
86 views

C# Unable to consume message on Kafka topic?

I've been looking through several examples of the Confluent.Kafka client (https://github.com/confluentinc/confluent-kafka-dotnet/), and whilst I can successfully get a producer to push a message into ...
0
votes
2answers
150 views

java.lang.ClassNotFoundException: MonitoringProducerInterceptor

I am exploring confluent Kafka and following this quickstart guide from confluent. I am getting java.lang.ClassNotFoundException: io.confluent.monitoring.clients.interceptor....
0
votes
0answers
20 views

Check the Offset of a Kafka Consumer using Confluent

I am using confluent to interact with Kafka in my c# application. I need to be able to check the offset of a consumer in order to ensure that it is processing values and is not hung up. How can I ...
0
votes
0answers
28 views

confluent kafka default kafka brokers logging

By default, confluent kafka log4j properties come with DailyRollingFileAppender. Is there any reason to have this as the default logging mechanism. Schema registry uses RollingFileAppender. I am ...
0
votes
0answers
14 views

Configuring commit interval for Confluent JDBC connector

In my app I'd like to configure the sink connecor to commit offsets more frequently. But it seems there is no configuration property to set this parameter explicitly. How can I do this?
0
votes
0answers
60 views

Read Kafka topic in multiple of offset

Assume I have a topic(t) in single Kafka partition, and a single consumer in consumer-group(X) is consuming topic t. Also, assume message ordering, no partition switching. Now I want to read the ...
1
vote
0answers
62 views

Kafka Stream Chained LeftJoin - Processing previous old message again after the new one

I have a stream that is a composite of other streams final KTable<Long, CompositeInfo> compositeInfoTable = compositeImcTable.leftJoin(compositeFundTable, ...
0
votes
0answers
13 views

Session Windows in Kstreams does not work as expected with Exactly once semantics

I am new to Kafka. I have created a Spring Boot, Spring Kafka/KStream consumer which uses SessionWindows to aggregate data. Now it works like its supposed to according to the documentation until I ...
0
votes
2answers
63 views

Kafka ACLs - unable to describe group or view offsets in Kafka Tool after setting ACLs

Using Kafka version 1.1 I´ve enabled SASL on one port for clients while moving the interbroker communication to other port with PLAINTEXT. Problem is that I am having troubles discovering info about ...
1
vote
1answer
44 views

Kafka Streaming Reset Issues

I have been attempting to build a Kafka Streaming application for use with Spark. I have a static dataset for testing. After running my code once through, Kafka sets the current offset such that I ...
0
votes
1answer
191 views

Kafka Configuration

Sorry i am new in Kafka and this question migth be so easy but i need some help. i did not figure out some configurations. There is a stream data, i want Consumers to see only last 5 minutes of ...
0
votes
1answer
45 views

Key and value avro messages distribution in Kafka topic partitions

We use kafka topic with 6 partitions and the incoming messages from producers have 4 keys key1,key2,key3,key4 and their corresponding values, I see that the values are distributed only with 3 ...
0
votes
0answers
28 views

EmbeddedSingleNodeKafkaCluster carries over topic between junit tests

I am using EmbeddedSingleNodeKafkaCluster (code - v4.1.1) and related classes for my junit tests. It used to work fine, till I introduced a unit test that is testing forward compatibility of a message ...
0
votes
1answer
29 views

KSQL Steam-table Left join partially (and silently) fails to populate values

I am joining a stream with a table. The result of the join is only partially successful. Some of the values are accurately populated while others are null. I checked to make sure that the values exist ...
0
votes
1answer
40 views

Unable to establish Kafka source connection with azure IoT hub

I have been trying to resolve this issue from past 5 days till today. I have azure Iot Hub up and running and many devices are able to connect and send messages to it. Meanwhile I was exploring ...
0
votes
0answers
102 views

AWS Toolkit for Visual Studio bug? AWS C# lambda failed to load the librdkafka native library of Kafka?

The following .Net core 2.1 lambda application with nuget package of Confluent.Kafka is created in Visual Studio with AWS toolkit for Visual Studio 2017. However, the following code using (var ...
0
votes
0answers
22 views

java.sql.SQLException: Result consisted of more than one row causes Kafka connector failure

I have a Kafka JDBC sink connector (from Kafka to mariaDB) with the following configuration: { "name": "sink-connector", "config": { "connector.class":"io.confluent....
1
vote
0answers
29 views

Confluent go kafka library starting from earliest offsets on restart

We currently use Kafka 0.8.2 for our brokers. We use the .Poll() method to grab messages and commit after we collect 1000 messages. We are able to consume from the cluster fine using this library, and ...
0
votes
1answer
41 views

Exactly once in confluent_kafka v0.11.2 for Python

How do I enable.idempotency or change the processing.guarantee to exactly_once in python3.6 for confluent kafka v0.11.4. A lot of documentation from confluent kafka only talks about Scala and Java. I ...
4
votes
0answers
114 views

Kafka SASL handshake takes too long

Description: authentication using SASL/SCRAM or SASL/PLAINTEXT takes around 9 seconds to complete. Is this normal? How to reproduce: One Kafka broker instance (v1.1.0) One C# producer (Confluent ...
0
votes
2answers
55 views

Confluent Kafka dot net ProduceAsync tasks

var msgs = new List<string> {“msg1”, “msg2”, “msg3”}; var tasks = new List<Task>(); Foreach(var msg in msgs) { tasks.add(_producer.ProduceAsync(...)); } var deliveryReports = ...
0
votes
1answer
196 views

Confluent Kafka Connect Docker Container Issue

I'm using the following docker compose snippet: connect: image: confluentinc/cp-kafka-connect:latest hostname: connect container_name: connect depends_on: - zookeeper - ...
0
votes
0answers
84 views

Kafka JDBC sink connector is slow

I have configured various JDBC sink connectors in order to stream data from kafka topics into MySQL tables. I noticed that once I send the POST request to Kafka Connect, some of the rows (about 1/3) ...
0
votes
1answer
103 views

How to install librdkafka with Kerberos?

So I am trying to write a script where the data comes from a database and then gets published into a kafka topic. I am using the confluent-kafka due to its speed. The kafka has kerberos configured ...
0
votes
0answers
30 views

Spark Avro producer isssue to publish to Confluent kafka topic issue

I have an isssue publish kafka message in avro message format from Spark job on Mapr cluster. I can able to publish avro message from a normal java process, But Spark Job is unable to detect the “io....
0
votes
0answers
53 views

Avoiding small files from Kafka connect using HDFS connector sink in distributed mode

We have a topic with messages at the rate of 1msg per second with 3 partitions and I am using HDFS connector to write the data to HDFS as AVRo format(default), it generates files with size in KBS,So I ...
0
votes
1answer
72 views

get task id's from kafka connect API to print in logs

I have a kafka connect sink code for which below json is passed as curl command to register tasks. Please let me know if anyone has any idea on how to get the task id's of my connect. For example in ...
1
vote
0answers
100 views

How do I configure a producer and consumer in Kafka REST Proxy?

How do I set up a producer and consumer in Kafka REST Proxy? If i use kafka-clients.jar and implement it in java, i can easily make all the settings. acks, commit method, and so on. I know that the ...
0
votes
1answer
149 views

Confluent's Kafka REST Proxy vs Kafka Client

I am curious about the advantages and disadvantages of Confluent's Kafka REST Proxy and the producer/consumer implemented with the kafka official client library. i know that Confluent's Kafka REST ...
0
votes
0answers
46 views

adding kafka connect jdbc driver

I'm trying to access a SAP Advantage DB with kafka connect using JDBC. I'm using the docker container, and I have added the jdbc driver Jar FROM: http://devzone.advantagedatabase.com/dz/content.aspx?...
0
votes
1answer
131 views

What are the benefits of the Kafka REST Proxy API?

I do not know the advantages of the Kafka REST Proxy API. It's a REST API, so I know it's handy for administration. Why do people use the Kafka REST Proxy API? Is it burdensome to add a Maven ...
0
votes
0answers
53 views

Microsoft CA Authority certificate bad OID

First of all, be kind. I'm a newbie and this is my first post here. I'm trying to use Microsoft CA Authority signed certificate with opensource Kafka Confluent. And I'm having trouble with Microsoft'...
0
votes
0answers
248 views

Kafka Broker: too many reads + high CPU usage

I have a Confluent Kafka 1 cluster with three brokers. I'll give you a bit of context first: After a Kafka broker hard restart indexes got corrupted, so I deleted them all and left the broker ...