Questions tagged [apache-kafka]

Apache Kafka is a distributed streaming platform designed to store and process high-throughput data streams.

0
votes
0answers
7 views

Confluent Kafka Backup and Recovery

Is there a procedure in Confluent-Kafka to take backup of Kafka broker data ? How does backup and restore work in Confluent- Kafka ? Note- The one method is to create another DC and configure inter ...
0
votes
1answer
8 views

Is state.dir directory on Kafka side or client side?

The documentation states: state.dir The state directory. Kafka Streams persists local states under the state directory. Each application has a subdirectory on its hosting machine that is located ...
0
votes
1answer
30 views

Kafka Consumer delay

I'm having the following problem: I'm recieving messages produced by the producer with a delay (somtimes up to a minute). I have no control over the producer, and I want to make sure that everythings ...
1
vote
0answers
17 views

Kafka key actors

I am trying to pin point key actors from Kafka world, with definitions as simple as possible in order to have a correct high-level overview. Hope it helps others, or help me in case I get something ...
0
votes
1answer
20 views

Kafka - New topic or Increase partition count

Say, I am having a kafka topic with 10 partitions. When a data rate is increased, I can increase the partitions to speed up my processing logic. But my doubt is that, whether increasing the ...
1
vote
2answers
17 views

Confluent - How to use external zookeeper instead of embedded zookeeper

I used to setup standalone Confluent Server with embedded Zookeeper(ZK). But now, my prod server has its own ZK cluster. So I want to use it instead of the embedded ZK in Confluent. Using ksql for ...
1
vote
0answers
20 views

How to repartition Spark DStream Kafka ConsumerRecord RDD

I am getting uneven size of Kafka topics. We want to repartition the input RDD based on some logic. But when I try to apply the repartition I am getting object not serializable (class: org.apache....
0
votes
0answers
15 views

spark streaming kafka : Unknown error fetching data for topic-partition

I'm trying to read a Kafka topic from a spark cluster using Structured Streaming API with Kafka integration in spark val sparkSession = SparkSession.builder() .master("local[*]") .appName("some-...
0
votes
0answers
25 views

Kafka Stream Consumer Group not showing offset

I have two kafka streams applications. I can see there consumer groups using: >bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092` streams-distribution-app streams-collection-...
0
votes
1answer
20 views

How to find the consumer topic and the group id from the __consumer_offsets topic in kafka?

I am trying to parse the logs from the __consumer_offsets topic in kafka. The idea is to find the group id, topic and the consumer which is creating load in my kafka cluster. The command I am ...
0
votes
1answer
9 views

Retention time of older messages in a Kafka topic after changing the configuration

I pushed some messages in a kafka-topic. I changed the retention time after that to a much longer time with : kafka-topics --zookeeper localhost:2181 --alter --topic $topic--config retention.ms=...
0
votes
0answers
16 views

Understanding Schema ID allocation in Confluent Schema Registry

I am trying to understand how globally unique UUIDs are generated for schemas in schema registry but fail to understand the following text present on this page. Schema ID allocation always happen ...
0
votes
1answer
19 views

Kafka FQDN in containers enviornment

Running kafka on a container and trying to create a new pgsql container on the same host. the pgsql container keeps exiting and the logs indicates ERROR: Failed to connect to Kafka at kafka.domain,...
0
votes
0answers
20 views

Post a message to Kafka topic using postman

I have created a topic into Kafka, And I am trying to post a message using postman but I am getting internal server error Content-type:application/json Body: { "records": [ { "key": "somekey", "...
0
votes
0answers
13 views

Kafka database connector: serializing and posting SourceRecord Avro to Rest Proxy

I am using the Debezium Kafka database connector with SQL Server. Normally, I'd send the SourceRecord instances it provides with a Producer and configure a transformer to use Avro. However, due to ...
0
votes
0answers
12 views

Is data at rest encrypted in heroku kafka add-on?

According to the article on multi-tenant kafka on heroku, data is encrypted at rest on basic plans. Is data encrypted at rest for standard and extended plans as well?
1
vote
0answers
17 views

Structured Streaming with mapGroupState causing GC and Performance Issues

In our application we are using structured streaming with MapGroupWithState in combination with read from Kafka. After starting the application, during the initial batches the performance is good, ...
0
votes
1answer
16 views

Duplicated messages reading via flink savepoints

I'm trying to use Apache Flink 1.6.0 to read some messages from a kafka topic, transform them and finally send them to another kafka topic. I use savepoints to save the state of the application in ...
0
votes
1answer
22 views

Problems with Avro deserialization in Kafka sink connectors

I'm trying to read data from DB2 using Kafka and then to write it to HDFS. I use distributed confluent platform with standard JDBC and HDFS connectors. As the HDFS connector needs to know the schema, ...
0
votes
0answers
11 views

KSQL table get old and new value

Is it possible in KSQL to stream out the old and new values from a table? We'd like to use a table as a store of values and when one changes stream out a "reversal" value which is the previous one, ...
0
votes
0answers
20 views

Kafka spring integration authorization with sasl

I am trying to connect to kafka server via spring integration module with SASL config and get error java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS ...
0
votes
0answers
14 views

Kafka Ksql Slow producer and fast consumer problem

I am stuck in a problem where my kafka is running on a different instance and my application is on different instance and they both are communicating using kafka ksql , whenever a new transaction came ...
0
votes
1answer
11 views

kafka client Communication useing log4j

I want to send log to Kafka broker using log4j. However, generic messages are well conveyed, and log4j does not. No errors are displayed. Please let me know what the problem is!! log4j2.xml <?...
1
vote
0answers
29 views

Kafka Streams with processing.guarantee set up to EXACTLY_ONCE issue

I'm working on a development environment with 3 (dockerized) kafka brokers on my system. The brokers have transaction.state.log.replication.factor set up to 3. In stream application config I set ...
0
votes
0answers
8 views

Apache Flink 1.6.0 connectivity issue with Kafka 1.1.0

I'm using a cluster of Apache Flink 1.6.0. We use FlinkKafkaConsumer011 in the job. Checkpointing is enabled (Statebackend is RocksDB, Interval 5 s, Timeout 1 minute, Minimum pause between checkpoints ...
0
votes
0answers
44 views

How to connect Elasticsearch to Kafka?

I am new in Kafka and elasticSearch, i want to push data of Elasticsearch to kafka topic, with a connector (source) I see for kafka to elasticsearch but me i want elasticsearch to kafka I make ...
1
vote
1answer
18 views

Spring KafkaEmbedded - problem consuming messages

I have problem using KafkaEmbedded from https://mvnrepository.com/artifact/org.springframework.kafka/spring-kafka-test/2.1.10.RELEASE I'm using KafkaEmbedded to create Kafka broker for testing ...
0
votes
0answers
28 views

Kafka Streams Shutdown Hook and Unexpected Exception Handling in the same Stream application

I was tasked with tearing down a Dev environment and setting it up again from scrap to verify our CI-CD processes; only problem was that I messed up creating one topic and so the Kafka Streams ...
0
votes
0answers
9 views

Kafka Streams: Global state store low level processor - process method

When does the process method gets invoked in the low level processor class override (for e.g. LoadService)? Wanted to convert the key to upper case while persisting into the state store, but it ...
0
votes
0answers
7 views

Kafka appender added in log4j2.xml for a Mule ESB appllication

"I added kafka appender in log4j2.xml in Mule ESB appllication, I am getting the below error message in the log, Any one suggest how can i proceed ? " java.lang.NoClassDefFoundError: org/apache/kafka/...
0
votes
0answers
17 views

the kafka max.poll.records does not work in spark streaming

my spark streaming version is 2.0, the kafka version is 0.10.0.1,spark-streaming-kafka-0-10_2.11. I use the direct way to get kafka records,I now want to limit the maximum number of messages I get ...
1
vote
1answer
23 views

How to discover and filter out duplicate records in Kafka Streams

Say you have a topic with a null key and the value is {id:1, name:Chris, age:99} Lets say you want to count up the number of people by name. You would do something like below: nameStream.groupBy((...
0
votes
1answer
22 views

Kafka Stream Reducer is not reducing records

Below is some sample code where we are trying to remove duplicates based on some record value (in this case id). When I publish 2 records with the same ID I receive both print statements. I was ...
0
votes
1answer
21 views

Message Queue, Kafka, Event System as “database” for an advert site? [on hold]

I'm researching with a friend about the idea of using an event queue/stream system, such as kafka or rabbitmq, as a way to store the adverts in a queue instead of a traditional database. The ...
0
votes
0answers
20 views

Clear a Kafka consumer group with stuck members

I have the problem that a lot of my Kafka clients from one consumer group did not shutdown correctly and thus the Kafka cluster thinks they are still connected. Thus, I cannot connect to the consumer ...
1
vote
1answer
32 views

Kafka Streams - one on two does not operate - wrong partition assignments

In an application of my company, in order to apply a few transformations to 2 group of messages called LIVE and PRE-MATCH, we create 2 Kafka Streams, one for each of these groups. Both of these ...
-2
votes
1answer
31 views

execute multiple command with jsch session and channelexec

I struggled to get this working but eventually got the script to execute a command (executing a sh script) on a remote unix server. I am trying to execute a second command and keep getting an error ...
-2
votes
1answer
36 views

Unable to call Kafka broker [on hold]

I am not able to call kafka broker from kafka consumer in Java. public SimpleCounterKafkaFileConsumer(){ Properties properties = new Properties(); properties.put("bootstrap.servers","...
0
votes
0answers
24 views

How to decrease the processing time and failed tasks using Apache Spark for events streaming from Apache Kafka?

I'm using Spark Streaming and Kafka, for storing parquets files on Amazon s3. My main problem is that the number of actives batches increase during the process and I just want to have only one active ...
0
votes
2answers
22 views

Missing required configuration “bootstrap.servers” error in Spark Streaming standard example

I am somewhat new to Scala and Spark, so feel free judging me, but not too hard. I am trying to launch the standard DirectKafkaWordCount example (provided with the Spark2 installation) to test how ...
0
votes
1answer
18 views

Which is the best authentication mechanism in kafka security?

Kafka has already provided different SASL authentication mechanism such as GSSAPI (Kerberos),PLAIN,SCRAM-SHA-256,SCRAM-SHA-512 and OAUTHBEARER. What is the best authentication mechanism in the above ...
0
votes
1answer
16 views

Idempotent Kafka Producer writing to multi-partitioned topic

Does an idempotent producer have to be transactional in order to ensure idempotency when publishing to a multi-partitioned topic? After reading Kafka documentation I am still unsure if it does or not. ...
0
votes
0answers
30 views

How to read data from Kafka Stream changelog topic?

I use Kafka Stream to do with my topic A, and I use inMemoryKeyValueStore. builder.addStateStore(Stores.keyValueStoreBuilder( //Stores.persistentKeyValueStore("AccurateADCounts"), ...
0
votes
3answers
42 views

Kafka-Streaming: How to collect pairs of messages and write to a new topic

This is a beginner's question to kafka-streaming. How would you collect pairs of messages using the java kafka-streaming library and write them to a new output topic? I was thinking about something ...
0
votes
1answer
15 views

Kafka streams reduce after groupby to stream sends partial reduce output on commit

We're having an issue where upon doing a groupby --> reduce --> toStream, partial reduce values are being sent downstream when a commit happens during the reduce. So if there are 65 keys to be ...
0
votes
0answers
29 views

Kafka Connect (Confluent 5.0, 4.1.2 or 3.0) not starting

we have a Kafka cluster (as a 3rd party hosted service), which has SSL enabled. We are now trying to setup Kafka Connect (Confluent 5.0) with a 3rd party Sink (WePay BigQuery connector). When starting ...
2
votes
1answer
24 views

Logstash: Kafka Output Plugin - Issues with Bootstrap_Server

I am trying to use the logstash-output-kafka in logstash: Logstash Configuration File input { stdin {} } output { kafka { topic_id => "mytopic" bootstrap_server => "[Kafka Hostname]:...
0
votes
1answer
18 views

kafka streams processor Api to deserialize a avro record

I am streaming the real time data from Kafka. But the data is in Avro format. Unable to deserialize as Json. Iam using Kafka Stream Low level Processor API. How to deserialize Avro record? def ...
-1
votes
1answer
19 views

Apache Zookeeper Issue with Changing default Port

I am trying install Zookeeper and Kafka for a basic test And have to change the Client port in Zookeeper to a port available/open like below, dataDir=/tmp/zookeeper # the port at which the clients ...
-1
votes
1answer
13 views

kafka stream with oracle database with open source

My requirement is to connect oracle database as a source with kafka connect,I am looking some opensource tool so we can connect oracle database with kafka connect ,I don't want to use golden gate and ...