Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

0
votes
0answers
7 views

Confluent Kafka Backup and Recovery

Is there a procedure in Confluent-Kafka to take backup of Kafka broker data ? How does backup and restore work in Confluent- Kafka ? Note- The one method is to create another DC and configure inter ...
0
votes
0answers
7 views

Kafka appender added in log4j2.xml for a Mule ESB appllication

"I added kafka appender in log4j2.xml in Mule ESB appllication, I am getting the below error message in the log, Any one suggest how can i proceed ? " java.lang.NoClassDefFoundError: org/apache/kafka/...
0
votes
1answer
28 views

pushing big files into Kafka topic through producer

Files are uploading through client on server folder. it is continuously uploading files from different client using TUS API(File resumeabl upload) . size of files are from 5 mb to 100 mb. Now i want ...
0
votes
1answer
55 views

Create multiple consumers for same topic in kafka

I am newbie and can see one example with one consumer in below github repository, but any ideas how to create multiple consumers for same topic in go lang? https://github.com/confluentinc/confluent-...
0
votes
0answers
9 views

Broker not available (sendRequest) after producer ready. Only sends messages within the ready event

My producer only sends messages on ready event.When trying to send a new message, I get a BrokerNotAvailableError { message: 'Broker not available (sendRequest)' Environment Node version: 8.12.0 ...
0
votes
1answer
23 views

How to send Custom Object to Kafka Topic with Producer

I would like to send my Account class with Producer to my Kafka Topic, then I will aggregate with Kafka Stream. However, I can not send Object I am getting error : Caused by: org.apache.kafka.common....
0
votes
1answer
37 views

Decoupling Kafka from an Application

I have an application that receives large amount of GET requests (around 250000 in 5 mins). The application parses the query params and publishes to Kafka. The code to publish is as follows: public ...
0
votes
1answer
26 views

generate json with backslash using jackson

. I am using Jackson library to generate json. Jackson is generating the json in below format. {"color":"yellow","type":"renault"} I want to generate the above format with backslash as below: {\"...
0
votes
2answers
43 views

Read a CSV file in real time using Kafka Connect

how can I integrate Kafka connect with Kafka producer in java so that I can read a CSV file in real time? I am having a hard time finding any source related to this. Right now I am using scala-ide to ...
0
votes
0answers
28 views

Kafka Producer Client API can not asynchronous sending when the target broker using SASL/PLAIN

I had a simple demo to transfer data from one kafka cluster which did not use SASL to another kafka cluster which did use SASL/PLAIN. and the codes look like this: Properties consumerProps = new ...
0
votes
1answer
26 views

In schema registry, consumer's schema could differ from the producer's, what actually it means

While producing AVRO data to Kafka, Avro serializer writing the same schema ID in the byte array which is used while writing the data. Kafka Consumer fetches the schema from Schema Registry based on ...
0
votes
1answer
26 views

Event based commit in transaction KafkaTemplate using KafkaTransactionManager

Spring managed KafkaTemplate provides template.send(record).addCallback(... template.executeInTransaction(... Now let's say I have a method doWork() which is triggered on a event (say a TCP/IP ...
0
votes
1answer
52 views

No output to Kafka topic: Spark Structured Streaming and Kafka Integration

I am trying to send stream output from Apache Spark 2.3.1 to Apache Kafka using kafka sink: import org.apache.spark.sql.SparkSession import org.apache.spark.sql.types.StructType import org.apache....
0
votes
0answers
19 views

DataSet to Kafka Using Flink? Is it possible

I have a usecase where i need to move records from hive to kafka. I couldn't find a way where i can directly add a kafka sink to flink dataset. Hence i used a workaround where i call the map ...
0
votes
0answers
29 views

Kafka Producer Warnings WARN [kafka-producer-network-thread ]

I am seeing below WARN in my Kafka producer. WARN [kafka-producer-network-thread | producer-4] o.a.k.c.NetworkClient [NetworkClient.java:588] Connection to node -1 could not be established. Broker ...
1
vote
0answers
57 views

Spring boot controller handle Java future

I have a normal spring boot application with controller and service like following: @RestController class EventController(private val service: Service) { @GetMapping(value = ["xxx"], provides = [...
0
votes
0answers
70 views

Flink - Uncaught exception in kafka-producer-network-thread

I'm creating a streaming job that reads two kafka topics and writes in one kafka topic. I'm working with these versions : flink 1.4.1, kafka_2.11-1.0.0 and flink-connector-kafka-0.11_2.11. Sometimes (...
1
vote
0answers
34 views

Can I use the same keystore for multiple kafka producer/consumer

I want to use SSL encryption and authentication, but we have many clients and we don't want to generate new keystore for each client, is it possible to use the same keystore for each clients and if ...
0
votes
1answer
19 views

Error in broker list argument in Kafka Producer

I started a docker container from Wurstmeister Kafka with the command: docker run -ti --name sender --entrypoint /bin/bash wurstmeister/kafka:latest Then i use the command: docker exec -ti <...
1
vote
1answer
31 views

Failed to send SSL Close message when using KafkaPublish processor to write to kafka topic

I am using KafkaPublish processor to publish messages to a kafka topic secured using SASL_SSL security protocol (we are not using Kerberos). I get the below error: WARN Failed to send SSL Close ...
1
vote
1answer
53 views

Is there any way to ensure that duplicate records are not inserted in kafka topic?

I have been trying to implement a queuing mechanism using kafka where I want to ensure that duplicate records are not inserted into topic created. I found that iteration is possible in consumer. Is ...
0
votes
1answer
46 views

Java/Scala Kafka Producer does not send message to topic

I'm having a problem with sending a serialized XML to my Kafka topic. Whenever I run my code, I don't get any exceptions or error message, but still I can't see any of my messages in the Kafka-topic. ...
1
vote
1answer
37 views

How to pause/start/stop Kafka Producer / Kafka Template

I'm using a spring boot app with kafka integration and I want to implement an endpoint to stop and start kafka from publishing messages. The message are triggered in a async way by another endpoints. ...
0
votes
1answer
18 views

Kafka Producer Slows down when putting different keys

I have a code that will send data to Kafka topic. public void sendMessage(String message, String key){ if (isAsync) { // Send asynchronously producer.send(new ProducerRecord<String, ...
0
votes
3answers
88 views

push multiple Line Text as one message in a kafka topic

I want to push a text consisting of multiple lines as one message into a kafka topic. After I enter: kafka-console-producer --broker-list localhost:9092 --topic myTopic and copy my text: My Text ...
0
votes
1answer
34 views

SalesForce's URL for Kafka provider

I'm trying to determine the URL for SalesForce as it relates to setting up a Kafka Provider. I'm using Bayeux client that needs a URL to SalesForce for connection: new KafkaOptions(new Uri("")); ...
0
votes
0answers
9 views

Pykafka 2.3.1 Producer Hang

We are using pykafka 2.3.1 and see producer hang every once a while. The error message is: Message not delivered!! UnknownTopicOrPartition('Produce request for topicname/0 to {ip:port} failed ...
2
votes
2answers
66 views

java maven project build failure

i have created a java project to push data to kafka server. and this java maven project is error free. But when i try to maven-> install , build is getting failed. Getting error as below: Failed to ...
0
votes
0answers
41 views

How to boost Kafk's consumer partition reassignment time

We are using Kafka 1.11 our system challenge is to keep low latency on non-down time SLA We use kafka as our message broker and during our deployments we got latency spikes. A - producer to topic ...
1
vote
1answer
45 views

Kafka Producer deployed on Kubernetes not able to produce to Kafka cluster running on local machine

I have a Kafka cluster running on the local machine with default settings outside of my minikube setup. I have created a producer in one of my web services and deployed it on minikube. For producer ...
0
votes
1answer
39 views

Kafka large message configuration

I read many topics about configuration and I still don't understand. when I do: ../kafka_2.11-1.1.0/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my_topic --config max.message.bytes=...
1
vote
2answers
51 views

Flink kafka - Flink job not sending messages to different partitions

I have the below configuration: One kafka topic with 2 partitions One zookeeper instance One kafka instance Two consumers with same group id Flink job snippet: speStream.addSink(new ...
2
votes
0answers
49 views

Client liberary Producer to send Message to KAFKA broker in .NET 4.0

I am building a project where Project Target Frame work is .NET Framework 4.0, i cannot go with .NET FrameWork 4.5. Can you suggest me any KAFKA client library for publishing message to KAKFA broker ...
0
votes
0answers
30 views

What is the best practice to connect an incoming web socket to Kafka?

I'm new to message queues in general and at the moment I'm working with apache Kafka. I have a python script that takes data from a crypto currency exchange and sends it to one of the Kafka topics. ...
2
votes
1answer
49 views

Which Kafka producer property to use when I no longer want to receive data after certain threshold?

Which Kafka producer property to use to achieve the following? I am using UDP Kafka Bridge which sends the message from UDP port to Kafka topic. If the memory on the Kafka producer exceeds certain ...
0
votes
1answer
41 views

Producing a Kafka message with a Null Value (Tombstone) from the Console

Is there any way to produce a message in the kafka-console-producer with a null value (ie. mark it for the compactor to delete it with a tombstone)? I've tried producing "mykey" and "mykey|". The ...
0
votes
0answers
40 views

Kafka throttle authentication requests over SSL

We have a scenario where a bunch of producers come online at the exact same time. These use librdkafka. We see that the flood of requests make Kafka open a bunch of file descriptors and eventually run ...
0
votes
0answers
15 views

Kafka : Co-relation between Producer Metrics BytesInPerSec and MessagesInPerSec

I would like to understand if there is any Co-relation between Kafka Producer Metrics, BytesInPerSec and MessagesInPerSec. I have gone through a lot of Kafka Documentation, however, nothing concrete ...
0
votes
2answers
81 views

Applying Multiple Filters + Write to Multiple Topics in a Loop on Kafka Streams

I have a requirement where I have a list of filters (where schema_field='val') and corresponding topics. I need to iterate over those list of filters and apply them, then write the filtered record ...
1
vote
1answer
25 views

Kafka messages not accepted by server (only from remote IP?)

My Kafka server works fine, if I produce and consume from a C++ application, or from the command line, from the local machine. But it does not work well from an external IP address: the topic gets ...
2
votes
1answer
65 views

Metadata information from kafka

I am new to Confluent/Kafka and I want to find metadata information from kafka I want to know list of producers list of topics schema information for topic Confluent version is 5.0 What are ...
0
votes
1answer
76 views

KAFKA : org.apache.kafka.common.errors.TimeoutException

I am facing issues while publishing messages in topic. With the same configuration, sometime it works and sometimes it gives timeout error. Below is the java error ***Record*** : Key1 => Value1 ...
0
votes
1answer
62 views

kafka IllegalStateError: Producer is closed forcefully

I have the following code : __kafka_client = KafkaProducer(value_serializer=lambda m: json.dumps(m).encode('ascii')) for(msg in messages): __kafka_client.send('TOPIC', dumps(msg))....
0
votes
1answer
102 views

Apache Kafka Consumer not receiving messages in Order

I am trying a POC for Kafka in my project and created two console apps in .net core 2.1 using Confluent.kafka library. I have installed Kafka on Ubuntu box and it is running properly. When i am ...
0
votes
1answer
40 views

Data ingestion using kafka from crawlers

I am trying to work with Kafka for data ingestion but being new to this, i am kind of pretty much confused.I have multiple crawlers, who extract data for me from web platform. Now, the issue is i ...
1
vote
1answer
49 views

reading data from Kafka and storing it in dynamo db

I need to read data from multiple topics in Kafka broker and store the data in Dynamo DB. Any reference code or any specific method i can go ahead with. I Tried using https://github.com/shikhar/kafka-...
0
votes
1answer
66 views

Confused about preventing duplicates with new Kafka idempotent producer API

My app has 5+ consumers consuming off of five partitions on a kafka topic.(using kafka version 11) My consumer's each produce a message to another topic then save some state to the database, then do a ...
1
vote
1answer
63 views

Is there a way to write a message in kafka with some timeout per message

I am new to kafka and I know that we can give e.g. delete.retention.ms on topic creation, and all messages that match to this criterion will be deleted. I would like to know whether there's a way to ...
0
votes
2answers
67 views

Unable to send GenericRecord data from Kafka Producer in AVRO format

Using confluent-oss-5.0.0-2.11 My Kafka Producer code is public class AvroProducer { public static void main(String[] args) throws ExecutionException, InterruptedException { Properties ...
1
vote
2answers
87 views

Not able to send large messages to Kafka

I want to send a large message from producer to Kafka so I've changed below properties. Broker (server.properties) replica.fetch.max.bytes=317344026 message.max.bytes=317344026 max.message.bytes=...