Questions tagged [confluent]

Confluent Inc. is the Palo Alto-based software company founded by the team that built Apache Kafka at LinkedIn and that scaled Kafka to accommodate over 1 trillion messages per day. Kafka has since become the foundation of the data infrastructures of thousands of companies such as Netflix, Paypal, ...

0
votes
0answers
7 views

Confluent Kafka Backup and Recovery

Is there a procedure in Confluent-Kafka to take backup of Kafka broker data ? How does backup and restore work in Confluent- Kafka ? Note- The one method is to create another DC and configure inter ...
1
vote
2answers
17 views

Confluent - How to use external zookeeper instead of embedded zookeeper

I used to setup standalone Confluent Server with embedded Zookeeper(ZK). But now, my prod server has its own ZK cluster. So I want to use it instead of the embedded ZK in Confluent. Using ksql for ...
0
votes
0answers
16 views

Understanding Schema ID allocation in Confluent Schema Registry

I am trying to understand how globally unique UUIDs are generated for schemas in schema registry but fail to understand the following text present on this page. Schema ID allocation always happen ...
0
votes
1answer
22 views

Problems with Avro deserialization in Kafka sink connectors

I'm trying to read data from DB2 using Kafka and then to write it to HDFS. I use distributed confluent platform with standard JDBC and HDFS connectors. As the HDFS connector needs to know the schema, ...
0
votes
0answers
44 views

How to connect Elasticsearch to Kafka?

I am new in Kafka and elasticSearch, i want to push data of Elasticsearch to kafka topic, with a connector (source) I see for kafka to elasticsearch but me i want elasticsearch to kafka I make ...
0
votes
0answers
29 views

Kafka Connect (Confluent 5.0, 4.1.2 or 3.0) not starting

we have a Kafka cluster (as a 3rd party hosted service), which has SSL enabled. We are now trying to setup Kafka Connect (Confluent 5.0) with a 3rd party Sink (WePay BigQuery connector). When starting ...
0
votes
1answer
28 views

Setup multiple kafka connect sinks

I am working on streaming the data from postgreSQL to HDFS. I had setup confluent environment on HDP 2.6 sandbox. My jdbc source configs for postgreSQL are name=jdbc_1 connector.class=io.confluent....
0
votes
2answers
22 views

KSQL : Left Join with Not Equal where condition is not satisfying the result

Please find the below issue and confirm for the same. Step-01 : Based on the joining condition , getting the value from both table and populating. As no matching value in table B, all columns are ...
0
votes
0answers
17 views

Kafka Connect sink to Redshift table not in public schema

I am unable to make a Kafka Connect sink work for a table that is not in the public schema. I am using Kafka Connect to send records to a Redshift database via a sink operation using ...
0
votes
0answers
15 views

Confluent Kafka-connect-JDBC connector showing hexa decimal data in the kafka topic

I'm trying to copy the data from a table in the oracle db and trying to put that data in a kafka topic. I've used the following JDBC source connector for that : name=JDBC-DB-source connector.class=io....
0
votes
0answers
18 views

Is CachedSchemaRegistryClient thread safe?

As far as I can tell by looking at the code, this class seems thread safe, with synchronized in the right place. But there's no comments/javadoc so it's not very explicit. I'm wondering if it's safe ...
0
votes
0answers
47 views

Kafka Connect S3 - get the header from the message

use-case : Consume the messages sent to a topic and store in AWS S3 im using Kafka-s3-connector to achieve this , its working perfectly fine but Each file is encoded as ...
1
vote
1answer
43 views

confluent - kafka-connect - JDBC source connector - ORA-00933: SQL command not properly ended

I've the following sql query in my kafka jdbc source connector properties file : query=SELECT * FROM JENNY.WORKFLOW where ID = '565231' If I run the same query in sql developer, it works fine and ...
0
votes
2answers
19 views

Print Data from Confluent Source and Sink connectors

i have source and sink connectors installed using confluent and they are working fine. but when i see docker logs using docker logs -f container-name the output is something like this [2018-09-19 ...
0
votes
2answers
27 views

Unknown magic byte with kafka-avro-console-consumer

I have been trying to connect with kafka-avro-console-consumer from Confluent to our legacy Kafka cluster, which was deployed without Confluent Schema Registry. I provided schema explicitly using ...
0
votes
0answers
22 views

confluent JDBC-Sink Examples.table doesn't exist

I am using confluent and kafka to sink the data from a Kafka topic to a Mysql table. My configs are name=jdbc-sink connector.class=io.confluent.connect.jdbc.JdbcSinkConnector tasks.max=1 topics=...
0
votes
1answer
36 views

Null Handling in confluent KSQL

Could you please let me know how to handle Null value in KSQL . I tried to handle 4 possible ways but not being resolved. Tried with 4 Ways to Replace NULL with a different Value in KSQL but giving ...
0
votes
1answer
39 views

Confluent 5.0.0 kafka connect hdfs sink: Cannot describe the kafka connect consumer group lag after upgrade

We upgraded from Confluent 4.0.0 to 5.0.0, after upgrading we cannot list the kafka connect hdfs sink connector consumer lag. $ /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <hostname&...
0
votes
0answers
39 views

kafka-connect-hdfs : SequenceFileWriter creates badfiles on connector restarts causing EOFException in the SequenceFileReader

In Kafka connect hdfs, we have the below SequenceFileWriter.java class to write kafka messages in SequenceFileFormat. import java.io.IOException; import io.confluent.connect.avro.AvroData; import io....
0
votes
0answers
28 views

Kafka Connect - Failed to flush, timed out / Failed to commit offsets

I am getting the following error: "ERROR WorkerSourceTask(id=test-mysql-dbc-source-0) Failed to flush, timed out while waiting for producer to flush outstanding N messages. ERROR Failed to commit ...
0
votes
1answer
37 views

Confluent Control Center not intercepting stream

I'm using CCC with a Kafka stream, which is populated by the Postgres Connector from Debezium. I'm using the following docker-compose.yml: version: '2' services: zookeeper-1: image: ...
0
votes
1answer
78 views

Kafka AVRO Consumer: MySQL Decimal to Java Decimal

I'm trying to consume records from a MySQL table which contains 3 columns (Axis, Price, lastname) with their datatypes (int, decimal(14,4), varchar(50)) respectively. I inserted one record which has ...
0
votes
1answer
19 views

Not able to run Kafka on docker after creating image(build)

Dockerfile From centos:centos6 COPY ./confluent-4.1.0 /etc/confluent-4.1.0 RUN echo 'export CONFLUENT_HOME="/etc/confluent-4.1.0"' >> /root/.bashrc RUN echo 'export PATH="$PATH:$CONFLUENT_HOME/...
3
votes
1answer
50 views

Confluent:Topic was not created after loading a Connector

I tried connecting to several databases including MySQL and MSSQL and i had no problem. But when i tried to connect to a certain remote MySQL database (That can be accesses only from my company's ...
1
vote
1answer
29 views

KSQL - calculate distance from 2 messages using GEO_DISTANCE

I've a kafka topic and each message in the topic has lat/lon and event timestamp. Created a stream referring to topic and would like to calculate distance between 2 point using geo_distance . example ...
1
vote
1answer
45 views

Kafka-mqtt connector with multiple topics

We are using Kafka-Mqtt connector from Confluent team. Is there any way to publish data from multiple Mqtt topics to multiple Kafka topics with same name as of Mqtt , i.e test1 in Mqtt -> test1 in ...
0
votes
1answer
35 views

How to solve Kafka Avro deserializing problem

we have a very strange behavior with an application which used kafka avro consumer and producer. We have a server where a confluent installation is running. All is configured manually so no confluent ...
-1
votes
1answer
23 views

How to configure kafka s3 sink connector for json using its fields AND time based partitioning?

I have a json coming in like this: { "app" : "hw", "content" : "hello world", "time" : "2018-05-06 12:53:04" } I wish to push to S3 in the following file format: /upper-directory/$...
2
votes
0answers
24 views

Update ksql stream with new topic schema

I write avro messages into kafka topic using schema registry. Then created stream based on the topic. The scream created with current schema. Then I add new field to the schema. The schema register ...
0
votes
0answers
24 views

Confluent Load Connector invalid JSON - parse error: Invalid numeric literal at line 1, column 7

I am playing around with Kafka on my Mac and keep getting the same error. Hoping for a little guidance as I've had no luck with Google currently. For reference I'm following this article https://www....
0
votes
1answer
22 views

Exception Occurred Subject not found error code - Confluent

I can see an error in my logs that Subject with name A.Abc-key is not present. I listed all the subjects and verified that the A.Abc-key is not present but the A.Abc-value is present On checking ...
0
votes
1answer
35 views

Can't start Confluent Kafka Connect on Windows (NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig)

As title says, I am getting NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig when starting ElasticSearch sink connector from command line. I installed Confluent package ...
0
votes
1answer
33 views

Kafka Sink HDFS Unrecognized token

I'm trying to write JSON with Kafka HDFS Sink. I have the following properties (connect-standalone.properties): key.converter.schemas.enable = false value.converter.schemas.enable = false schemas....
0
votes
0answers
28 views

Kafka-Connect-JDBC Source connector: Unable to read rows from table with NUMERIC column

I'm trying to set-up source connector for the Adventureworks database in postgres, the table is described in the above image. The source configuration is given below. When the connector runs it is ...
0
votes
0answers
55 views

Byte formate in avro

How can I stop the conversion of number into byte formate while loading Oracle table into Kafka using confluent 5.0.0.? The conversion to byte is creating a lot of issues while reusing the data in ...
0
votes
1answer
35 views

Error creating hive table from avro schema

I am trying to create a hive table by extracting the schema from Avro data which is stored in s3. Data is stored in s3 using the s3 Kafka connector. I am publishing a simple POJO to the producer. ...
0
votes
1answer
62 views

Error trying to start zookeeper server- Confluent setup

I am trying to setup Confluent-4.1.1 on Ubuntu 16.04. To start the ZooKeeper server, I ran ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties.txt from the root directory of Confluent by ...
0
votes
1answer
32 views

Debezium Kafka connect. Decimal schema error

I'm using PostgreSQL. In my table numeric column defined like this: "net_weight" numeric(19,2) But in Kafka, I'm receiving instead of decimal numbers (E.g. 14.0) I'm getting something like "E4g=" ...
-2
votes
0answers
16 views

how to create different tables from a single topic in KSQL?

I have a Kafka topic which has customer, purchase, product etc data. I want to create different streams from this single topic.
0
votes
1answer
28 views

Kafka Connect HDFS configuration setting?

I want to configure Kafka Connect to write the maximum possible of records to HDFS (I need to write around 100B records per day, Each record is around 1.2Kb). I have Kafka server with 256GB RAM. I'm ...
0
votes
1answer
37 views

sbt resolver for confluent platform

I am unable to add confluent repo in my sbt. I looked at pom example and found definition of adding repo in maven. <repositories> <repository> <id>confluent</id> &...
0
votes
0answers
33 views

kafka-connect jdbc distributed mode

We are working on building the Kafka-connect application using JDBC source connector in increment+timestamp mode. We tried the Standalone mode and It is working as expected. Now, we would like to ...
0
votes
0answers
55 views

SerializationException: Error serializing Avro message

I'm using KStreams which have a AVRO schema and are hooked up with the schema registry. When I start processing the stream, I get a NullPointerException as follows: Caused by: org.apache.kafka.common....
0
votes
1answer
31 views

Kafka Connect “worker_id” set to loopback “127.0.1.1:8083”?

I have a three node Confluent Kafka cluster. Each node is runing zookeeper, kafka broker, schema-registry, and kafka connect. I have configured an S3 sink. It seems to be functioning, but the ...
0
votes
0answers
11 views

Confluent-Kafka “Consumer instance not found” error even-if consumer instance is not timedout

I'm observing Consumer instance not found error, at the time of consumer registration even if consumer instance is not timedout. Using Confluent API's. Following are steps followed for this negative ...
0
votes
1answer
69 views

Error when use kafka-avro-console-consumer consume avro message sent by spring cloud stream kafka

I tried official schema-registry-confluent example (consumer/producer) with my local installed confluent 4.0.0, it can send "sensor" avro message when send post request and receive in listener, but ...
0
votes
1answer
30 views

Confluent Platform and java.nio.file.DirectoryNotEmptyException

I use All-In-One Confluent Platform https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html I performed the steps described in the documentation above and was able to run Confluent ...
1
vote
1answer
35 views

Confluent Platform unable to properly send/consume message into the Kafka topic

Currently, I use Maven with io.fabric8 docker-maven-plugin in order to automatically spin up Kafka and ZooKeeper. This is my current configuration which works fine: <properties> <...
0
votes
1answer
45 views

Confluent KSQL server on Windows - could not find or load main class

I'd like to connect to my own Kafka instance in order to try KSQL. I downloaded the latest version of Confluent Open Source platform - https://www.confluent.io/download/ I follow the https://docs....
0
votes
0answers
24 views

Confluent Kafka : confluent_kafka.avro.serializer.SerializerError: unable to fetch schema

When we tried to read the avro format kafka through kafka topic , by using of confluent_kafka.avro.serializer could not read the schema topic . Code snippet: schema_registry_client =...