Hello ποΈ !
Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed π¦ all of that in a new home-made tool named producer-file
. Here are the features :
- π¨ Sending line by line
- πͺ CSV to JSON conversion
- π¦ Support message headers by file and/or by line
- β‘οΈ Delta-changes mode (only send changes occurred after last send)
βοΈ More about the shell way :
![opt-nc](https://res.cloudinary.com/practicaldev/image/fetch/s--akFu7-GX--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://res.cloudinary.com/practicaldev/image/fetch/s--0QsNl5GP--/c_fill%2Cf_auto%2Cfl_progressive%2Ch_150%2Cq_auto%2Cw_150/https://dev-to-uploads.s3.amazonaws.com/uploads/organization/profile_image/5458/98dc3bfe-45f7-48c6-a182-21944edf46b6.jpg)
![](https://res.cloudinary.com/practicaldev/image/fetch/s--uIgDHfBs--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://res.cloudinary.com/practicaldev/image/fetch/s--qsqU1oih--/c_fill%2Cf_auto%2Cfl_progressive%2Ch_150%2Cq_auto%2Cw_150/https://dev-to-uploads.s3.amazonaws.com/uploads/user/profile_image/446871/3e9ded5c-f368-4906-a277-35e56c9f97a7.png)
β¨οΈ Pipe xlsx files into/from Kafka... From cli with (k)cat π
adriens for opt-nc γ» Oct 10 '22 γ» 2 min read
Motivations
Using jq
+ kcat
with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities π€·π»ββοΈ.
π½οΈ Demo
Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :
Top comments (2)
Here is a related and worhtwhile content :
Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`
Ce projet explique comment dΓ©velopper un producer/consumer sans la lib interne historique au profit de celle dΓ©veloppΓ©e et maintenue par Spring.
Projet d'exemple d'utilisation de la lib
org.springframework.kafka:spring-k...
kafka
docker
&docker-compose
(si utilisation dekafka.yml
)Avant de passer Γ la suite, prendre soin de consulter les ressources ci-dessous:
demo.user
Entrer dans le container
kafka
:CrΓ©er le producer :
Copier les lignes :
demo.sms
Initialiserβ¦