⌨️ Pipe xlsx files into/from Kafka... From cli with (k)cat 🙀

🪝 Teaser

What you'll discover in this post is how we could, within a dedicated KillerCoda scenario :

• Literally pipe data to and from Kafka, from (and to) any file, with the | operator
• Visualize data on a cool web interface on Redpanda Console

🍒 on the 🍰

We'll play with real life data from isee.nc :

☝️ Now... to fully enjoy this post, please be sure to really

👉 deeply understand what pipe is, how and why it was created... and all the amazing opportunities it opened since its early ages.

🪄 Intro : Unix Pipeline 🦸‍♂️

Spend 5' to deeply understand how and why the | was created and why it still matters (a lot) today.

🧰 Tools (and Philosophy) 🏺

Now, you can jump start in the demo where we'll do everything from the terminal thanks to all these amazing, UNIX Philosophic single-purposed tools :

• cURL
• pipe
• kcat
• xls2csv
• jq
• tail
• head
• cut
• comm
• sort
• column
• sed
• awk
• wc

🍿 Full demo

🔖 Resources

• Dedicated Killercoda scenario

Comments

[Antoine Arthur]

Quite interesting resource and very useful Killercoda scenario. I've done it a few times to get the hang of Kafka and its mechanisms.

[adriens]

Thanks a lot for the kind feedback bro' 🙏

[adriens]

Some related content :

[Robin Moffatt]

Sounds cool - but where's the actual code? Is it only in the video?

[adriens]

Check the Killecoda scenario : killercoda.com/opt-labs/course/kaf...

[adriens]

Adrien SALES

@rastadidi

Thank you @computer_phile 😘 we also love piping a lot of stuff : from files into #kafka 😎 twitter.com/rastadidi/stat…

20:47 PM - 10 Oct 2022

Adrien SALES @rastadidi

✋ Ever wondered if you could send an #excel rows into an @apachekafka instance from any #linux terminal❔ 🪄Discover that on video or enjoy it by yourself on @killercoda ... and visualize data from @StatEcoNC on @redpandadata 🪄 @confluentinc https://t.co/z0InxPPFrR

[adriens]

Adrien SALES

@rastadidi

✋ Ever wondered if you could send an #excel rows into an @apachekafka instance from any #linux terminal❔ 🪄Discover that on video or enjoy it by yourself on @killercoda ... and visualize data from @StatEcoNC on @redpandadata 🪄
@confluentinc
dev.to/optnc/pipe-xls…

20:44 PM - 10 Oct 2022