Streaming SQL in Stateful DataFlows

Streaming SQL Functionality

SQL Streaming Queries and Stream Processing Operations is released in Stateful DataFlow Beta 7 running on Fluvio 0.15.2

With SQL Streaming on Stateful DataFlow you can:

• Run ad-hoc queries on saved state objects and materialized views based live event streams.
• Use SQL queries to run stream processing operations in data flows.

For those who are not aware of Fluvio or Stateful DataFlow yet:

Fluvio - Open Source distributed streaming engine written in Rust.
Git Repo - https://github.com/infinyon/fluvio

Stateful DataFlow - Stream processing layer built on Fluvio built using the wasm component model.

Example projects to try on your machine - https://github.com/infinyon/stateful-dataflows-examples/

SQL: From Static Tables to Streaming Data

Remember when SQL was the only way to talk to your data? It wasn't just a query language - it was the query language. But its story goes deeper than syntax.

The Universal Language of Data

Just as merchants in medieval Mediterranean ports needed a shared language to trade (that's where "lingua franca" came from), the tech world needed SQL to make data accessible across different systems and teams.

If you're in a room with a DBA, a data analyst, and a business analyst. What's the one language they all speak? Likely SQL.

SELECT product_name, COUNT(*) as orders
FROM sales
WHERE region = 'EMEA'
GROUP BY product_name

Look familiar? Whether you're running Oracle, Postgres, or MySQL, this just works. Well sort of!

Why SQL Still Matters

Three key factors made SQL a long-term utility that stood the test of time:

1. It's Human-Friendly
   Instead of telling machines HOW to get data, you just say WHAT you want. SELECT * FROM users WHERE status = 'active' reads almost like English.

2. It's Everywhere
   From startups to Fortune 500s, SQL skills travel. Write once, run anywhere - from healthcare to fintech.

3. It Just Works
   Need to analyze sales data? Track user behavior? SQL's got you covered, backed by decades of tooling and optimization.

The Real-Time Challenge

In a world of Artificial Intelligence, Web3, and global markets, event streaming is no longer a luxury - it's a basic need. Ask yourself:

• Is your application combining data from multiple sources in real-time?
• Are your customers happy with stale insights?
• Do you need fresh data on demand?

Bridging Static and Streaming

What if you could use familiar SQL syntax for real-time data processing? What if your team could leverage their existing SQL skills for stream processing?

We've been exploring these questions and implementing solutions that bring SQL's simplicity to streaming data. Want to see how? Check out the full article where we dive into:

• Practical examples using NY Transit data
• Real-world streaming SQL queries in action
• How to implement stream processing without learning a new language

Longer article on InfinyOn blog.