What you will find in this article?
Real-time analytics have become a vital part of today's applications, especially when it comes to understanding user behavior and improving your platform based on data-driven insights. Even more so, if your users rely on your product to deliver them with insights.
In this post, we are going to explore how we can create a real-time analytics dashboard for page views using Tinybird, Tremor.so, and Next.js.
We will be using Tinybird for data ingestion and real-time data analysis, Tremor for data visualization, and Next.js for server-side rendering.
Papermark - the open-source DocSend alternative.
Before we kick off, let me introduce you to Papermark. It's an open-source project for securely sharing documents with built-in real-time, page-by-page analytics, powered by Tinybird and visualized by Tremor.
I would be absolutely thrilled if you could give us a star! Don't forget to share your thoughts in the comments section ❤️
https://github.com/mfts/papermark
Setup the project
Let's set up our project environment. We will be setting up a Next.js app, installing Tinybird CLI and configuring the needed services and tools.
Set up tea
It's a good idea to have a package manager handy, like tea
. It'll handle your development environment and simplify your (programming) life!
sh <(curl https://tea.xyz)
# --- OR ---
# using brew
brew install teaxyz/pkgs/tea-cli
tea
frees you to focus on your code, as it takes care of installing python
and pipenv
(which I'm using to run tinybird-cli
), node
, npm
and any other packages you may need. The best part is, tea
installs all packages in a dedicated directory (default: ~/.tea
), keeping your system files neat and tidy.
Set up Next.js with TypeScript and Tailwindcss
We will use create-next-app to generate a new Next.js project. We will also be using TypeScript and Tailwind CSS, so make sure to select those options when prompted.
npx create-next-app
# ---
# you'll be asked the following prompts
What is your project named? my-app
Would you like to add TypeScript with this project? Y/N
# select `Y` for typescript
Would you like to use ESLint with this project? Y/N
# select `Y` for ESLint
Would you like to use Tailwind CSS with this project? Y/N
# select `Y` for Tailwind CSS
Would you like to use the `src/ directory` with this project? Y/N
# select `N` for `src/` directory
What import alias would you like configured? `@/*`
# enter `@/*` for import alias
Install Tinybird
Tinybird's command line interface (CLI) helps us to manage data sources and pipes. I'm installing tinybird-cli
using pipenv
, which manages a virtual environment for my pip packages, in our local environment.
# Navigate to your Next.js repo
cd my-app
# Create a new virtual environment and install Tinybird CLI
# if you install `tea` in the previous step, then tea will take care of installing pipenv and its dependencies
pipenv install tinybird-cli
# Activate the virtual environment
pipenv shell
Configuring Tinybird
Head over to tinybird.co and create a free account. You need to have a Tinybird account and an associated token. You can get the token from your Tinybird's dashboard.
# Authenticate with tinybird-cli using your auth token when prompted
tb auth
A .tinyb
file will be added to your repo's root. No need to modify it. However add it it to your .gitignore
file to avoid exposing your token.
echo ".tinyb" >> .gitignore
Building the application
Now that we have our setup in place, we are ready to start building our application. The main features we'll cover are:
- Tinybird Pipes and Datasources
- Page View Recording
- Tremor Bar Chart
#1 Tinybird Pipes and Datasource
The ability to programmatically configure pipes and datasources for Tinybird offers a significant advantage. This flexibility enables us to treat our data infrastructure as code, meaning that the entire configuration can be committed into a version control system. For an open-source project like Papermark, this capability is highly beneficial. It fosters transparency and collaboration, as contributors can readily understand the data structure without any ambiguity.
We set up Tinybird pipes and datasource as follows:
mkdir -p lib/tinybird/{datasources,endpoints}
1. Datasources
This is basically a versioned schema for page_views
. This the only datasource we need for ingesting, storing and reading analytics about page views. Feel free to add/remove fields.
# lib/tinybird/datasources/page_views.datasource
VERSION 1
DESCRIPTION >
Page views are events when a user views a document
SCHEMA >
`id` String `json:$.id`,
`linkId` String `json:$.linkId`,
`documentId` String `json:$.documentId`,
`viewId` String `json:$.viewId`,
# Unix timestamp
`time` Int64 `json:$.time`,
`duration` UInt32 `json:$.duration`,
# The page number
`pageNumber` LowCardinality(String) `json:$.pageNumber`,
`country` String `json:$.country`,
`city` String `json:$.city`,
`region` String `json:$.region`,
`latitude` String `json:$.latitude`,
`longitude` String `json:$.longitude`,
`ua` String `json:$.ua`,
`browser` String `json:$.browser`,
`browser_version` String `json:$.browser_version`,
`engine` String `json:$.engine`,
`engine_version` String `json:$.engine_version`,
`os` String `json:$.os`,
`os_version` String `json:$.os_version`,
`device` String `json:$.device`,
`device_vendor` String `json:$.device_vendor`,
`device_model` String `json:$.device_model`,
`cpu_architecture` String `json:$.cpu_architecture`,
`bot` UInt8 `json:$.bot`,
`referer` String `json:$.referer`,
`referer_url` String `json:$.referer_url`
ENGINE "MergeTree"
ENGINE_SORTING_KEY "linkId,documentId,viewId,pageNumber,time,id"
2. Pipes
In Tinybird, a pipe is a series of transformations that are applied to your data. These transformations could include anything from simple data cleaning operations to complex aggregations and analytics.
We have one versioned pipe (also sometimes called endpoint) to retrieve page_view data. This Tinybird pipe, named endpoint
, calculates the average duration users spend on each page of a specific document within a defined timeframe, and presents the results in ascending order by page number.
# lib/endpoints/get_average_page_duration.pipe
VERSION 1
NODE endpoint
SQL >
%
SELECT
pageNumber,
AVG(duration) AS avg_duration
FROM
page_views__v1
WHERE
documentId = {{ String(documentId, required=True) }}
AND time >= {{ Int64(since, required=True) }}
GROUP BY
pageNumber
ORDER BY
pageNumber ASC
Now that we have Tinybird datasources and pipes set up, you need to push them to your Tinybird's account using the CLI.
# Navigate to the directory containing your datasource and pipe files
cd lib/tinybird
# Push your files to Tinybird
tb push datasources/*.datasource pipes/*.pipe
3. Typescript functions
Let's set up the appropriate typescript functions to actually send and retrieve data from Tinybird. We are using zod
and chronark's zod-bird
library
This function is for retrieving data from Tinybird:
// lib/tinybird/pipes.ts
import { z } from "zod";
import { Tinybird } from "@chronark/zod-bird";
const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! });
export const getTotalAvgPageDuration = tb.buildPipe({
pipe: "get_total_average_page_duration__v1",
parameters: z.object({
documentId: z.string(),
since: z.number(),
}),
data: z.object({
pageNumber: z.string(),
avg_duration: z.number(),
}),
});
This function is for sending data to Tinybird:
// lib/tinybird/publish.ts
import { z } from "zod";
import { Tinybird } from "@chronark/zod-bird";
const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! });
export const publishPageView = tb.buildIngestEndpoint({
datasource: "page_views__v1",
event: z.object({
id: z.string(),
linkId: z.string(),
documentId: z.string(),
viewId: z.string(),
time: z.number().int(),
duration: z.number().int(),
pageNumber: z.string(),
country: z.string().optional().default("Unknown"),
city: z.string().optional().default("Unknown"),
region: z.string().optional().default("Unknown"),
latitude: z.string().optional().default("Unknown"),
longitude: z.string().optional().default("Unknown"),
ua: z.string().optional().default("Unknown"),
browser: z.string().optional().default("Unknown"),
browser_version: z.string().optional().default("Unknown"),
engine: z.string().optional().default("Unknown"),
engine_version: z.string().optional().default("Unknown"),
os: z.string().optional().default("Unknown"),
os_version: z.string().optional().default("Unknown"),
device: z.string().optional().default("Desktop"),
device_vendor: z.string().optional().default("Unknown"),
device_model: z.string().optional().default("Unknown"),
cpu_architecture: z.string().optional().default("Unknown"),
bot: z.boolean().optional(),
referer: z.string().optional().default("(direct)"),
referer_url: z.string().optional().default("(direct)"),
}),
});
4. Configure Auth Token for production
Don't forget to add TINYBIRD_TOKEN
to your .env
file. It's advised that you create a token that has the minimum scope for your operations:
- Read from specific pipe
- Append to specific datasource
Congrats! 🎉 You successfully configured Tinybird and ready to move to the next
#2 Page View Recording
We will capture the page view event in our PDFviewer component and publish it to our Tinybird's datasource.
Let's build an API function to send data to Tinybird every time a page is viewed:
// pages/api/record_view.ts
import { NextApiRequest, NextApiResponse } from "next";
import { publishPageView } from "@/lib/tinybird";
import { z } from "zod";
import { v4 as uuidv4 } from 'uuid';
// Define the validation schema
const bodyValidation = z.object({
id: z.string(),
linkId: z.string(),
documentId: z.string(),
viewId: z.string(),
time: z.number().int(),
duration: z.number().int(),
pageNumber: z.string(),
...
});
export default async function handle(
req: NextApiRequest,
res: NextApiResponse
) {
// We only allow POST requests
if (req.method !== "POST") {
res.status(405).json({ message: "Method Not Allowed" });
return;
}
const { linkId, documentId, viewId, duration, pageNumber } = req.body;
const time = Date.now(); // in milliseconds
const pageViewId = uuidv4();
const pageViewObject = {
id: pageViewId,
linkId,
documentId,
viewId,
time,
duration,
pageNumber: pageNumber.toString(),
...
};
const result = bodyValidation.safeParse(pageViewObject);
if (!result.success) {
return res.status(400).json(
{ error: `Invalid body: ${result.error.message}` }
);
}
try {
await publishPageView(result.data);
res.status(200).json({ message: "View recorded" });
} catch (error) {
res.status(500).json({ message: (error as Error).message });
}
}
And finally the PDFViewer component that sends the request:
// components/PDFViewer.tsx
import { useState, useEffect } from 'react';
const PDFViewer = () => {
const [pageNumber, setPageNumber] = useState<number>(1)
useEffect(() => {
startTime = Date.now(); // update the start time for the new page
// when component unmounts, calculate duration and track page view
return () => {
const endTime = Date.now();
const duration = Math.round(endTime - startTime);
trackPageView(duration);
};
}, [pageNumber]); // monitor pageNumber for changes
async function trackPageView(duration: number = 0) {
await fetch("/api/record_view", {
method: "POST",
body: JSON.stringify({
linkId: props.linkId,
documentId: props.documentId,
viewId: props.viewId,
duration: duration,
pageNumber: pageNumber,
}),
headers: {
"Content-Type": "application/json",
},
});
}
return (
// Your PDF Viewer implementation
);
}
export default PDFViewer;
#3 Tremor Bar Chart
Let's now create a bar chart to display the page views for each document. We are using tremor.so to build our beautiful dashboard.
# Install tremor with their CLI
npx @tremor/cli@latest init
// components/bar-chart.tsx
import { BarChart } from "@tremor/react";
const timeFormatter = (number) => {
const totalSeconds = Math.floor(number / 1000);
const minutes = Math.floor(totalSeconds / 60);
const seconds = Math.round(totalSeconds % 60);
// Adding zero padding if seconds less than 10
const secondsFormatted = seconds < 10 ? `0${seconds}` : `${seconds}`;
return `${minutes}:${secondsFormatted}`;
};
export default function BarChartComponent({data}) {
return (
<BarChart
className="mt-6 rounded-tremor-small"
data={data}
index="pageNumber"
categories={["Time spent per page"]}
colors={["gray"]}
valueFormatter={timeFormatter}
yAxisWidth={50}
showGridLines={false}
/>
);
}
// lib/swr/use-stats.ts
import { useRouter } from "next/router";
import useSWR from "swr";
import { getTotalAvgPageDuration } from "@/lib/tinybird/pipes";
export function useStats() {
const router = useRouter();
const { id } = router.query as { id: string };
const { data, error } = useSWR(
id,
() => getTotalAvgPageDuration({ documentId: id, since: 0
}),
{
dedupingInterval: 10000,
}
);
return {
durationData: data,
isLoading: !error && !data,
error,
};
}
// pages/document/[id].tsx
import { useDocument } from "@/lib/swr/use-document";
import { useStats } from "@/lib/swr/use-stats";
import BarChartComponent from "@/components/bar-chart";
export default function DocumentPage() {
const { document, error: documentError } = useDocument();
const { stats, error: statsError } = useStats();
if (documentError) {
// handle document error
}
if (statsError) {
// handle stats error
}
return (
<>
<main>
{document && (
<header>
<h1>{document.name}</h1>
{stats && <BarChartComponent data={stats.durationData} />}
</header>
)}
</main>
</>
);
}
Voila 🎉 The Bar Chart with accurate page-by-page time!
Conclusion
That's it! We've built a real-time analytics dashboard for page views using Tinybird, Tremor, and Next.js. While the example here is simple, the same concepts can be expanded to handle any kind of analytics your app might need to perform.
Thank you for reading. I'm Marc, an open-source advocate. I am building papermark.io - the open-source alternative to DocSend with millisecond-accurate page analytics.
Help me out!
If you found this article helpful and got to understand Tinybird and dashboarding with Tremor better, I would be eternally grateful if you could give us a star! And don't forget to share your thoughts in the comments ❤️
Top comments (9)
Amazing work Marc! And thanks for speaking so highly of Tinybird.
For those interested, Tinybird has a Web Analytics Starter Kit that includes a JavaScript tracker and some predefined SQL metrics. Can be a good kick start if you're working on a similar project.
The starter kit is great 🤩
There’s still so much to explore in Tinybird. Materialized Views is next on the list.
Amazing!
I didn't know about Tinybird and Tremor.so!
I'm gonna need to utilize them for my next project 🚀
Thanks Nevo 🤗
I’m really glad my article is helpful for you. My article says it all - it’s never been easier to bake in analytics for your users.
Very Informative article. Thank you.
You’re very welcome ☺️ so happy it’s helpful to you
Now 10x more value for Papermark
10x more value for any project that uses real-time analytics 🤗📈
Thanks! 🤩 so glad it's helpful