DEV Community

Cover image for lexical analysis / tokenization
Victor Hugo
Victor Hugo

Posted on

lexical analysis / tokenization

A long time ago some friends showed me one project, one screen split in a half, on the left has a code editor with prisma models and on the right a diagram with nodes with the prisma model. It is a really good whey to see your models but I forgot about this project.

In this year I decided to learn Golang and was thinking about projects that I could do to learn more this language, in my research I discovered a technique called Lexical analysis, also known as tokenization, when I discovered it the first thing that came into my mind was that project with prisma models and diagrams and them I decided to recreate this project using this tokenization technique.

what is tokenization

Lexical analysis (also called tokenization) is the process of converting raw source code into structured tokens, which are the smallest meaningful units of a programming language. This technique was created to create programming languages. But you can use this technique to anything you want.

In that whey I created an API in Golang that reads the prisma models and tokenize it with a struct with type and value, the program uses position to know where the program is currently reading when it gets a piece of the text and tokenize it, it advances the position in the text until it finds the next token.

After the Tokenization, it gets the models tokens and formats it with an object with id, model name, the content of the model with the columns and the relations between the models and returns to the api.

How do I use it

in frontend I have a code editor that accepts prisma models and a diagram canvas. Every code you put into the code editor it sends to the api and render the nodes in the canvas with the data returned

Here is the link of the project if you want to test

a diagram explaning the flow of project

Top comments (0)