DEV Community

Understanding LangChain's RecursiveCharacterTextSplitter

Youdiowei Eteimorde on August 12, 2023

Large language models are powerful tools with extensive capabilities; nonetheless, they grapple with a distinct limitation known as the context win...

Read full post

scitlec • Aug 31 '23

The maintainers of the Langchain documentation should link to your useful explanation.

Thanks!

Pham Minh Quang • Sep 10 '23

I totally agree! The langchain documentation is just suck.

Youdiowei Eteimorde • Sep 1 '23

Thanks for your kind words 🥰 Who knows they might.

Cristian Molina • Jan 25

What about the chunk_overlap param?

Youdiowei Eteimorde • Jan 27 • Edited

The chunk_overlap parameter determines how much the chunks overlap with each other.

For example let's split your comment into three chunks.

What about | the chunk_ | overlap param?

Let's overlap each chunk with 5 characters:

What about the | about the chunk_ | chunk_overlap param?

If we didn't use chunk overlapping your comment would have lost is meaning when split.

Cristian Molina • Jan 28

Thanks! That makes sense but what value should I use if, for instance, I need to save the texts In a vectorDB later to augment a RAG?
Does it matter? If this is significant I'd add this information to the article.
Thanks again.

Youdiowei Eteimorde • Jan 29

It is all depends on your data and what you are trying to achieve. The whole Augmenting LLMs with external knowledge is still in it's infancy. So you can experiment with different params to see how your LLM performs during RAG.

James Stover • Jan 17

Something doesn’t quite work right as I see some words throughout my text after splitting are broken apart with a space making 2 non-words of each of them. They have quite a few characters in between, so it isn’t frequent, but in a large body of text, these add up. I am concerned about the detrimental impact to the vector embeddings and retrieval then.