Danil Nikiforov danilNil

## README.md

      
              2 files
            
          
              0 forks
            
          
                3 comments
              
            
              13 stars
            
          
                izikeros
                / README.md
            
            
              Last active
              January 12, 2025 13:46
            
              
                [split text fixed tokens] Split text into parts with limited length in tokens #llm #tokens #python
              
          
    Text Splitter


A Python script for splitting text into parts with controlled (limited) length in tokens. This script utilizes the tiktoken library for encoding and decoding text.
Table of Contents


Introduction
Installation