tictonix

• In this implementation, the embedding matrix has columns corresponding to tokens (each column is an embedding for one token).
• All calculations are performed exclusively on CPU (without GPU).

Installing

Add to your Cargo.toml:

[dependencies]
tictonix = "1.0.1"

Logging

The project uses crate log which provides a logging facade.

At the moment logging is implemented only for IO operations (saving and loading files), with a success message.

You can use any compatible implementation (e.g. env_logger, fern, simple_logger, tracing, ...) To do this, just initialize the selected logger in your application before starting.

Example of initialization with env_logger:

fn main() {
    Builder::new()
        .filter_level(LevelFilter::Info)
        .filter_module("tictonix", LevelFilter::Info)
        .init();
    // Your code
}

Usage

See examples for usage.

Documentation

See documentation for the project.

Glossary

• Tokenization is the process of breaking text into separate elements called tokens. Tokens can be words, characters, sub-words, or other units, depending on the chosen tokenization method. This process is an important step in text preprocessing for Natural Language Processing (NLP) tasks.
• LLMs (large language models) are large language models based on deep learning architectures (e.g., Transformer) that are trained on huge amounts of textual data. They are designed to perform a wide range of tasks related to natural language processing, such as text generation, translation, question answering, classification, and others. LLMs are capable of generalizing knowledge and performing tasks on which they have not been explicitly trained (zero-shot or few-shot learning).
• Transformer is a neural network architecture proposed in 2017 that uses the attention mechanism to process sequences of data such as text. The main advantage of Transformer is its ability to process long sequences and take context into account regardless of the distance between elements of the sequence. This architecture is the basis for most modern LLMs (such as GPT, BERT and others).
• Embedding is a numerical (vector) representation of text data (tokens, words, phrases or sentences).
• Positional Encoding is a technique used in the Transformer architecture to convey information about the order of elements in a sequence. Since Transformer has no built-in order information (unlike recurrent networks), positional encoding adds special signals to token embeddings that depend on their position in the sequence. sequence. This allows the model to take into account the order of words or other elements in the input data.

P.S.

Don't forget to leave a ⭐ if you found this project useful.