Showing content from http://cran.rstudio.com/web/packages/rJava/../FeatureExtraction/../testthat/../rtiktoken/index.html below:
CRAN: Package rtiktoken
rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Version: 0.0.7 Depends: R (≥ 4.2) Suggests: testthat (≥ 3.0.0) Published: 2025-04-14 DOI: 10.32614/CRAN.package.rtiktoken Author: David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut] (tiktoken-rs Rust library), Authors of the dependent Rust crates [aut] (see AUTHORS file)
rtiktoken author details Maintainer: David Zimmermann-Kollenda <david_j_zimmermann at hotmail.com> BugReports: https://github.com/DavZim/rtiktoken/issues License: MIT + file LICENSE URL: https://davzim.github.io/rtiktoken/, https://github.com/DavZim/rtiktoken/ NeedsCompilation: yes SystemRequirements: Cargo (Rust's package manager), rustc >= 1.65.0 Materials: README NEWS CRAN checks: rtiktoken results Documentation: Reference manual: rtiktoken.pdf Downloads: Package source: rtiktoken_0.0.7.tar.gz Windows binaries: r-devel: rtiktoken_0.0.7.zip, r-release: rtiktoken_0.0.7.zip, r-oldrel: rtiktoken_0.0.7.zip macOS binaries: r-release (arm64): rtiktoken_0.0.7.tgz, r-oldrel (arm64): rtiktoken_0.0.7.tgz, r-release (x86_64): rtiktoken_0.0.7.tgz, r-oldrel (x86_64): rtiktoken_0.0.7.tgz Old sources: rtiktoken archive Linking:
Please use the canonical form https://CRAN.R-project.org/package=rtiktoken to link to this page.
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4