Class TokenizerGpt3

Namespace: OpenAI.Tokenizer.GPT3

Assembly: AntRunnerLib.dll

GPT3 Tokenizer.

public static class TokenizerGpt3

Inheritance: System.Object

TokenizerGpt3

Inherited Members: System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Methods

Encode(String, Boolean)

Encode This method use LF style EOL, if you use CR LF style EOL you need to set cleanUpWindowsEOL to true

public static IEnumerable<int> Encode(string text, bool cleanUpCREOL = false)

Parameters

text System.String
cleanUpCREOL System.Boolean: set it true to get rid of CR

Returns

IEnumerable<System.Int32>

TokenCount(String, Boolean)

Get token count. This method use LF style EOL, if you use CR LF style EOL you need to set cleanUpWindowsEOL to true

public static int TokenCount(string text, bool cleanUpCREOL = false)

Parameters

text System.String
cleanUpCREOL System.Boolean: set it true to get rid of CR

Returns

System.Int32