sun::parsing::Token

struct · Source (opens in a new tab)

struct sun::parsing::Token

A scanned token with its category, source position, and spelling.

Public Functions

• charLiteral
• comment
• eof
• floatNum
• getCharValue
• getFloat
• getIdentifier
• getInteger
• getLifetimeName
• getString
• getTemplateString
• identifier
• integer
• intrinsicIdentifier
• isEof
• isIntrinsicIdentifier
• lifetime
• make
• stringLiteral
• templateString
• typedFloat
• typedInteger

charLiteral

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::charLiteral(TokenKind k, uint64_t value, const sun::support::Position &s, const sun::support::Position &e, std::string txt)

Character ('a') and byte (b'a') literal factory; value is the decoded Unicode scalar value or byte.

Related: Token, TokenKind, sun::support::Position, value

comment

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::comment(TokenKind k, std::string txt, const sun::support::Position &s, const sun::support::Position &e)

Comment token factory (COMMENT or BLOCK_COMMENT); text is the raw comment including delimiters.

Related: Token, TokenKind, sun::support::Position, Comment

eof

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::eof(const sun::support::Position &pos)

Factories for value-carrying tokens.

Related: Token, sun::support::Position

floatNum

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::floatNum(double num, const sun::support::Position &s, const sun::support::Position &e, std::string txt)

Creates a floating-point token retaining its value and spelling.

Related: Token, sun::support::Position

getCharValue

public · function · Source (opens in a new tab)

std::optional< uint64_t > sun::parsing::Token::getCharValue() const

Decoded scalar value of a character literal, or byte of a byte literal.

getFloat

public · function · Source (opens in a new tab)

std::optional< double > sun::parsing::Token::getFloat() const

Returns the floating-point token value.

getIdentifier

public · function · Source (opens in a new tab)

std::optional< std::string > sun::parsing::Token::getIdentifier() const

Returns the identifier stored by this object.

getInteger

public · function · Source (opens in a new tab)

std::optional< uint64_t > sun::parsing::Token::getInteger() const

Digits of an integer literal as a value.

The lexer has already rejected anything above the u64 maximum, and a leading minus is a separate token.

getLifetimeName

public · function · Source (opens in a new tab)

std::optional< std::string > sun::parsing::Token::getLifetimeName() const

Bare name of a lifetime token ('a yields "a").

getString

public · function · Source (opens in a new tab)

std::optional< std::string > sun::parsing::Token::getString() const

Returns the string token value.

getTemplateString

public · function · Source (opens in a new tab)

std::optional< std::string > sun::parsing::Token::getTemplateString() const

Get template string content.

identifier

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::identifier(std::string id, const sun::support::Position &s, const sun::support::Position &e)

Creates an identifier token with its spelling and source range.

Related: Token, sun::support::Position

integer

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::integer(uint64_t num, const sun::support::Position &s, const sun::support::Position &e, std::string txt)

Creates an integer token retaining its value and original spelling.

Related: Token, sun::support::Position

intrinsicIdentifier

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::intrinsicIdentifier(std::string id, const sun::support::Position &s, const sun::support::Position &e)

Creates a built-in identifier token with its source range.

Related: Token, sun::support::Position

isEof

public · function · Source (opens in a new tab)

bool sun::parsing::Token::isEof() const

Reports whether this token marks the end of the input.

isIntrinsicIdentifier

public · function · Source (opens in a new tab)

bool sun::parsing::Token::isIntrinsicIdentifier() const

Reports whether this token names a built-in operation.

lifetime

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::lifetime(std::string name, const sun::support::Position &s, const sun::support::Position &e)

Lifetime name factory; name carries the bare name without the apostrophe ('a lexes as LIFETIME with name "a").

Related: Token, sun::support::Position

make

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::make(TokenKind k, const sun::support::Position &s, const sun::support::Position &e)

Generic factory for simple tokens (uses lookup table).

Indexes a flat array rather than searching getTokenInfo()'s std::map: this runs once per operator/keyword/punctuation token, and a red-black tree walk per token was measurable in the lexer's profile.

Related: Token, TokenKind, sun::support::Position, getTokenInfo()

stringLiteral

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::stringLiteral(std::string str, const sun::support::Position &s, const sun::support::Position &e)

Creates a string token with its decoded contents and source range.

Related: Token, sun::support::Position

templateString

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::templateString(std::string str, const sun::support::Position &s, const sun::support::Position &e)

Template string token factory.

Related: Token, sun::support::Position

typedFloat

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::typedFloat(double num, std::string suffix, const sun::support::Position &s, const sun::support::Position &e, std::string txt)

Creates a floating-point token with an explicit numeric type suffix.

Related: Token, sun::support::Position

typedInteger

public · function · static · Source (opens in a new tab)

static Token sun::parsing::Token::typedInteger(uint64_t num, std::string suffix, const sun::support::Position &s, const sun::support::Position &e, std::string txt)

Factories for suffixed numeric literals (21u8, 1.5f32); the lexer has already validated the suffix.

Related: Token, sun::support::Position

Public Fields

• end
• kind
• precedence
• start
• suffix
• text
• value

end

public · variable · Source (opens in a new tab)

sun::support::Position sun::parsing::Token::end

No documentation comment.

Related: sun::support::Position

kind

public · variable · Source (opens in a new tab)

TokenKind sun::parsing::Token::kind

No documentation comment.

Related: TokenKind

precedence

public · variable · Source (opens in a new tab)

int sun::parsing::Token::precedence = -1

No documentation comment.

start

public · variable · Source (opens in a new tab)

sun::support::Position sun::parsing::Token::start

No documentation comment.

Related: sun::support::Position

suffix

public · variable · Source (opens in a new tab)

std::string sun::parsing::Token::suffix

No documentation comment.

text

public · variable · Source (opens in a new tab)

std::string sun::parsing::Token::text

No documentation comment.

value

public · variable · Source (opens in a new tab)

std::variant<std::monostate, std::string, uint64_t, double > sun::parsing::Token::value

No documentation comment.