sun::parsing

namespace · Source (opens in a new tab)

Turns source text into syntax trees and provides source formatting.

Turns Sun source text into tokens and syntax trees.

Classes

• sun::parsing::InterpolatedStringParser (public)
• sun::parsing::Lexer (public)
• sun::parsing::LoweringPass (public)
• sun::parsing::NFA (public)
• sun::parsing::DFA (public)
• sun::parsing::RegexParser (public)
• sun::parsing::Parser (public)

Structs

• sun::parsing::TokenInfo (public)
• sun::parsing::Token (public)
• sun::parsing::State (public)
• sun::parsing::RegexCapture (public)
• sun::parsing::TokenChar (public)
• sun::parsing::Comment (public)

Functions

• attachDocComments
• compoundToBinaryOp
• decodeUtf8
• docCommentAbove
• encodeUtf8
• formatProgram
• formatSource
• getKeywordSpelling
• getTokenInfo
• hexByte
• hexDigit
• isAssignmentOp
• isKeyword
• isPublicableStatementStart
• isScalarValue
• isSurrogate
• normalizeBody
• putchard
• simple
• utf8SequenceLength

attachDocComments

public · function · Source (opens in a new tab)

void sun::parsing::attachDocComments(sun::ast::BlockExprAST &program, const std::string &source)

Store the doc comment of every declaration in the program on its node — functions, classes and their fields and methods, interfaces, enums and their variants, module-level variables — so it travels with the tree into .moon bundles and reaches editor tooling without the source at hand.

Attaches source documentation to declarations in a parsed program.

Related: sun::ast::BlockExprAST

compoundToBinaryOp

public · function · Source (opens in a new tab)

std::optional< TokenKind > sun::parsing::compoundToBinaryOp(TokenKind kind)

Map compound-assignment token kinds (+=, -=, ...) to the underlying binary operator; nullopt for anything else.

Shared by the parser, semantic analysis, and codegen.

Related: TokenKind

decodeUtf8

public · function · Source (opens in a new tab)

std::optional< uint32_t > sun::parsing::decodeUtf8(std::string_view s, int &length)

Decode the UTF-8 sequence at the start of s.

Returns the scalar value and writes the number of bytes consumed to length, or nullopt if the bytes are not well-formed UTF-8 (bad continuation byte, overlong form, surrogate, or out of range).

docCommentAbove

public · function · Source (opens in a new tab)

std::string sun::parsing::docCommentAbove(const std::string &source, int line)

Returns the documentation comment preceding a source line.

encodeUtf8

public · function · Source (opens in a new tab)

int sun::parsing::encodeUtf8(uint32_t scalar, char *out)

Encode scalar as UTF-8 into out (at least 4 bytes).

Returns the number of bytes written.

formatProgram

public · function · Source (opens in a new tab)

std::string sun::parsing::formatProgram(const sun::ast::BlockExprAST &program, const std::map< int, sun::parsing::Comment > &comments, const std::string &source)

Format an already-parsed lossless (pre-lowering) program.

Formats a parsed program with its original comments and source context.

comments must come from the same parse (Parser::getComments()) and source must be the exact text that was parsed — literals and types are sliced from it.

Related: sun::ast::BlockExprAST, sun::parsing::Comment, Parser::getComments()

formatSource

public · function · Source (opens in a new tab)

std::string sun::parsing::formatSource(const std::string &source, const std::string &filePath="<fmt>")

Parse source and reprint it in the canonical style (2-space indent, K&R braces, comments preserved).

Parses and formats Sun source text for the supplied file.

Throws SunError on parse failure; the input is never partially formatted.

Related: SunError

getKeywordSpelling

public · function · Source (opens in a new tab)

std::optional< std::string_view > sun::parsing::getKeywordSpelling(TokenKind kind)

The spelling of a keyword token (a token whose regex is a bare word, so it would otherwise lex as an identifier); nullopt for any other kind.

Derived from the regex table so a new keyword is covered without a second list.

Related: TokenKind

getTokenInfo

public · function · Source (opens in a new tab)

const std::map< TokenKind, TokenInfo > & sun::parsing::getTokenInfo()

Returns the token category information.

Related: TokenKind, TokenInfo

hexByte

public · function · Source (opens in a new tab)

std::optional< char > sun::parsing::hexByte(std::string_view digits)

Decode the first two hex digits as a byte, or reject missing or invalid digits.

hexDigit

public · function · Source (opens in a new tab)

int sun::parsing::hexDigit(char c)

Value of one hex digit, or -1.

isAssignmentOp

public · function · static · Source (opens in a new tab)

static bool sun::parsing::isAssignmentOp(TokenKind kind)

True for '=' and every compound-assignment operator.

Related: TokenKind

isKeyword

public · function · Source (opens in a new tab)

bool sun::parsing::isKeyword(TokenKind kind)

Reports whether the token category is a language keyword.

Related: TokenKind

isPublicableStatementStart

public · function · static · Source (opens in a new tab)

static bool sun::parsing::isPublicableStatementStart(TokenKind kind)

Statement kinds that may carry a public modifier at item level.

Related: TokenKind

isScalarValue

public · function · Source (opens in a new tab)

bool sun::parsing::isScalarValue(uint32_t v)

A Unicode scalar value is anything up to kMaxScalar that is not a surrogate.

isSurrogate

public · function · Source (opens in a new tab)

bool sun::parsing::isSurrogate(uint32_t v)

UTF-16 surrogates are not scalar values and may not appear on their own.

normalizeBody

public · function · static · Source (opens in a new tab)

static void sun::parsing::normalizeBody(std::unique_ptr< ExprAST > &slot, bool isIfBody)

Normalize an if/loop body block to the shape the core pipeline expects: empty block -> synthetic literal, single-statement block -> the statement.

The parser keeps bodies as blocks for losslessness.

Related: ExprAST

putchard

public · function · static · Source (opens in a new tab)

static double sun::parsing::putchard(double X)

The actual C function that prints a character.

simple

public · function · Source (opens in a new tab)

std::optional< char > sun::parsing::simple(char c)

Handles newline, tab, carriage return, backslash, and null escapes.

Returns the character the escape stands for, or nullopt if c does not name one of them (the caller decides what to do with quotes and hexadecimal or Unicode escapes).

utf8SequenceLength

public · function · Source (opens in a new tab)

int sun::parsing::utf8SequenceLength(unsigned char lead)

Number of bytes in the UTF-8 sequence that starts with lead, or 0 if lead is not a valid leading byte.

Variables

• kMaxScalar
• tokenRegexes

kMaxScalar

public · variable · Source (opens in a new tab)

uint32_t sun::parsing::kMaxScalar = 0x10FFFF

No documentation comment.

tokenRegexes

public · variable · static · Source (opens in a new tab)

const std::map<TokenKind, std::string> sun::parsing::tokenRegexes

No documentation comment.

Related: TokenKind

Enums

• TokenKind

TokenKind

public · enum · Source (opens in a new tab)

enum class sun::parsing::TokenKind

Identifies the lexical categories recognized by the Sun parser.

TOK_EOF

No documentation comment.

COMMENT

No documentation comment.

BLOCK_COMMENT

No documentation comment.

DECLARE

No documentation comment.

DEF

No documentation comment.

EXTERN

No documentation comment.

VAR

No documentation comment.

MANIFEST

No documentation comment.

MODULE

No documentation comment.

USING

No documentation comment.

CLASS

No documentation comment.

PARTIAL

No documentation comment.

PUBLIC

No documentation comment.

PACKED_CLASS

No documentation comment.

INTERFACE

No documentation comment.

ENUM

No documentation comment.

EXTENDS

No documentation comment.

IMPLEMENTS

No documentation comment.

THIS

No documentation comment.

NULL_LITERAL

No documentation comment.

TRUE_LITERAL

No documentation comment.

FALSE_LITERAL

No documentation comment.

IF

No documentation comment.

MATCH

No documentation comment.

ELSE

No documentation comment.

FOR

No documentation comment.

WHILE

No documentation comment.

BREAK

No documentation comment.

CONTINUE

No documentation comment.

RETURN

No documentation comment.

FUNCTION

No documentation comment.

TEST_FUNCTION

No documentation comment.

TRY

No documentation comment.

CATCH

No documentation comment.

THROW

No documentation comment.

THROWS

No documentation comment.

UNSAFE

No documentation comment.

STATIC_PTR

No documentation comment.

PTR

No documentation comment.

RAW_PTR

No documentation comment.

REF

No documentation comment.

CONST

No documentation comment.

ARRAY

No documentation comment.

ARROW

No documentation comment.

FAT_ARROW

No documentation comment.

UNDERSCORE

No documentation comment.

TYPE_I8

No documentation comment.

TYPE_I16

No documentation comment.

TYPE_I32

No documentation comment.

TYPE_I64

No documentation comment.

TYPE_U8

No documentation comment.

TYPE_U16

No documentation comment.

TYPE_U32

No documentation comment.

TYPE_U64

No documentation comment.

TYPE_F32

No documentation comment.

TYPE_F64

No documentation comment.

TYPE_BOOL

No documentation comment.

TYPE_VOID

No documentation comment.

TYPE_CHAR

No documentation comment.

STRING

No documentation comment.

CHAR_LITERAL

No documentation comment.

BYTE_LITERAL

No documentation comment.

TEMPLATE_STRING

No documentation comment.

BRACE_OPEN

No documentation comment.

BRACE_CLOSE

No documentation comment.

BRACKET_OPEN

No documentation comment.

BRACKET_CLOSE

No documentation comment.

PLUS

No documentation comment.

MINUS

No documentation comment.

STAR

No documentation comment.

SLASH

No documentation comment.

PERCENT

No documentation comment.

AMPERSAND

No documentation comment.

PIPE

No documentation comment.

CARET

No documentation comment.

LEFT_SHIFT

No documentation comment.

RIGHT_SHIFT

No documentation comment.

TILDE

No documentation comment.

PLUS_ASSIGN

No documentation comment.

MINUS_ASSIGN

No documentation comment.

STAR_ASSIGN

No documentation comment.

SLASH_ASSIGN

No documentation comment.

PERCENT_ASSIGN

No documentation comment.

AMP_ASSIGN

No documentation comment.

PIPE_ASSIGN

No documentation comment.

CARET_ASSIGN

No documentation comment.

LEFT_SHIFT_ASSIGN

No documentation comment.

RIGHT_SHIFT_ASSIGN

No documentation comment.

AND

No documentation comment.

OR

No documentation comment.

NOT

No documentation comment.

LESS

No documentation comment.

LESS_EQUAL

No documentation comment.

GREATER

No documentation comment.

GREATER_EQUAL

No documentation comment.

EQUAL

No documentation comment.

EQUAL_EQUAL

No documentation comment.

NOT_EQUAL

No documentation comment.

PAREN_OPEN

No documentation comment.

PAREN_CLOSE

No documentation comment.

COMMA

No documentation comment.

SEMI_COLON

No documentation comment.

DOUBLE_COLON

No documentation comment.

COLON

No documentation comment.

QUESTION

No documentation comment.

ELLIPSIS

No documentation comment.

DOT

No documentation comment.

INTEGER

No documentation comment.

FLOAT

No documentation comment.

INTRINSIC_IDENTIFIER

No documentation comment.

IDENTIFIER

No documentation comment.

TYPED_FLOAT

No documentation comment.

TYPED_INTEGER

No documentation comment.

LIFETIME

No documentation comment.

UNKNOWN

No documentation comment.

COUNT

No documentation comment.

Type Aliases

• Segment
• SourceFileId

Segment

public · typedef · Source (opens in a new tab)

using sun::parsing::Segment =  sun::ast::InterpolatedStringAST::Segment

A literal or embedded-expression segment of an interpolated string.

Related: sun::ast::InterpolatedStringAST::Segment