Moon Library Format

Moon Library Format

Part of the Sun ABI draft. These are proposed requirements, not guarantees of current releases. Open decisions must be resolved before stabilization.

Role of a Moon bundle

A .moon file packages compiled code with the declarations needed to import it. One bundle contract covers the container format, interface schema, generic semantics, machine ABI, and embedded dependencies. Its requirements are defined in the compatibility model.

The sections below describe the current format and proposed changes. The draft does not yet specify every byte needed to implement a reader or writer.

Current container structure

For a small geometry library, picture the file as:

[header][geometry bitcode][optional native archive][MoonLibrary protobuf]
                                                   ↑ header points here

A module record points back to its bitcode bytes. Two module records can point to exactly the same image; this does not require storing that image twice. The table below is a byte-layout reference for that picture.

The container contains a small header, compiled payload bytes, and one MoonLibrary protobuf. The library record owns the module and archive indexes; there are no separate custom index tables. Several modules may reference the same bitcode image.

Byte offsetSize in bytesContents
04Moon magic, 0x53554E4C
44Container revision
88Offset of the MoonLibrary protobuf
168Length of the MoonLibrary protobuf
HVariableLLVM bitcode and native archive payloads
Library offsetLibrary lengthComplete MoonLibrary protobuf, ending at the end of the file

H is sizeof(MoonHeader), normally 24 bytes. Header integers still use native byte order and C++ structure layout. A portable header encoding remains separate work; protobuf encoding does not make the existing header portable.

Container revision 15 introduces the complete library record. The reader requires an exact match with MoonHeader::VERSION; earlier bundles must be rebuilt. A matching container revision does not authorize semantic import: compatibility validation still precedes use of declarations or compiled code.

Library record

Suppose the header points to the final protobuf correctly, but a module claims its bitcode extends beyond the file. The reader must reject that range before trying to decode it. A correct outer header does not validate every inner record.

include/moon_bundling/moon.proto defines the following structure:

MessageContents
MoonLibraryLibrary content identity, bundle compatibility contract, ordered module records, and ordered native archive records
LibraryModuleComplete ModuleMetadata, bitcode offset, and bitcode length
LibraryArchiveArchive-set identity, archive name, byte offset, and byte length

Offsets refer to positions in the container file. Large bitcode and archive bytes remain outside protobuf. Module metadata is nested directly in the library record, so declarations and the payload references describing the library are available through one schema.

The reader bounds the library protobuf before parsing, then checks record counts, identities, duplicate keys, archive names, and payload ranges before authorizing semantic use. Exact shared bitcode ranges are allowed; unrelated overlapping ranges and payload references into the library protobuf are rejected. Module content identities must agree with the library identity. A missing contract is an error; the former module contract field remains reserved.

Producer identity remains inside the contract message as diagnostic metadata, excluded from the compatibility hash. The library content identity is separate from that hash and continues to identify the artifact used for symbol isolation.

Remaining format proposals

Portable header integers, optional complete source inputs, and native Sun object packaging remain separate work. The compatibility model still requires complete semantic and runtime requirements; changing the container does not establish stable ABI support.

Module metadata

For area(Rectangle), metadata tells the importer about the function and its rectangle declaration; bitcode holds the compiled body. A retained generic body may additionally name a private helper, so that helper’s identity must remain available even though application source cannot name it.

include/moon_bundling/moon.proto defines the current protobuf metadata, reusing AST schemas. It carries module names and visibility, declaration records and portable references, artifact identities, dependencies, target triple, static initialization order, declarations, retained generic bodies, globals, and C binding assertions.

Currently, ordinary compiled function bodies are not exported as generic source bodies. Global initializers are omitted from declaration metadata; their compiled storage and initialization code reside in the payload. Private declarations may still be needed by generic bodies and must preserve their identities and visibility.

Reading a protobuf field does not mean the consumer understands its meaning. The proposed reader checks required language features as well as field encoding; see generic semantics.

Optional complete source

Imagine these rebuild inputs (illustrative filenames):

geometry.sun        public entrypoint
rectangle.sun       private implementation
init.sun            global initialization
shape_data.sun      generated input, or a reproducible recipe to generate it

Embedding only geometry.sun is not complete source. A consumer rebuilding for another target needs the private implementation and initialization too. This source-packaging feature is proposed, not an existing rebuild guarantee.

A producer may embed all source code so a consumer can choose to rebuild the bundle with its own compiler and target settings. This can make a library usable when its existing compiled contract is unsupported. Source embedding and consumer recompilation are both optional; bundles containing only compiled code and the usual import metadata remain supported.

Complete source means more than exported generic bodies. It includes private implementations, global initializers, and source for embedded dependencies, plus the inputs needed to build them. The consumer still needs a toolchain that understands the recorded source language and can satisfy external requirements. Recompilation does not promise that arbitrary future compilers accept old source or produce identical behavior.

Minimal exampleLevelRequirement when offering complete source recompilation
A public wrapper calls a private helper that was omitted from the source package.MUSTInclude all source needed to rebuild the claimed scope, including private code, initializers, and embedded dependency code. Identify any binary-only components rather than describe them as rebuildable.
The library needs a generated file and an exact external dependency.MUSTRecord source language versions, module paths, exact dependency identities, relevant build settings, generated inputs or their reproducible generation recipe, resources, and required external tools or libraries.
The caller explicitly chooses to rebuild an otherwise compatible library.MUSTLet the consumer choose recompilation, including when the existing compiled contract is supported.
The bundle contains source, but the caller chooses its compiled payload.MUST NOTRecompile merely because source is present. A consumer may explicitly configure recompilation as a fallback after compiled-contract rejection.
A required source-language feature or generated input is unavailable.MUSTCheck source and build requirements independently of the old compiled ABI. Reject missing inputs or unsupported source semantics with a clear error.
Sun code rebuilds for AArch64, but its retained archive is x86-64.MUSTRebuild native components when their source and toolchain are available, or check the compatibility of any retained binaries.
Rebuilding geometry produces new code with a new contract.MUSTProduce a distinct artifact with one newly generated bundle contract and hash covering rebuilt code and any retained binaries. Preserve the original bundle.

The source-package encoding, build-description schema, and consumer options remain to be specified. Source files and build metadata do not introduce per-artifact ABI contracts; the resulting compiled bundle still has one contract.

Dependency and native archive handling

If geometry embeds math revision A, installing math revision B alongside it does not replace the already embedded instructions. Similarly, isolating two archives’ symbol names prevents a name collision; it does not make their allocators interchangeable.

Current bundles include linked bitcode from compiled dependencies. The dependency list is not a dynamic-loader instruction to replace that code with the newest available library. Exact referenced declaration identities remain significant when importing metadata and specializing generic bodies.

Native static archives are carried with an archive-set identity. Their symbols are isolated using that identity, and the same carried archive set can be deduplicated. This isolates names; it does not establish compatible C runtimes, allocation families, process-wide state, or platform requirements. Those remain part of the runtime and ownership contracts.

Bundle producer and compatibility contract

geometry.moon
  artifact identity: which exact library implementation is this?
  bundle contract:   what does all its compiled code require?
  producer metadata: which compiler built it?

Changing a producer label alone does not change the required layouts or calls. Changing an embedded dependency’s required runtime behavior does.

The metadata stores one bundle contract hash and the information needed to derive its complete contract. Existing type, declaration, and target metadata can supply part of that description. The experimental schema is include/abi/compatibility.proto, referenced once through MoonLibrary.bundle_contract. The container header references that bundle metadata directly. External native symbol references are part of this same protobuf (native_link_symbols) as diagnostic metadata, excluded from the compatibility hash and import authorization. Embedded dependency diagnostics are retained; no per-archive compatibility hash is introduced.

Minimal exampleLevelRequirement
Geometry embeds math code requiring an extra runtime service.MUSTCover the bundle's code, generic bodies, and all embedded dependency requirements with one contract and hash.
Only the recorded compiler build name changes.MUSTStore the bundle producer's build identity and ordering separately from the contract.
Three embedded libraries are described by the outer bundle’s contract.MUST NOTRequire separate compatibility manifests, hashes, or compiler-age metadata for embedded artifacts.
An old generic refers to a helper from one exact artifact.MUSTPreserve exact identities and references needed by declarations and generic bodies.

The compatibility procedure accepts or rejects the complete bundle contract. Older compatible bundles produce warnings. Differing experimental compiler fingerprints also warn after validation. Unsupported required contracts and concrete mismatches reject the compiled payload. If the consumer selects source recompilation, it validates the rebuilt artifact instead. The producer's version does not authorize an import.

Proposed encoding requirements

Compare two new fields: an optional build label can be skipped without changing execution; a new required interpretation of a generic operation cannot. The schema must distinguish those cases rather than reject every unfamiliar field or accept all of them.

The proposed encoding uses fixed-width little-endian container integers, byte-counted UTF-8 text, and defined offsets. Current files do not yet follow this proposal. Target payload byte order is a separate property.

Minimal exampleLevelRequirement
A reader on another host must interpret the same header integers.MUSTDefine an encoding independent of C++ padding and producer host byte order.
A reader needs to know whether a length counts bytes or entries.MUSTDefine header bytes, existing indexes, bundle metadata fields, shared ranges, strings, resource limits, and integrity checks.
An unknown debug label differs from an unknown required call rule.MUSTProvide a way to mark required capabilities without making every unknown field mandatory.
An older reader skips an added diagnostic label.MUSTUse tagged, skippable metadata fields with stable defaults. Older readers may ignore an unknown optional field only when doing so leaves import and execution behavior unchanged.
Ignoring a new field would change how a generic body is interpreted.MUSTMark new semantics as required when ignoring them would change interpretation or behavior. Readers that cannot satisfy those requirements must reject the bundle.
A field formerly storing a name must not start storing a layout.MUST NOTReuse a field identifier for a different meaning or change an existing field's meaning.
A producer adds only a safely skippable diagnostic field.MUST NOTRequire a new container revision solely for additive optional metadata that existing readers can safely skip.
A compiled-code reader does not understand the embedded source format.MUSTAllow compiled-code consumers to skip an unsupported optional source package without changing the compiled contract.
An old generic reader encounters a required unknown operation.MUSTReject unknown required features under the generic semantics contract.
A tiny file claims to contain billions of module records.MUSTValidate them before allocation.
A payload begins inside the file but ends past its last byte.MUSTCheck ranges without integer overflow; reject truncation, invalid references, and conflicting duplicate identities.
Two modules share exactly one bitcode image; an archive overlaps only half of it.MUST NOTAccept arbitrary overlap between unrelated records. Shared bitcode ranges are allowed; exact overlap rules still need a specification.
An input hash stays fixed while serialized output bytes are corrupted.MUST NOTTreat build-input identity as a checksum of serialized bytes.
Check the actual payload bytes independently of the build-input identity.MUSTDefine payload-integrity digests separately from artifact identity.

A digest does not prove that an untrusted producer followed Sun's safety rules.

Import validation order

For the geometry file, validate its byte ranges before reading its declarations, then validate those declarations and its payload before app uses them. Reading the file must not run geometry’s startup code as a side effect.

For compiled-code imports, the proposed reader performs these checks before generated code uses the bundle:

  1. Validate the header, container version, index bounds, and record encoding.
  2. Decode supported metadata and reject missing required capabilities.
  3. Validate declaration references, exact dependencies, and duplicate identities.
  4. Validate the complete bundle contract and its hash against supported implementations and the target.
  5. Validate imported declarations, layouts, and signatures under that contract before lowering dependent code.
  6. Load or link supported payloads and specialize supported generic bodies.
Minimal exampleLevelRequirement
Lazy loading must validate a function before its first use.MUSTComplete the checks above before using the compiled bundle. For selected source recompilation, validate the source inputs and resulting artifact before use. Lazy loading must reject incompatible data before use.
Listing module names must not execute library initializers.MUST NOTRun startup code merely because the container is being parsed.

Format decisions still required

  • Exact byte-level specification and migration from native header encoding.
  • Bounds, resource limits, overlap rules, and integrity coverage.
  • Whether future bundles carry native Sun objects, optional bitcode, or both.
  • Stable field identifiers, extension defaults, older readers, and supported revision lifetimes.
  • Versioned interface and generic-body schemas with reference validation.
  • Optional source-package encoding, completeness rules, build descriptions, and consumer recompilation options.
Minimal exampleLevelRequired tests
Keep a saved old bundle instead of rebuilding it with today’s compiler.MUSTRetain old artifacts in the conformance suite.
Truncate a valid file, then try an overflowing payload length.MUSTCover malformed offsets, overflowing lengths, truncation, shared blobs, duplicate records, unknown capabilities, and unsupported payloads.

Compiled payloads

Moon currently carries LLVM bitcode, declarations, and generic bodies. The initial design can keep this model.

Minimal exampleLevelRequirement
LLVM reads the module, but its Sun signature disagrees with the metadata.MUSTCover bitcode reader compatibility, Sun ABI compatibility, and generic semantics in the single bundle contract; matching one alone is insufficient.

LLVM's backward bitcode compatibility helps newer consumers load older code; it does not define Sun layouts or permit older readers to consume arbitrary future bitcode. See the LLVM compatibility policy (opens in a new tab).

Native objects with optional bitcode are a possible future packaging design. They would reduce dependence on bitcode readers while preserving the need for compatible metadata, generic specialization, platform ABI, and runtime services. No payload-format migration is required by this draft.