RFC-0121/Consensus Encoding

Consensus Encoding

status: deprecated

Maintainer(s): Stanley Bondi

Licence

The 3-Clause BSD Licence.

Copyright 2022 The Tari Development Community

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of this document must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS DOCUMENT IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS", AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Language

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY" and "OPTIONAL" in this document are to be interpreted as described in BCP 14 (covering RFC2119 and RFC8174) when, and only when, they appear in all capitals, as shown here.

Disclaimer

This document and its content are intended for information purposes only and may be subject to change or update without notice.

This document may include preliminary concepts that may or may not be in the process of being developed by the Tari community. The release of this document is intended solely for review and discussion by the community of the technological merits of the potential system outlined herein.

Goals

The aim of this Request for Comment (RFC) is to describe the encoding used for various consensus-critical data types, as well as the construction of hash pre-images and signature challenges used in base-layer consensus.

Description

A Tari base node must validate each block containing a block header as well as set of transaction inputs, transaction outputs and transaction kernels, each containing a number of fields pertinent to their function within the [base layer]. The data contained within these structures needs to be consistently encoded (represented as bytes) across platforms and implementations so that the network can agree on a single correct state.

This RFC defines the low-level specification for how these data types MUST be encoded to construct a valid hash and signature on the Tari network.

Consensus Encoding

The primary goal of consensus encoding is to provide a consistent data format that is committed to in hashes and signatures.

Consensus encoding defines what "raw" data is included in the encoding, the order in which it should appear and the length for variable length elements. To keep encoding as simple as possible, no type information, field names etc. are catered for in the format as this is always statically known. This is particularly appropriate for hashes and signatures where many fields must be consistently represented and concatenated together.

The rest of this section defines some encodings for common primitives used in the Tari codebase.

Unsigned integer encoding

Varint encoding is used for integer fields greater than 1 byte. Describing varint is out of scope for this RFC but there are many resources online to understand this fairly basic encoding. The only rule we apply is that the encoding has a limit of 10 bytes, a little more than what is required to store a 64-bit integer.

Dynamically-sized vec encoding

This type refers to a contiguous block of data of any length. Because the size is dynamic, the size is included in the encoding.

|len(data)| data for type | data for type | ...

Fixed size arrays

If the size of the array is constant (static). The length is omitted and the data is encoded.

| data for type | ...

Optional or nullable encoding

An optional field starts with a 0x00 byte to indicate the value is not provided (None, null, nil etc) or a 0x01 byte to indicate that the value is provided followed by the encoding of the value.

| 0 or 1 | encoding for type |

Ristretto Keys

RistrettoPublicKey and RistrettoPrivateKey types defined in the tari_crypto crate both have 32-byte canonical formats and are encoded as a 32-byte fixed array.

The tari_crypto Rust crate provides an FFI interface that allows generating of the canonical byte formats in any language that supports FFI.

Commitment

A commitment is a RistrettoPublicKey and so has identical encoding.

Schnorr Signature

See the TLU on Schnorr Signatures

A Schnorr signature tuple is <R, s> where R is a RistrettoPublicKey and s is a the signature scalar wrapped in RistrettoPrivateKey.

The encoding is fixed at 64-bytes:

| 32-byte public key | 32-byte scalar |

Signature

A signature tuple consists of a <R, s> where R is the public nonce and s is the signature scalar.

The encoding is fixed at 64-bytes:

| 32-byte commitment (R) | 32-byte scalar (s) |

Commitment Signature

A commitment signature tuple consists of a <R, u, v> where R is the Pederson commitment \(r_u.G + r_v.H\) for the signature scalars u and v.

The encoding is fixed at 96-bytes:

| 32-byte commitment (R) | 32-byte scalar (u) | 32-byte scalar (v) |

Example

Given the following data and types:

{
  // Type: Fixed array of 5 bytes
  short_id: [1,2,3,4,5],
  // Type: variable length bytes
  name: Buffer.from("Case"),
  // Type: unsigned integer
  age: 40,
  // Type: struct
  details: {
      // Type: variable length bytes
      kind: Buffer.from("Hacker"),
  },
  // Type: nullable varint
  dob: null
}

Encoded (hex) as follows:

short idlennameagelenkindnull?dob
0102030405044361736528054861636b657200

Note that nested structs are flattened and the order must be preserved to allow decoding. The 00 null byte is important so that for e.g. the kind bytes cannot be manipulated to produce the same encoding as non-null dob.

Block Header

The block hash pre-image is constructed by first constructing the merge mining hash. Each encoding is concatenated in order as follows:

  1. version - 1 byte
  2. height - varint
  3. prev_hash - fixed 32-bytes
  4. timestamp - varint
  5. input_mr - fixed 32-bytes
  6. output_mr - fixed 32-bytes
  7. output_mmr_size - varint
  8. witness_mr - fixed 32-bytes
  9. kernel_mr - fixed 32-bytes
  10. kernel_mmr_size - `varint
  11. total_kernel_offset - 32-byte Scalar, see RistrettoPrivateKey
  12. total_script_offset - 32-byte Scalar, see RistrettoPrivateKey

This pre-image is hashed and block hash is constructed, in order, as follows:

  1. merge_mining_hash - As above
  2. pow_algo - enumeration of types of PoW as a single unsigned byte, where Monero = 0x00 and Sha3 = 0x01
  3. pow_data - raw variable bytes (no length varint)
  4. nonce - the PoW nonce, u64 converted to a fixed 8-byte array (little endian)

Output Features

pub struct OutputFeatures {
  pub version: OutputFeaturesVersion,
  pub maturity: u64,
  pub flags: OutputFlags,
  pub metadata: Vec<u8>,
  pub unique_id: Option<Vec<u8>>,
  pub parent_public_key: Option<PublicKey>,
  pub asset: Option<AssetOutputFeatures>,
  pub mint_non_fungible: Option<MintNonFungibleFeatures>,
  pub sidechain_checkpoint: Option<SideChainCheckpointFeatures>,
}

Output features consensus encoding is defined as follows (in order):

  1. version - 1 unsigned byte. This should always be 0x00 but is reserved for future proofing.
  2. maturity - varint
  3. flags - 1 unsigned byte
  4. metadata - dynamic vector
  5. unique_id - nullable + dynamic vector
  6. parent_public_key - nullable + 32-byte compressed public key
  7. asset - nullable + AssetOutputFeatures
  8. mint_non_fungible - nullable + MintNonFungibleFeatures
  9. sidechain_checkpoint - nullable + SideChainCheckpointFeatures
AssetOutputFeatures
MintNonFungibleFeatures
SideChainCheckpointFeatures

Transaction Output

pub struct TransactionOutput {
    pub version: TransactionInputVersion,
    pub features: OutputFeatures,
    pub commitment: Commitment,
    pub proof: RangeProof,
    pub script: TariScript,
    pub sender_offset_public_key: PublicKey,
    pub metadata_signature: ComSignature,
    pub covenant: Covenant,
}

The canonical output hash is appended to the output Merkle tree and commits to the common data between an output and the input spending that output i.e. output_hash = Hash(version | features | commitment | script | covenant).

The encoding is defined as follows:

Witness hash

The witness hash is appended to the witness Merkle tree.

  • proof - Raw proof bytes encoded using dynamic vector encoding
  • metadata_signature - [CommitmentSignature]
Metadata signature challenge

See Metadata Signature for details.

Transaction Input

The following struct represents the full transaction input data for reference. The actual input struct does not duplicate the output data to optimise storage and transmission of the input.

pub struct TransactionInput {
  pub version: u8,
  pub input_data: ExecutionStack,
  pub script_signature: ComSignature,
  pub output_version: TransactionOutputVersion,
  pub features: OutputFeatures,
  pub commitment: Commitment,
  pub script: TariScript,
  pub sender_offset_public_key: PublicKey,
  pub covenant: Covenant, 
}

The transaction input canonical hash pre-image is constructed as follows:

  • input_version - 1 byte
  • output_hash - See [TransactionOutput]
  • sender_offset_public_key - RistrettoPublicKey
  • input_data - TariScript Stack
  • script_signature - [CommitmentSignature]

Transaction Kernel

The following struct represents the full transaction input data for reference. The actual input struct does not duplicate the output data to optimise storage and transmission of the input.

pub struct TransactionKernel {
    pub version: TransactionKernelVersion,
    pub features: KernelFeatures,
    pub fee: MicroTari,
    pub lock_height: u64,
    pub excess: Commitment,
    pub excess_sig: Signature,
}

The transaction kernel is encoded as follows:

The canonical hash pre-image is constructed from this encoding.

Script Challenge

For details see RFC-0201_TariScript.md.

The script challenge is constructed as follows: