You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.

8.2 KiB

Straightway.Peerspace Chunking

In Straightway.Peerspace, data on the network layer is is transmitted in data chunks of fixed size. It is the responisbility of the transport layer to cut the data into chunks of this size and encrypt the chunks properly. This document describes the data format of the chunks.


Each data chunk has a fixed size of CHUNK_SIZE. It is subdivided into a header and the payload.

The content encoding must follow one of the encoding versions defined below. Which encoding version is used, is defined by the first byte of the data chunk header.

  • Bit 7:
    • 0: Bits 0-6 contain the version number
    • 1: Bits 0-6 are free chunk content. This is especially useful for redundancy chunks, which can use the bits 0-6 for the combined xor of the chunk version.
  • Bits 0-6: Version number or free chunk content

The further header structure is defined by the encoding version.

It may be necessary in the future to enhance the header structure. This may be realized by defining a new header version. Since chunks with all previous header version still may be around, always all header versions must be supported by the software.

Integer values are always encoded in big endian byte order.

Version 0

A version 0 chunk contains a header only consisting of the version number. The rest of the chunk is the payload.

Size in Bytes |   1||   *|
      Meaning |0x00||PYLD|

Version 1

A version 1 chunk contains a header consisting of the version number and a field MSZE (Minus SiZE), which contains the difference of the chunk size and the payload size in bytes. The rest of the chunk os the payload.

            Version|    ||Payload
Size in Bytes |   1|   1||   *|
      Meaning |0x01|MSZE||PYLD|

A version 1 chunk can be used for "almost full" payload. If you have a payload of e.g. the CHUNK_SIZE - 3, then it is impossible to encode this using a version 0 or version 2 chunk: The first one has a fixed payload size of CHUNK_SIZE - 2, the second one a maximum payload size of CHUNK_SIZE - 4. In this case, a version 1 chunk can do the job with a MSZE value of 1, meaning the actual payload size is the maximum payload size - 1, which is equal to CHUNK_SIZE - 3.

The minimum payload size for version 1 chunks is CHUNK_SIZE - 257.

Version 2

A version 2 chunk contains a header starting with a verson byte of 0x01 and a series of variable size control blocks, terminated by the CEND marker (0x00). The the payload size and the payload content follow. If the payload does not take the whole size of the chunk, it is filled up with zeros. If the chunk does not contain a CEND marker, the payload is empty by definition.

            Version||ControlBlock* ||*|CEND||Payload
              +----++----+----+----++ +----++----+----+----+
Size in Bytes |   1||   1|   2|CSZE||*|   1||   2|PSZE|   *|
              +----++----+----+----++ +----++----+----+----+
      Meaning |0x02||TYPE|SZE+|CONT||*|0x00||PSZE|PYLD|0x00|
              +----++----+----+----++ +----||----+----+----+

Each control block has the following fields:

  • TYPE: The type of control block. Must not be 0x00. For possible types see below. In the future, more types of control blocks may be defined for the version 2 chunk header.
  • SZE+: This field has two sub fields:
    • CSZE: Bits 0-11: The size of the control block content (CONT), in bytes.
    • CPLS: Bits 12-15: Can be used to store additional info defined by the block type.
  • CONT: The content of the control block, length as specified by CSZE

Public Key Control Block (0x01)

  • Type ID: 0x01
  • CPLS: Unused
  • Multiplicity: 0..1
  • Contains the public key (including an encoding of the algorithm type) used to create a signature. May refer to signature encoded in a warpping version3 chunk or to something completely different.

Referenced Chunk Control Block (0x02)

  • Type ID: 0x04
  • CPLS: Unused
  • Multiplicity: 0..*
  • Contains a reference to another data chunk. The own payload of the referencing chunk plus the combined payloads of all directly and indirectly referenced chunks (in depth-first order) is called the aggregated payload.
    • The payload of the referencing chunk is the first part, the aggregated payloads of the referenced chunks are added in the order of occurance of their referenced chunk control blocks.
    • To prevent infinitely large aggregated chunks, it is not allowed to create reference cycles. Chunks containing reference cycles shall be regarded as corrupt and be ignored.
    • Only untimed data chunks can be referenced, i.e. no data chunks being list items.

Redundancy Chunk Control Block (0x03)

  • Type ID: 0x05

  • CPLS: Unused

  • Multiplicity: 0..*, only once after a Referenced Chunk Control Block

  • Contains a reference to a redundant data chunk, allowing to reconstruct any other of the referenced chunks, if n-1 chunks referenced before this block are available. To achieve this, the redundant data chunk contains the bitwise xor combination of all referenced chunks. To also cover the control blocks of referenced chunks, the redundancy chunk is an unversioned chunk with bit 7 of the fist byte set to 1. It uses all bytes of the chunk for the xor combination, including the first 6 bits of the first byte.

    A chunk may contain more than one Redundancy Chunk Control Block. Each of these blocks refers to the references before it, either to the beginning of the chunk or until the previous Redundancy Chunk Control Block.

Version 3

A version 3 chunk contains a symmetric content key and a payload.

            Version|Cont. Key||Payload
Size in Bytes |   1|   2|KYZE||   *|
      Meaning |0x03|KYSZ|CKEY||PYLD|

Following the version byte, it has the following fields:

  • KYSZ: The size of the following asymmetric key in bytes.
  • CKEY: The asymmetric content key, including an identification of the algorith,, normally by itself encrypted, e.g. with the public part of an asymmetric key pair.
  • PYLD: The payload of the chunk, encrypted with CKEY.

The recepient must know how to decrypt the content key. The payload starts with the next byte after the key and is encrypted using that content key. The payload will usually (but not necessarily) contain the largest fitting encrypted chunk, so that it fits into the size of FULL_PAYLOAD_SIZE = CHUNK_SIZE - KYSZ - 4. The BLOCK_SIZE of the ciper algorithm may cause that the effective payload size is FULL_PAYLOAD_SIZE - FULL_PAYLOAD_SIZE % BLOCK_SIZE.

Version 4

A version 4 chunk contains a digital signature and the signed payload.

            Version|          Hash Code||Payload
Size in Bytes |   1|   1|   2|KYZE||   *|
      Meaning |0x03|SMOD|SGSZ|SIGN||PYLD|

Following the version byte, it has the following fields:

  • SMOD: The mode for verifying the signature:
    • 0x0: Signature is verifyable with a public key which must be known otherwise to the recepient
    • 0x1: Signature is verifyable with public key extracted from chunk key
    • 0x2: Signature is verifyable with a key which can be extracted from the payload (e.g. from an embedded version 2 chunk's public key control block (0x01)
    • More signature types may be added in the future.
  • SGSZ: The size of the following digital signature in bytes.
  • SIGN: The digital siganture of the payload, including an identification of the hashing algorithm.
  • PYLD: The payload of the chunk.

The payload starts with the next byte after the signature. The payload has a size of FULL_PAYLOAD_SIZE = CHUNK_SIZE - SGSZ - 6. It may contain a sub chunk of that size.