Discussion on the blueprint dccl-protobuf-overhaul registered at https://blueprints.launchpad.net/goby/+spec/dccl-protobuf-overhaul

The goal of this blueprint is to rewrite DCCL to use Google Protocol Buffers as the object typing instead of custom XML defined objects used in DCCL v1 and prior.

As opposed to a fixed header (6 bytes) + configurable body for prior DCCL, the new DCCL2 will have a minimal fixed header of 1 or 2 bytes: DCCL ID of 1 byte (if 0-127) or 2 bytes (up to 32768) using a single bit flag to self-delimit this field.

The remainder of the message will be a configurable header + configurable body. The user is now responsible for making the configurable header a nonce if using encryption (adding time somewhere is a good way to do this). As before (DCCL1), the header is unencrypted and viewable by all, the body is encrypted. The header can be used for data that need to be examined without access to the cryptographic key. As with DCCL1, all fields are encoded to the bit (byte boundaries are dissolved).

Layout of the new DCCL2 message (as a byte string: each char in the string is a single byte == 8 bits)

• LSB = most significant byte/bit (2^0)
• MSB = least significant byte/bit (2^n)

string index
0 1 2 3 4 ............... n
[header][body]

[header] is
[fixed header = (optional CCL ID) + 1 or 2 byte DCCL ID][variable header read LSB to MSB from left to right]

[body] is
[variable body reads LSB to MSB][zero padding to nearest full byte][any junk here will be ignored (above actual MSB)]

The rationale for putting the header in the LSB, followed by the body is such that any padding / junk bits that are added to the higher indices of the byte string will be ignored in decoding as decoding starts from the LSB and moves up until the message schema is satisfied.

When possible, the DCCL "field codecs" will use the schema information present in the basic Google Protocol buffers message Descriptor. To support the DCCL codec needs for additional schema information (such as maximum and minimum bounding on reals, etc.), we will use the ability to define custom option extensions for both the FieldOptions and MessageOptions. All of these will live in the package or namespace "dccl".

For example, defining

import "google/protobuf/descriptor.proto";

package dccl;

extend .google.protobuf.FieldOptions
{
  optional double min = 50012;
  optional double max = 50013;
}

Allows us to write a protobuf message with the form

message TestMsg
{
  optional double double_default = 1 [(dccl.min)=-100,
                                      (dccl.max)=126];
}

which gets compiled into the definition (Descriptor) of TestMsg and can be used by the DCCL field codecs.

This replaces the analogous DCCL1 XML file snippet

<message>
  <layout>
    <float>
      <name>double_default</name>
      <max>126</max>
      <min>-100</min>
    </float>
  </layout>
</message>

As a further configuration option, the users of DCCL2 can define their own codecs (encoder / decoders) for types or subtypes of any given message. In the absence of a custom specified codec, the defaults (same as presented in http://gobysoft.com/dl/dccl_oceans10.pdf) are used.

Syntactically, you enable a different codec as such

message TestMsg
{
  optional double double_default = 1 [(dccl.min)=-100,
                                      (dccl.max)=126,
                                      (dccl.codec)="my_better_double_codec"];
}

where the corresponding derived class of DCCLFieldCodec must be loaded at runtime with the name "my_better_double_codec" or a DCCLException will be thrown upon any calls to DCCLCodec.

Protobuf types are mapped onto DCCL types using C++ type mapping given by Google. The potential confusion is that Enums are mapped onto a const google::protobuf::EnumValueDescriptor* so we can reflect (introspect) on the enum type (which is normally not possible in C++). On top of this, all messages (derived google::protobuf::Message) can be mapped directly to their type (e.g. TestMsg) or to the base class (google::protobuf::Message) which can be inspected using the Protobuf Reflection / Descriptor interface.

DCCL1 always used fixed size codecs of arbitrary bit size. That is, a field was always N bits (which could be determined by the XML message structure). In DCCL2, variable codecs are allowed. However, variable codecs must provide a maximum size (given a Descriptor), minimum size (given a Descriptor), and actual size (given data). Fixed codecs must only provide an actual size (size == minimum size == maximum size, by definition). This allows for strict upper and lower bounding on the size of messaging which is very helpful in designing messages for a fixed datagram physical layer (e.g. the WHOI Micro-Modem)

A very basic variable sized codec could be defined as such

[exists bit = (field exists) ? true : false][field if exists bit = true]

so min size = 1, max size = 1 + size of data field

Of course, the tradeoff is overhead of 1 bit compared to the fixed size equivalent. However if the field is omitted often, this is a much better choice.

Maintaining transitional backwards compatibility for pAcommsHandler is going to be necessary. We will need a module that allows conversion from the old XML files an equivalent protobuf message than be passed to DCCL v2.

This library is most of the internals of DCCL1 renamed to "libgoby_transitional" and compiled along with the rest of the Goby MOOS modules. Thus, the Xerces dependency is removed for non-pAcommsHandler users