Before we Get Started...

After conferring with the author of the Telemetry library, I am convinced that it will not meet my requirements at this time. I am currently writing this library as part of the curve tracer project, but I feel that it would be best to publish it to the community as its own project. I will fork the project as soon as it fits the needs of this project and is mature enough to stand on its own. Before forking the project, I do feel that I should come up with some project name. I'm thinking something like "Dispatch" or something along those lines.

In the last post, we discussed some ways that we would like to interface with the serial library. In this post, we would like to translate some of those interfaces into bytes. We are going to go through most of the publish examples from the last post and translate that into fields, then decompose each field into bytes. We will refrain from sending negative and large numbers so that the user can visually see the translation from fields into bytes.

In all cases, the left-most field or byte is transmitted first.

The Format

The 'publish' function must have some translation to bits within a frame. Our overall organization of the frame will be:

"topic\0" (string, length not fixed)	reserved	dimensions (u4, 1 nybble)	length (u16, 2 bytes)	fs (u4, 1 nybble/dimension)	payload (series of bytes, length based on dimension, length, fs, and payload)

We should take a look at each of these in more detail:

Parameter	Description
"topic\0"	The 'topic' is a zero-terminated string. The topic is transmitted across the medium as-is, so keep the topic as short as reasonably possible.
reserved	There are 4 bits that are reserved for future use on the first byte. These are the upper bits in the byte. The 'dimensions' is in the lower nybble of this byte.
dimensions	Strings always have a dimension of 1. The number of dimensions references the number of variables that are being sent within this transmission. For instance, if (x,y) were being sent as a single transmission, then the transmission has two dimensions. The maximum number of dimensions is 15.
length	The length is the number of characters if the transmission is a string. Otherwise, it is the length of the data being sent. Note that it is the length of a single data set. For instance, if a set of (x,y) variables were being sent in a single transmission, and each variable consisted of a 10-element array, the length would be 10 and not 20.
format specifiers	The format specifier is a way for the transmitter to communicate to the receiver how the data is encoded. For each dimension, there will be one format specifier. For instance, a string has one dimension and, thus, will have one format specifier. The (x,y) data will have two dimensions and, thus, will have two format specifiers. The format specifiers are n nybble wide. When a transmision is of one dimension, the format specifier will lay on the lower nybble and the upper nybble will be unused.
payload	The payload is simply a series of bytes. In a string, the payload contains a series of ASCII characters. In a numeric example, the payload contains the data. Multidimensional data is kept intact as a block and not mixed. Meaning x₀x₁x₂...x_n then y₀y₁y₂...y_n.

Format Specifier Assignments

In the previous post, we outlined the format specifiers. In this post, we will assign numbers to the format specifier so that we can write and read the format specifier field numerically. The format specifier bit values will correspond to the "Enumerated Value" column, meaning that if a type "u8" is the format specifier, the value that will actually be sent is "2".

Format Specifier	Signed/Unsigned	Width (bytes)	Enumerated Value
(str)	-	-	1
u8	unsigned	1	2
s8	signed	1	3
u16	unsigned	2	4
s16	signed	2	5
u32	unsigned	4	6
s32	signed	4	7

Publishing Data

In this section, we are going to follow the format of the previous post so that the user can easily look back and forth.

The Simplest Example

publish("foo", "my string")

Should translate into the below fields. Each field is also broken down into its bytes. Note that the sequence is using 'little-endian' format. For instance, the 'length' field consists of 2 bytes. The lower index is the LSB while the higher index is the MSB. The same is true throughout the message.

Fields:	"foo\0"				dim = 1	length = 9		fs = STR	"my string\0"
Bytes:	'f'	'o'	'o'	'\0'	1	9	0	1	'm'	'y'	' '	's'	't'	'r'	'i'	'n'	'g'

The Simplest Numeric Example

uint8_t bar[] = {100};
publish("foo, u8", bar);

Breaks down into:

Fields:	"foo\0"				dim = 1	length = 1		fs = U8	Payload = {100}
Bytes:	'f'	'o'	'o'	'\0'	1	1	0	2	100

Something to take note of, there is a lot of overhead to sending a single byte of data per transmission! This transmission is 9 bytes long, but only actually transmits 1 byte of data! That is a transmission efficiency of 11%, very bad. That isn't entirely fair, as the topic is also part of the valid data, but you should come away knowing that sending a single piece of data per transmission represents an inefficient use of available bandwidth which may be acceptable or even necessary in your application.

Arrays

A more efficient transmission method is to send an entire array of data:

uint8_t bar[4] = {10,20,30,40};
publish("foo:4,u8", bar);

Breaks down into:

Fields:	"foo\0"				dim = 1	length = 4		fs = U8	Payload = {10,20,30,40}
Bytes:	'f'	'o'	'o'	'\0'	1	4	0	1	10	20	30	40

By adding 3 more elements, our transmission efficiency increases to 33.3%, which isn't great, but is 3x better than sending only 1 element of data. As you can see, as one sends an infinitely larger set of data, the data efficiency will approach 100%. For microcontrollers, a reasonable limit to set would be 128 elements of 8-bit data (92.1%). The library itself can currently support a limit of 65535 elements (based on the 16-bit 'length' field), which would result in a 99.986% efficiency, but this is impractical on the devices we are working with and is included in the library so that transmissions aren't limited to 255 element transmission.

Format Specifier

The format specifier determines the width of each element within the payload:

int8_t bar[] = {10};
publish("foo,s16", bar);

Fields:	"foo\0"				dim = 1	length = 1		fs = U16	Payload = {10}
Bytes:	'f'	'o'	'o'	'\0'	1	1	0	4	10	0

A string/u8/s8 format specifier implies one byte/element. A u16/s16 implies two bytes/element. A u32/s32 implies four bytes/element.

Multi-Dimensional Data

Multi-dimensional data must be able to be transmitted across the channel as a single transmission as well.

int16_t bar[] = {10};
uint16_t baz[] = {20};
publish("foo,s16,u16", bar, baz);

Fields:	"foo\0"				dim = 2	length = 1		fs[0] = S16, fs[1] = U16	Payload[0] = {10}		Payload[1] = {20}
Bytes:	'f'	'o'	'o'	'\0'	2	1	0	0x45	10	0	20	0

Putting it All Together

int16_t bar[4] = {10,20,30,40};
uint8_t baz[4] = {100,90,80,70};
publish("foo:100,s16,u8", bar, baz);

Fields:	"foo\0"				dim = 2	length = 4		fs[0] = S16, fs[1] = U8	Payload[0] = {10,20,30,40}								Payload[1] = {100,90,80,70}
Bytes:	'f'	'o'	'o'	'\0'	2	1	0	0x25	10	0	20	0	30	0	40	0	100	90	80	70

Next Up...

Up to this point, we have talked about publishing data to a particular topic, but how would the receiver get access to that data? That is the topic of Part 3 of our series!