Skip to main content

Intro to Aeron Part 3

· 6 min read
Sanjeev Sarda
Sanjeev Sarda
High Performance Developer

Some introductory notes I took on Aeron whilst reading their documentation. Covers flow and congestion control, status messages and the transport protocol.

Aeron

Flow and Congestion Control​

Flow Control​

Flow control is essentially about what strategy we employ to handle slow consumers/subscribers - whether backpressure is applied to publishers. You are basically deciding whether the slowest or fastest consumer should dictate the pace of publication.

Flow control uses status messages generated by media driver receivers.

There are 3 main points of flow control in an Aeron stream:

  1. Client Publication to the media driver Sender.
  2. From media driver Sender to media driver Receiver.
  3. From the media driver Receiver to the client Subscription.

Congestion Control​

Techniques for slowing down publication to prevent overrunning the network - this comes down to utilising window based techniques i.e. modifying a Receiver's window by sending status message to the media driver Sender/Subscription to tell it to slow down. This is because the amount of data the Publisher is allowed to publish is also a function of the Receiver's window length.

The default impl is a StaticWindowCongestionControl which offers no congestion control via a dynamic window, but just a static window.

Status Messages​

Generated by Subscriptions and sent to Publications periodically. Includes:

  • Term ID - term Id for highest consumed byte in the stream
  • Consumption term offset - byte offset within the active term of the highest consumed byte in the stream
  • Receiver Window - window of how much data the subscription is willing to receive.

Status messages are generated:

  • Periodically if not already sent within a certain time period
  • On term buffer rotation
  • If we've experienced sufficient growth in the stream (?)

Media Driver​

The media driver is made up of 3 Agents running their own distinct duty cycle.

The media driver deserves to be covered here in more detail (code level), but basically, communication between the media driver conductor and the client conductor occurs over an Agrona broadcast buffer and a series of control messages are exchanged in both directions which drive the core functionality of Aeron Transport.

alt text

Client Side Threading​

The Receiver and Sender on the client side are part of the client application's threading model/duty cycle. You publish messages from your own thread and are responsible for polling for messages as a subscriber.

There is a client conductor thread which runs the client side conductor duty cycle in order to exchange control messages with the media driver.

Transport Protocol​

Header Format​

The header contains:

  • Frame length
  • Version
  • Flags
  • Type - header type

alt text

Connection Setup​

To setup a stream we send a SETUP frame and we expect to get status messages back. If we're using multicast then you can use a different endpoint (group?) for data vs. control messages (NAKs and SMs).

For unicast, we expect the receiver to respond with a status message that provides us with the initial buffer status and rec window. The sender gets that status message and starts streaming data, making sure it obeys the window.

For multicast, the reciever ellicits a setup frame from the publisher when they see data frames for streams of interest.

Data Frame​

Represents an app PDU or a fragment. Uses a flag to indicate whether it's the start or end of a batch in order to support fragmentation.

Padding Frames​

Used to pad the end of a term (padding is smaller than the next message we want to send, term rotation is needed). Also used to gap fill an unblocked message. When transmitted, it is the last or only frame in a packet.

Data Recovery via Retransmit Requests​

Receiver will send a NAK to the publication on missing data. The receiver has to send the indicated frame back - this may be rate controlled. The receiver which has responded to the nak must not respond to a nak on the same frame for some time (clustering of NAKs from receivers and duplicate retransmissions).

Status Messages​

Used in flow and congestion control to define the receiver window.

RTT Measurement​

Realtime delays can vary, we may want to adjust behaviour based on RTT. We use an RTTM frame for measuring the RTT.

Receivers and senders can initiate the measurement, as well as specify a specific sender or receiver to do the measurement with. Senders initiate this via data channel, whereas receivers initiate this via the control channel.

The response is the same as the request except for a flag being removed to distinguish a response to the request - there is also a field which indicates the nanoseconds it took for the response to be generated (in order to account for message construction and sending overhead).

Lifetime and Heartbeats​

Data frames with no data, just the current position (highest sent term Id and term offset).

Name Resolution​

List of resolved entities, DNS resolution.

Aeron Over UDP​

There are 3 modes - unicast, multicast and multi-destination cast.

We have the ability through the URI to specify the endpoint as well as a distinct control details:

aeron:udp?[interface=local-interface[:local-port]|]
endpoint=receiver-address:receiver-port[|
control=explicit-control-address:control-port]

Multicast​

When using multicast, we use odd IP addresses for data, and the next even numbered IP address for control (convention or enforced?).

e.g. aeron:udp?endpoint=224.10.9.7:4050 data goes on *.7 and the 4050 port, but control goes on *.8 on port 4050

MDC​

In MDC, the senders manages the list of destinations. In manual mode, the publisher adds the destinations. In dynamic mode, destinations or subscribers add themselves to the publisher/sender's list of destinations.

Aeron over SHM​

No need for control messages over /dev/shm

Memory Ordering for Frame Writing​

Lock free techniques are used to ensure a reader knows once a write operation is completed.

  1. Atomically write frame length to be < 0
  2. SFENCE
  3. Write remaining header fields
  4. Copy in body
  5. SFENCE
  6. Atomically write length field in header to be > 0 to readers know it's available

https://github.com/aeron-io/aeron/wiki/Media-Driver-Operation - Media Driver Operations

https://github.com/aeron-io/aeron/wiki/Transport-Protocol-Specification - Transport Protocol Spec

https://github.com/aeron-io/aeron/wiki/Flow-and-Congestion-Control - Flow and Congestion Control