Comparing LLMs for User Stories on NYSE Pillar

In the second part of this series we look at using LLMs open source for writing user stories for the NYSE Pillar system.

User Stories for the NYSE you say?

For those readers who don't know, the NYSE is an exchange where you can buy and sell listed securities. The way you as a retail investor do that is typically through a broker. Your broker has servers which connect to the NYSE to facilitate this and there is a message protocol that is used to communicate between the NYSE and your broker. You can think about that protocol as an OpenAPI/Swagger spec except that instead of using REST, it's using a custom binary protocol.

Most often there isn't a machine readable spec so you can't auto-gen your code like you would with say a crypto exchange. Hopefully one day traditional financial exchanges will just offer a REST interface to the end user.

Writing these protocol handlers can be fairly mundane but prone to error, a lot of people roll their own code gen for this purpose so they can go from an intermediary representation (think of it as a table of fields) to a piece of code that can read/write those fields.

If we were writing a new protocol handler as a product shop then we'd most likely start off the process by writing user stories to ensure correct process is being followed and we end up with something that matches user needs. We may also use the user stories to validate any automated generation technique we're using.

So before we start writing our protocol handler (yes, we're going to use LLMs for that too) let's write some user stories.

First shot at writing user stories

Llama3

As a DMM (Designated Market Maker), I want to send a 
New Order Single request to create a new market order 
or limit order in the exchange book, so that my clients 
can buy or sell securities efficiently.

Acceptance Criteria:

The system shall generate a unique ClOrdID for each 
new order single request.

The system shall ensure that the ClOrdID provided is 
unique among all orders sent for the full length of 
the trading session.

The system shall send the New Order Single request as 
part of a multi-message, between GT Begin and GT End, 
along with one or more instances of the Manual Action 
Request depending on the state of the book and the 
scope of the action committed by the DMM in DBK GUI.

The system shall publish the multi-message to the 
gateway session subscribed for the symbol.

The system shall validate the SellIndicator field 
against the allowed values (Sell, Sell Short, or 
Sell Short Exempt) and reject the request if invalid.

User Story:

As a DMM, I need a way to create new market orders or limit 
orders in the exchange book efficiently. To achieve this, 
I want to send a New Order Single request that includes a 
unique ClOrdID, the symbol ID of the security, and other 
relevant information such as the order quantity and price.

The system shall ensure that the ClOrdID is unique among all 
orders sent for the full length of the trading session, so 
that I can easily track and manage my clients' orders. The 
request shall also include a multi-message format with one 
or more instances of the Manual Action Request, depending 
on the state of the book and the scope of the action 
committed by the DMM in DBK GUI.

The system shall publish the request to the gateway session 
subscribed for the symbol, allowing me to monitor the status 
of my clients' orders in real-time. Additionally, the system 
shall validate the SellIndicator field against the allowed 
values (Sell, Sell Short, or Sell Short Exempt) and reject 
the request if invalid, ensuring that only valid orders 
are executed in the exchange book.

Assumptions:

The DMM has access to a DBK GUI for managing market data.

The gateway session subscribed for the symbol is 
active and available for monitoring order status.

The SellIndicator field values (Sell, Sell Short, or Sell 
Short Exempt) are correctly populated in the request.

Mistral

Title: Sending a New Order as a Trader


As a trader or someone responsible for sending orders to the
 exchange, I want to be able to submit a new order with all 
 necessary details, so that the exchange processes my order 
 efficiently and sends relevant responses back in a timely manner.

Scenario:

I log into the trading platform and select the desired 
security symbol.

Based on market conditions, I input the following 
order details:

Order type (limit/market)

Side (buy/sell)

Quantity

Price (for limit orders)

The system validates my input and checks for any potential 
conflicts, such as overlapping or conflicting orders.

Once the order details are confirmed, I submit the order 
to the exchange via the GT stream.

The exchange receives the order message and processes 
it according to its rules and logic.

Depending on the type of order, I may receive various 
responses back from the exchange, such as:

Order confirmation message (if the order is accepted)

Rejection message (if there's a problem with the order)

Fills or fill updates (if the order gets filled partially or fully)

The trading platform displays the relevant responses to me in real-time, 
enabling me to stay informed and make any necessary adjustments accordingly.

On the face of things, the stories being generated look pretty decent, but you'd need to really dig into the NYSE Pillar spec yourself to be sure (which of course you should be doing anyway if you were to write a protocol handler for it).

I also compared user stories for cancelling orders, but I won't bore you with that too much. Suffice to say they also looked good.

As a starting point machine generated user stories may make sense, but now the product owner or a PM has to validate (and take responsibility for) these machine generated stories and as ever, the devil is in the details.

Who is even going to read these user stories?

Well, if you're creating a product then it's best it is backed up by user stories and process. In short it depends on your organisational structure - they may be useful for those with less domain specific knowledge or project managers ;-)

All jokes aside, as a developer I would prefer to have good quality user stories and acceptance criteria to support my work, so the other "person" that is going to read these stories is an LLM.

Your user stories are also a form of documentation, so you can also produce LLM assisted user-facing documentation.

Reusing User Stories

The other way LLMs can be useful is to provide base user stories which are then customised for different areas of a business. For example, the user story for a signup process is pretty similar, as is the user story for sending orders to an exchange.

In an organisation like a brokerage or an exchange you could use an LLM to refine or even define the initial user story for say sending a new order request, which is then subsequently built upon by different teams (asset class customisation for example).

As another example, if you're a startup factory or a yCombinator/funder then you may want everyone to start off using the same well validated user story for the signup process - you still give the different teams their own freedom to implement things as they see fit, but you make sure they're covering all the edge cases.

So what's next?

Write a user story to Jira tool and get cracking.

Stay tuned for the next part in the series where we'll start writing a protocol handler for the NYSE!