Financial Tick Data Pipeline

One shared header

The four programs share exactly one header, and it reads as a design document. It defines how a chunk is framed, what kinds of chunk exist, how the shared-memory region is laid out, and what every exit code means.

Two magic numbers guard the boundaries — one on each chunk header, one on the shared-memory block — so a stage that receives something unexpected can say so rather than parsing garbage confidently. It is a cheap check that turns a silent corruption into a loud failure.

The exit codes are named constants rather than numeric literals, and the signal cases follow the shell convention. That matters because the dispatcher is what reads them: a child that has died has to explain why through the only channel it has left.

Everything in shared memory is fixed-capacity — a bounded table of per-symbol entries rather than a growing one — because a region has to have a size before anybody can map it.

Inside the processor

Chunks arriving on the FIFO meet a classic bounded producer-consumer queue. Two counting semaphores track empty and full slots: the reader blocks when the queue is full, and workers block when it is empty.

There are two mutexes, not one. The queue has its own and the aggregation map has its own, so a worker folding a symbol's running totals is not holding the lock the reader needs to enqueue the next chunk. A single global lock would have been simpler and would have serialized precisely the work the thread pool exists to parallelize.

Workers are shut down with a poison pill — a chunk type that means stop — rather than a separate flag somebody has to remember to check. The drain travels the same queue as the data, so there is one shutdown mechanism instead of two that can disagree.

The signal handler does the one thing a signal handler may safely do: set a flag of the correct type and return. Everything else happens back on the main path, where it is allowed to happen.

The two functions that matter

The shared header carries two small helpers that say more about Unix experience than any amount of architecture: a write loop and a read loop.

Neither assumes the kernel will move everything it was asked to move in one call. Both loop until the byte count is satisfied, and both retry rather than fail when a call is interrupted by a signal. A single write followed by optimism is the most common way a pipeline like this quietly corrupts data on a busy machine, and handling it is the difference between code that works on the sample file and code that works under load.

The reports are produced by duplicating file descriptors rather than by opening a file and printing into it — the human-readable report is standard output, redirected — and each child's output and errors are routed to its own log. It is the same technique a shell uses, applied on purpose.

One run, end to end

A single wrapper script carries a run from nothing to a report. It checks that a compiler and make are present before it does anything, builds the four executables, launches the dispatcher with whatever arguments it was given, and summarizes what happened.

Worker count and queue size are both command-line options, which is the right shape for a project about concurrency: the two numbers a reader would most want to vary are the two the program asks for.

A sample input file is committed, so the project runs on a clean checkout without anyone having to find data first — a small courtesy that a surprising number of systems projects skip.

When a run does not finish

The more interesting path is the one where a run is interrupted. A pipeline that creates a FIFO in the filesystem, a named shared-memory object and a named semaphore has left three things behind that outlive the process, and on Linux they persist until something explicitly removes them.

So the dispatcher cleans up on interruption as well as on success. A signal sets a flag, the stages wind down, every child is reaped with a wait rather than abandoned, and the three named objects are unlinked before exit. No zombies, and nothing left behind for the next run to collide with.

The exit code carries the reason out. Bad arguments, an IPC failure, a child dying, an IO failure and each of the two termination signals all have their own named code, so a script — or a person — can tell what went wrong before opening a log.

Aggregation

The transformation itself is deliberately modest: group by symbol, and carry a handful of numbers per group.

The one worth naming is the volume-weighted average price, which weights each observed price by the volume traded at it rather than treating every tick as equally informative. One share changing hands and ten thousand changing hands are not the same evidence about what something is worth.

Computing it across a process boundary is what makes this a systems exercise rather than an arithmetic one. The shared entry stores the running price-volume sum and the running volume — not the average — because partial averages cannot be merged. The division happens once, at the end, in the process that writes the report.

Why it matters

The project is small — five files and roughly thirty kilobytes of C++ — and it is the most direct evidence in this portfolio of what I can do below the level of a framework.

Processes, interprocess transport, memory that outlives the code that created it, synchronization primitives, cleanup on the path nobody tests, and an aggregation that has to stay correct across a boundary. None of that is visible in a web application, and all of it decides whether one behaves when it is under pressure.

Known limits

The repository is public, so the left column below can be read rather than taken on trust.

The right column is short and unflattering. Nothing has been measured — no throughput, no behaviour on a large input, no profiling. There are no automated tests. And the repository carries a single commit, so its history shows a finished thing rather than how it came to be one.

For a project whose whole subject is systems behaviour, the absence of a single measured number is the honest headline, which is why it also sits in the hero rather than only down here.

Current status

The project is complete as an academic exercise, the repository is public, and it runs on a clean checkout against the committed sample input.

What it does not have is any measurement. The rows below keep that apart from what is genuinely there.