Streaming dataset

Watch a Dataset pipeline run.

Five small demos making @johnhenry/math-plus-data/@johnhenry/iteration's async streaming primitives visible: epoch reshuffling, prefetch overlap timing, concurrent-map ordering, sliding-window smoothing, and tee's independent consumers.

A "streaming dataset" here means data processed as an async iterable -- a sequence of items produced and consumed one (or a few) at a time, rather than loaded into memory all at once. This panel makes that normally-invisible pipeline visible by charting exactly when each item is produced or consumed, using @johnhenry/math-plus-data's Dataset (built on @johnhenry/iteration's async iteration primitives) as the engine underneath every demo below.

This matters whenever a data source is too big to hold in memory, arrives incrementally (files, network responses, sensor readings, ML training batches), or is expensive enough per item that overlapping work -- instead of doing it all strictly one step at a time -- changes how long a pipeline takes to run. The five demos below each isolate one primitive: reshuffling across epochs, overlapping produce/consume with .prefetch(), trading output order for latency with mapConcurrent, smoothing a stream with a sliding window, and splitting one stream into independently-paced consumers with tee.

Watch epochs reshuffle

A synthetic dataset of size items run through Dataset.epochs(epochCount, {reshuffle: {seed, bufferSize}}) -- each swatch is one item (color = its original position), and its position in the row is where that epoch put it.

size epochs seed buffer size

Prefetch vs. no-prefetch timing

Two pipelines process the same itemCount synthetic items, each taking produceMs to produce and consumeMs to consume. One is wrapped with .prefetch(prefetchN), overlapping the next item's production with the current item's consumption; the other isn't. The chart fills in live as each pipeline actually runs.

items produce ms consume ms prefetch n

■ with prefetch (0 arrived) ■ without prefetch (0 arrived)

Concurrent map: ordered vs. completion order

itemCount items alternate between a slow (slow ms) and fast (fast ms) simulated transform, run through mapConcurrent({concurrency}) twice -- once with the default ordered: true, once with ordered: false. Each row is one run's output sequence, left to right; color = the item's original position. Ordered always comes back 0, 1, 2, ... no matter how long an item takes; unordered lets fast items (light colors here, since fast items sit at odd original indices) overtake slow ones queued ahead of them.

items fast ms slow ms concurrency

Sliding-window smoothing

A synthetic noisy signal of n samples run through windowedAsync(values, windowSize) -- an overlapping fixed-size window over the stream -- averaged per window into a moving average. Gray is the raw signal; blue is the smoothed one. A bigger window averages over more neighbors, trading responsiveness for smoothness -- the same primitive behind streaming moving averages and rolling metrics dashboards.

samples seed noise window size

■ raw signal ■ windowed average (window 1)

Tee: independent consumers

One source producing itemCount items every produce ms is split with teeAsync(2) into two independent branches, read at different speeds (fast ms and slow ms per item). Unlike .prefetch(n) above -- which caps how far ahead the producer can get with a buffer of size n -- tee puts no bound on that buffer at all: the fast branch finishes on its own schedule, completely unaffected by how far behind the slow branch is, while every item the fast branch has already read but the slow branch hasn't sits buffered in memory until the slow branch catches up. Both branches still see every item, in the same order.

items produce ms fast ms slow ms

■ fast branch (0 arrived) ■ slow branch (0 arrived)