Consumption layer

The consumption layer is the layer that is responsible for serving the data to the user. It basically consists of two parts:

• The streaming AWS Lambda function that queries today’s data from the Curated S3 Bucket of the storage layer
• The “historical” DuckDB database file on S3 that is created and updated by the Daily Aggregation Lambda

Streaming Lambda function

The AWS Lambda function is a Function URL function that uses a Lambda layer that contains a DuckDB binary to query intra-day data from the Curated S3 Bucket of the storage layer. It streams its output to the Frontend as Apache Arrow IPC messages.

Historical DuckDB database on S3

The historical DuckDB database file is created and updated by the Daily Aggregation Lambda, and contains the historical data until the previous day. It is stored in the duckdb/data.duckdb key of the Curated S3 Bucket of the storage layer.

It is remotely attached to the frontend via the included DuckDB WASM module.