| Title: | Rust Ingest Backend for GiottoDisk Stores |
|---|---|
| Description: | Prototype Rust backend that reads spatial omics inputs fast and writes GiottoDisk point tile stores, reproducing the R writer's tiles, rows and column types. One shared core (per-tile row order, tile layout, parquet writer) with one module per input format: parquet (Xenium, Atera preview, MERSCOPE, bin points) and zarr (Atera v1). The tile plan stays tilework's own; only its counting moves to Rust. |
| Authors: | Ruben Dries [aut, cre] |
| Maintainer: | Ruben Dries <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-11-17 16:47:16 UTC |
| Source: | https://github.com/giotto-suite/GiottoIngest |
The zarr input: Rust counterpart of GiottoDisk's per-grid-cell Atera v1
transcript writer (.atera_tx_cell_worker()). The caller supplies
GiottoDisk's Atera tile plan and lookups (.atera_tx_tile_plan(),
.atera_tx_lookups()). Each
grid cell is decoded, routed to its s x s store tiles and written as
tile_index=<k>/part-NNNNN-0.parquet under root; threads cells are
decoded at once in this process.
gi_atera_tx_cells( zarr_path, tiles, ords, s, tile_index, gene_names, cw_gene, root, qv_threshold = 20, flip = TRUE, window_rows = 4194304, buffer_rows = 4194304, threads = 1L )gi_atera_tx_cells( zarr_path, tiles, ords, s, tile_index, gene_names, cw_gene, root, qv_threshold = 20, flip = TRUE, window_rows = 4194304, buffer_rows = 4194304, threads = 1L )
zarr_path |
path to |
tiles |
grid cell names ( |
ords |
zero-based grid cell ordinals, for |
s |
tiles per side of each grid cell's split |
tile_index |
list of store tile indices per grid cell ( |
gene_names, cw_gene
|
the archive's gene names and codeword to gene mapping |
root |
directory the |
qv_threshold |
minimum quality score kept ( |
flip |
negate y |
window_rows |
rows decoded per read |
buffer_rows |
rows buffered per grid cell before a write |
threads |
grid cells decoded at once |
one list per grid cell: tile_index, counts, and bounds
(s^2 x 4 matrix, xmin xmax ymin ymax)
What this GiottoIngest build can do. GiottoDisk gates each fast path on its capability, so an older build without it falls back to R:
gi_capabilities() has_capability(name)gi_capabilities() has_capability(name)
name |
capability name |
"points_parquet": point tile stores from parquet sources
(gi_source(), gi_write_points()).
"points_atera_zarr": Atera v1 transcripts.zarr.zip grid cells
(gi_atera_tx_cells()).
gi_capabilities(): character vector; has_capability(): TRUE or
FALSE
Tile membership follows GiottoDisk's contiguous getTile() over the given
set of tiles.
gi_count_tiles(src, bounds, threads = 1L)gi_count_tiles(src, bounds, threads = 1L)
src |
from |
bounds |
matrix, one row per tile: xmin, xmax, ymin, ymax |
threads |
decoding threads |
numeric counts, one per tile
Rows and extent of a point source
gi_extent(src, threads = 1L)gi_extent(src, threads = 1L)
src |
from |
threads |
decoding threads |
list(n, ext), ext as xmin, xmax, ymin, ymax
Each polygon goes to the tile holding its envelope centroid, as
GiottoDisk's polygon tile write selects it; its vertex rows are written to
one parquet file per tile in row_index order.
gi_route_polygons(src, bounds, group, out_dir, tile_files, threads = 1L)gi_route_polygons(src, bounds, group, out_dir, tile_files, threads = 1L)
src |
from |
bounds |
matrix of the tiles written (non-empty plan tiles) |
group |
polygon id column |
out_dir |
directory for the per-tile files |
tile_files |
file name per tile |
threads |
threads |
list(counts, groups, unassigned): vertex rows per tile, polygons,
and polygons whose centroid fell in no tile
The lazy query a GiottoDisk reader builds, reduced to what the Rust passes need: files, coordinate columns, a y flip and numeric row filters.
gi_source(files, x, y, flip_y = FALSE, filters = list())gi_source(files, x, y, flip_y = FALSE, filters = list())
files |
parquet file paths, in the order the reader scans them |
x, y
|
coordinate columns |
flip_y |
negate y |
filters |
list of |
a list used by the gi_* functions
A tileApply() method for gi_source() objects: every tile of a pass is
counted in one Rust scan of the source, under GiottoDisk's contiguous
getTile() edge rules. With it, tilework's own
tilework::quadtreePlan() plans a point store from a source, so the plan
is GiottoDisk's by construction:
x |
|
tiles |
a |
threads |
decoding threads |
... |
ignored |
src <- gi_source(files, "x_location", "y_location", flip_y = TRUE)
plan <- tilework::quadtreePlan(src, tiles = seed, threshold = threshold,
threads = 8L)
seed is the starting tilePlan GiottoDisk builds from the source
extent (gi_extent()). Only counting is supported: FUN is ignored, and
sdimx, sdimy and contiguous (the source carries its coordinates;
the edge rules are always the contiguous ones).
a list with one count per tile
Write a point tile store's tiles
gi_write_point_tiles( src, bounds, tile_dirs, id_col, to_string = character(), drop = character(), root, geo_json, spill_rows = 4e+06, geom = TRUE, threads = 1L )gi_write_point_tiles( src, bounds, tile_dirs, id_col, to_string = character(), drop = character(), root, geo_json, spill_rows = 4e+06, geom = TRUE, threads = 1L )
src |
from |
bounds |
matrix of the tiles to write (non-empty plan tiles) |
tile_dirs |
directory name of each tile under |
id_col |
feature id column, written as |
to_string |
columns cast to string, as the reader's schema does |
drop |
columns left out |
root |
directory the tiles are written under |
geo_json |
geoparquet metadata written in every tile |
spill_rows |
rows buffered in memory before spilling to disk |
geom |
write the WKB |
threads |
threads |
numeric rows per tile
The one call a reader needs after planning: writes every non-empty tile
of plan as GiottoDisk's point store writer does, into
root/tile_index=<k>/part-0.parquet with k the tile's index in the plan
(empty tiles leave gaps, as in GiottoDisk), and the store's geoparquet
metadata in every tile.
gi_write_points( src, plan, root, id_col, to_string = character(), drop = character(), geom = TRUE, threads = 1L, spill_rows = 4e+06 )gi_write_points( src, plan, root, id_col, to_string = character(), drop = character(), geom = TRUE, threads = 1L, spill_rows = 4e+06 )
src |
from |
plan |
a |
root |
the store's |
id_col |
feature id column, written as |
to_string |
columns cast to string, as the reader's schema does |
drop |
columns left out |
geom |
write the WKB |
threads |
threads |
spill_rows |
rows buffered in memory before spilling to disk |
Only the non-empty tiles are passed on, as R writes them: the contiguous edge rules (which tiles' right and bottom edges are inclusive) are taken over that set.
list(tile_index, counts): the plan indices written and their
row counts