Record files (*.mrec)¶
A molrs record is one self-describing package on disk: a meta document
plus one or more sections — a snapshot (frame), a topology (system), a
frame sequence (trajectory), a force field (forcefield), observables, or a
run status. It is a directory whose name ends in .mrec, with every array
stored as Zarr V3, and a closed directory can be packed into one
*.mrec.zip file.
molrs writes and reads records by the molrec
contract. The layout chapter
names every group, array and attribute; this page shows how molrs maps onto
it. The examples are Python, but the Rust molrs::io::write_mrec and
molrs::io::mrec classes are the same doors with the same rules, and the WASM
package reads the same files.
| What you have | Write | Read |
|---|---|---|
A Frame (snapshot) |
molrs.io.write_mrec(sink, frame=…) |
molrs.io.mrec.MrecFile(source).frame() |
A Frame (topology) |
molrs.io.write_mrec(sink, system=…) |
MrecFile(source).system() |
A Trajectory in memory |
molrs.io.write_mrec(sink, trajectory=…) |
molrs.io.mrec.MrecReader(source).to_trajectory() |
| A run too large for memory | molrs.io.mrec.MrecWriter |
molrs.io.mrec.MrecReader |
A ForceField |
molrs.io.write_mrec(sink, forcefield=…) |
MrecFile(source).forcefield() |
| Any record | meta= on molrs.io.write_mrec |
MrecFile(source).sections(), MrecFile(source).meta() |
write_mrec takes every section as a keyword, so one call writes a snapshot
with its topology and force field. A sink is a path (pathlib.Path, written
as a directory store) or a bytearray (which receives the packed record); a
source is a path or the bytes of a packed record.
Write and read a frame¶
Build a small frame. Canonical columns (x, element, res_id, chain, …)
adopt their schema dtype on insert, so res_id below is stored as uint64
however the array was spelled.
from pathlib import Path
import numpy as np
import molrs
atoms = molrs.core.Block({
"element": ["O", "H", "H"],
"x": np.array([0.000, 0.757, -0.757]),
"y": np.array([0.000, 0.586, 0.586]),
"z": np.zeros(3),
"res_id": np.array([1, 1, 1]),
"res_name": ["HOH", "HOH", "HOH"],
"chain": ["A", "A", "A"],
})
bonds = molrs.core.Block({
"atomi": np.array([0, 0], dtype=np.uint64),
"atomj": np.array([1, 2], dtype=np.uint64),
})
frame = molrs.core.Frame(
{"atoms": atoms, "bonds": bonds},
box=molrs.core.Box.cube(20.0),
)
frame.meta["title"] = "water"
frame.meta["temperature"] = 300.0
print(frame["atoms"].dtype("res_id"))
molrs.io.write_mrec(path, frame=frame) writes the frame section and a meta document. The
meta document is what you supplied; a record carries no version key, and a
reader keeps any meta key it does not recognise.
molrs.io.write_mrec(Path("water.mrec"), frame=frame, meta={"producer": "quickstart"})
record = molrs.io.mrec.MrecFile(Path("water.mrec"))
print(record.sections())
print(record.meta())
back = record.frame()
print(back["atoms"]["chain"], back["atoms"]["res_id"])
print(back.box.lengths)
Frame meta is typed on disk: the writer adds a _meta_types attribute,
so every value reads back at the dtype it was stored with, float("nan")
included. A JSON object comes back as a frozen MetaDocument.
frame.meta["n_steps"] = molrs.core.MetaValue("i32", 5000)
frame.meta["run"] = {"ensemble": "NVT", "thermostat": "langevin"}
molrs.io.write_mrec(Path("water.mrec"), frame=frame)
back = molrs.io.mrec.MrecFile(Path("water.mrec")).frame()
print(back.meta.dtype("n_steps"), back.meta["n_steps"])
print(back.meta["run"]["ensemble"])
A topology goes in the system section, next to or instead of a snapshot:
molrs.io.write_mrec(path, frame=frame, system=topology) writes both, and
write_mrec(path, system=topology) / MrecFile(path).system() handle a
topology on its own. Each MrecFile read decodes its own section only, so a
damaged trajectory in the same store cannot fail MrecFile(path).frame(); a
section the record does not hold reads as None.
Topology conventions¶
The molrec conventions give fixed names to the residue and chain facts the
format readers produce. chain, res_id, res_name, icode, altloc,
occupancy and b_factor are canonical atoms columns; the PDB, mmCIF, GRO
and extxyz readers fill them under those names. Extra blocks with fixed
meanings are constraints, virtual_sites, drudes and members (bead →
atom membership in a coarse-grained frame). molrs.core.schema prints the whole
vocabulary.
A uint64 column can declare which block its values index. Writers store
that as the block's targets attribute and refuse a reference that does not
resolve, so a dangling index fails at write time instead of at analysis time:
contacts = molrs.core.Block({
"site": np.array([0, 2], dtype=np.uint64),
"distance": np.array([2.8, 3.1]),
})
contacts.set_target("site", "atoms")
frame["contacts"] = contacts
molrs.io.write_mrec(Path("water.mrec"), frame=frame)
print(molrs.io.mrec.MrecFile(Path("water.mrec")).frame()["contacts"].targets())
Declared precision¶
Coordinates rarely carry 16 significant digits of information. An f64
column can declare an absolute precision p in its own units. The writer then
stores each value rounded to a binary grid — the largest power of two q ≤ p,
so the error is at most p / 2 — and compresses the column with byte shuffle
plus zstd. The declaration is stored with the column and reads back.
for key in ("x", "y", "z"):
frame["atoms"].set_precision(key, 1e-3)
molrs.io.write_mrec(Path("water.mrec"), frame=frame)
back = molrs.io.mrec.MrecFile(Path("water.mrec")).frame()
print(back["atoms"].precision("x"))
print(np.abs(back["atoms"]["x"] - frame["atoms"]["x"]).max() <= 0.5e-3)
On a 3000-atom trajectory moving 0.05 Å per frame, lossless coordinates
take 24 B/atom/frame; p = 1e-3 Å takes about 7.6 and p = 1e-2 Å about
5.8. Without a declaration nothing is rounded. Memory is never touched: only
the stored copy is rounded. A store with a declared precision needs a reader
that decodes zstd and shuffle, as every molrs build does (the WASM package
included).
Trajectories¶
A Trajectory that fits in memory is written by one call and read back
through MrecReader.to_trajectory():
frames = []
for i in range(4):
f = frame.copy()
f["atoms"]["x"] = frame["atoms"]["x"] + 0.01 * i
frames.append(f)
traj = molrs.core.Trajectory(
frames,
step=np.arange(4, dtype=np.int64) * 100,
time=np.arange(4, dtype=np.float64) * 0.2,
)
molrs.io.write_mrec(Path("run.mrec"), trajectory=traj, meta={"producer": "quickstart"})
loaded = molrs.io.mrec.MrecReader(Path("run.mrec")).to_trajectory()
print(len(loaded), loaded.step)
Only what changes between frames is written: a block identical to the previous frame's is not stored again.
Streaming with a pinned schema¶
A long run is written frame by frame. Pin a SequenceSchema first — the
blocks, columns and dtypes every frame will carry — then append. Derive the
schema from representative frames (SequenceSchema.from_frames;
from_frames([]) is the empty schema) and add declarations:
declare_precision(block, column, p)— round that column on every frame, as above. A change smaller than half a grid step is not an update, so a rattling-but-static block costs nothing.declare_target(block, column, target)— auint64column indexes rows oftarget.declare_aligned(block, target)—blockhas one row per row oftargetat every frame (per-atom forces next toatoms). The writer refuses a frame that breaks it.
from molrs.io.mrec import SequenceSchema, MrecReader, MrecWriter
schema = SequenceSchema.from_frames(frames[:1])
for key in ("x", "y", "z"):
schema.declare_precision("atoms", key, 1e-3)
with MrecWriter("stream.mrec", schema, meta={"producer": "quickstart"}) as writer:
for i, f in enumerate(frames):
writer.append(f, step=100 * i, time=0.2 * i)
with MrecReader(Path("stream.mrec")) as reader:
print(len(reader), reader.step)
last = reader[len(reader) - 1]
print(last["atoms"]["x"])
MrecReader decodes one frame per access, so a store larger than
memory can be walked with a plain for frame in reader: loop. A writer
flushes every flush_every frames; if the process dies, the frames already
committed are readable and MrecWriter.open(path) resumes appending.
Packing¶
A closed directory store packs into a single *.mrec.zip — one file to copy,
upload or serve. pack_mrec_zip replaces the directory with the archive and
returns its path; MrecReader and the WASM readers open the archive directly.
from molrs.io.mrec import pack_mrec_zip
zipped = pack_mrec_zip("stream.mrec")
print(zipped)
with MrecReader(Path(zipped)) as reader:
print(len(reader))
Force fields¶
A forcefield section stores a force field as data: a document (styles,
units, mixing rule, special-bond weights) and one table per style. Units are
recorded as declared and read back as recorded. ForceFieldSection.from_forcefield and
ForceFieldSection.to_forcefield map between the two, and every record writer takes
a force field next to the structure it parameterizes:
ff = molrs.ff.forcefield.ForceField("water", units="real")
atom_style = ff.def_style("atom", "full")
o = atom_style.def_type("OW", mass=15.9994, charge=-0.8476)
h = atom_style.def_type("HW", mass=1.008, charge=0.4238)
ff.def_style("bond", "harmonic").def_type("OW-HW", o, h, k=450.0, r0=1.0)
ff.def_style("pair", "lj/cut", {"cutoff": 10.0}).def_type(
"OW", o, epsilon=0.1553, sigma=3.166
)
molrs.io.write_mrec(Path("water.mrec"), frame=frame, forcefield=ff)
record = molrs.io.mrec.MrecFile(Path("water.mrec"))
print(record.sections())
section = record.forcefield()
print(section.name, sorted(section.tables))
restored = section.to_forcefield()
print([(s.category, s.name) for s in restored.get_styles(molrs.ff.forcefield.Style)])
A pair style's table holds each atom type's own row (itom == jtom) and any
explicit cross row (itom != jtom, e.g. a CHARMM NBFIX), which prices that
type pair in place of the style's mixing rule. Both kinds round-trip. A
pair is found by its two types in either order, so a table holds one row per
pair: a section restating a pair with other parameters is refused on read.
MrecFile.forcefield() returns None for a record without a force field.
molrs.io.write_mrec(path, forcefield=ff) writes a force-field package with
no structure at all.
From Rust and the browser¶
The Rust doors have the Python names (features zarr and filesystem):
molrs::io::write_mrec(sink, &record) writes a whole MolRec (a path, or a
Vec<u8> that receives the packed record), and molrs::io::mrec::MrecFile::open(src)
reads it per section (sections(), meta(), frame(), system(),
forcefield(), record()); molrs::io::mrec also holds pack_mrec_zip,
and for streaming SequenceSchema, MrecWriter and MrecReader
(MrecReader::open(src)). Their rustdoc on docs.rs carries
compiled examples.
use std::path::Path;
use molrs::io::mrec::{MolRec, MrecFile};
use molrs::io::write_mrec;
fn main() -> std::io::Result<()> {
let mut record = MolRec::new();
record.frame = Some(molrs::core::Frame::new());
write_mrec(Path::new("water.mrec"), &record)?;
let file = MrecFile::open(Path::new("water.mrec"))?;
let frame = file.frame()?.expect("a frame section");
println!("{:?} {}", file.sections()?, frame.len());
Ok(())
}
In the browser, @molcrafts/molrs reads records from bytes:
new MrecFile(source) reads one section at a time (sections(), frame(),
system(), meta()) and new MrecReader(source) decodes trajectory frames,
each over a file map, a packed *.mrec.zip's bytes, or a lazy
range-request store. See the
package README.