D7 is Photonscore's on-disk format for
time-resolved single-photon data (.photons files). This repository is its
open-source reference implementation, for the "seven" container (without the
legacy HDF5 / SQLite / directory-bundle / QAdata backends), three ways:
- the native C++ library, with a C API and its shared library;
Photonscore.D7, a managed reader and writer for .NET, on NuGet;photonscore-d7, Python bindings over the native library, on PyPI.
All three read and write the same files, byte for byte.
Every dependency of the C++ library is vendored under
third_party/. What is not, configure.py fetches: protoc
at the exact version of the vendored protobuf runtime, and the .NET SDK
when the machine has no suitable one. Each file is pinned by version and
hash, and goes inside the checkout, nowhere else.
Nothing below needs this repository built.
.NET: Photonscore.D7 on NuGet, managed code only, for .NET 8 and .NET 10:
dotnet add package Photonscore.D7Its README shows reading and writing.
Python: photonscore-d7 on PyPI, the native library inside, NumPy its only dependency; one wheel per platform (Windows x64, macOS on Apple silicon, Linux x86_64) for every Python 3:
pip install photonscore-d7Its README shows reading and writing.
C and C++: build from source, below. The C API
(c_api.h) and its shared library suit
any language with a foreign-function interface.
bootstrap.sh (Linux, macOS) and bootstrap.cmd (Windows) fetch
relocatable CPythons from
python-build-standalone,
pinned by release and SHA-256 in
third_party/python/PINS, into
third_party/python/<X.Y>/. They ask first, as configure.py does:
./bootstrap.sh # 3.12; --all for 3.11 to 3.15, --yes for CI
third_party/python/3.12/bin/python configure.py.\bootstrap.cmd
third_party\python\3.12\python.exe configure.py. ./env.sh (Bash, zsh) or .\env.cmd then put the checkout's Python
(D7_PYTHON, default 3.12) and dotnet first on PATH, and keep
dotnet's own caches in .cache/ too.
Run configure.py once per checkout. It needs Python 3.9 or later and
nothing beyond the standard library. It lists what it would download,
with sizes, and asks before fetching anything:
configure.py will download, into this checkout only:
- protoc 3.20.1 (linux-x86_64) 1.7 MB -> .cache/
- .NET SDK 10.0.403 (linux-x64) 240.2 MB -> third_party/dotnet/
- .NET runtime 8.0.33 (linux-x64) 31.3 MB -> third_party/dotnet/
total 273.2 MB
Proceed? [y/N]
| Option | Effect |
|---|---|
--no-dotnet |
Skip .NET: the native library only. |
--yes |
Download without asking, as CI does. |
A machine's own dotnet is used when it has an SDK 10 or later and the
8.0 runtime; otherwise the pinned ones go into third_party/dotnet/.
configure.py then generates the protobuf code into gen/, and on
Windows sets the projects up for the Visual Studio it finds.
Linux and macOS need a C++17 compiler and GNU make:
python3 configure.py
make -j8Windows needs Visual Studio 2019 or later with the C++ desktop workload:
python configure.py
msbuild d7.sln -m -p:Configuration=Release -p:Platform=x64d7.sln opens in Visual Studio as well. Everything built lands in out/
(out\x64\Release\ on Windows; .NET's under out/dotnet/):
| Output | What it is |
|---|---|
libd7.a / libd7.lib |
Static library: D7 together with the protobuf runtime, oneTBB and fmt. |
libphotonscore_d7.so / .dylib, photonscore_d7.dll |
Shared library exporting the pscr_d7_* C API (c_api.h). |
write_demo |
C++: create a small sample .photons file. |
list_metadata |
C++: print a file's datasets and attributes. |
read_n_events |
C++: read the first N events of a file. |
list_metadata_c_api |
list_metadata again, over the C API and the shared library. |
read_n_events_c_api |
read_n_events again, over the C API and the shared library. |
Write a small sample file, then inspect it and read it back:
out/write_demo /tmp/sample.photons
out/list_metadata /tmp/sample.photons
out/read_n_events /tmp/sample.photons 5make test runs the C++ tests (googletest) in a fresh out/test-run/;
on Windows, msbuild builds d7_test.exe too:
mkdir out\test-run && cd out\test-run && ..\x64\Release\d7_test.exeThe .NET tests read a real recording as well,
d7.photons (3.1 MB,
LnTCapture, 2019):
dotnet test dotnet/Test.D7python/d7_build.py packs the built shared library with the
photonscore_d7 package into a wheel; pip install . does the same
through it. The tests read d7.photons too:
python python/d7_build.py out/wheels
python -m pip install out/wheels/*.whl pytest
python -m pytest --basetemp=out/pytest python/testsOn macOS, build the library with MACOSX_DEPLOYMENT_TARGET=11.0, which
the wheel's tag names. PyPI's Linux wheel is built in the manylinux_2_28
image and checked by auditwheel (.github/workflows/ci.yml).
Link libd7 (above) and include the umbrella header
d7.h. A vault is created/opened with Open, then
written with CreateDataset/Append or read with Read:
#include "photonscore/lib/d7/d7.h"
#include <cstdint>
#include <memory>
#include <vector>
namespace d7 = photonscore::d7;
using photonscore::base::TypeId;
// --- write ---
{
std::unique_ptr<d7::Vault> vault;
auto status = d7::Open("scan.photons", d7::OpenOptions::ForWriting(), &vault);
// ... check status.ok() after every call ...
d7::DatasetInfo di;
di.name = "/photons/x";
di.type = TypeId::kUInt16;
di.rank = 1; // 1-D stream; size[0] grows on append
status = vault->CreateDataset(di);
std::vector<uint16_t> x = {10, 11, 13, 16, 20};
status = vault->Append("/photons/x", x.size(), x.data());
status = vault->set_attribute("instrument", "LINcam");
status = vault->Close(); // flushes the global index + epilogue
}
// --- read back ---
{
std::unique_ptr<d7::Vault> vault;
auto status = d7::Open("scan.photons", d7::OpenOptions::ForReading(), &vault);
const d7::DatasetInfo di = vault->dataset_info("/photons/x");
std::vector<uint16_t> x(di.items());
status = vault->Read("/photons/x", 0, x.size(), x.data());
}OpenOptions::ForWriting() creates a read-write file (backend "seven");
ForReading() opens read-only. See
examples/write_demo.cc,
examples/list_metadata.cc and
examples/read_n_events.cc for complete,
compilable programs. The public surface lives in
photonscore/lib/d7: d7.h, vault.h,
open_options.h, dataset_info.h.
photonscore/
lib/
base/ small utilities (Status, Slice, DateTime, TypeId)
fs/ filesystem helpers
mmap/ memory-mapped file helpers
path/ path manipulation
d7/ the D7 reader/writer + C API + seven.proto
test/ the C++ tests, and test/data/d7.photons
proto/ shared protobuf definitions (base.proto)
examples/ C++ and C-API usage examples (write_demo, list_metadata, read_n_events)
dotnet/
Photonscore.D7/ the managed reader and writer (net8.0, net10.0)
Test.D7/ its tests
python/
photonscore_d7/ the Python package, ctypes over the C API
tests/ its tests
d7_build.py the wheel builder, and pyproject.toml's backend
third_party/
fmt/ {fmt} formatting library (MIT)
oneTBB/ Intel oneAPI TBB (Apache-2.0)
protobuf/ Protocol Buffers runtime (BSD-3-Clause)
googletest/ GoogleTest, for the C++ tests (BSD-3-Clause)
dotnet/ the .NET SDK, when configure.py fetched one (not tracked)
python/ PINS, and the CPythons bootstrap fetched (not tracked)
msvc/ Visual Studio projects of d7.sln
bootstrap.* relocatable CPythons into third_party/python/
env.* the checkout's Python and dotnet on PATH, caches inside
configure.py downloads, gen/ and the Visual Studio setup
Makefile the Linux and macOS build
VERSION.txt the version of every package built here
.github/ CI on three systems; release.yaml publishes on a v* tag
out/ everything built (not tracked)
A .photons file is an append-only stream of Protocol Buffers messages laid
over a fixed 16 KB block framing. The message schema is
FileEntry; the framing and
codecs are the "seven" backend under
photonscore/lib/d7/backend/seven.
The file is a sequence of 16 KB blocks (kBlockSize = 0x4000). Each block
starts with a 2-byte header describing the fragment it carries
(log_format.h):
| Bits | Meaning |
|---|---|
| 0..13 | Fragment length |
| 14 | End of fragment |
| 15 | Begin of fragment |
A serialized message ("fragment") that fits in one block has both the begin
and end bits set. A larger message (up to kMaxFragment = 16 MB) is split
across consecutive blocks — the first carries the begin bit, the last the end
bit. When a block has exactly two spare bytes the writer emits an empty begin
marker (0x8000); with a single spare byte it pads with 0x00. This framing
lets a reader land on any block boundary and resynchronise.
Above the framing the file is just a dump of
FileEntry messages, one
after another. FileEntry is a oneof of four payloads:
Header— always the first entry (block 0). Carries the signature"D7 Photons Data", the format version, theindex_step, the page size, and a table of contents: oneDatasetInfo(name, element type, shape) per dataset.Data— one encoded page of a single dataset (see Differential encoding below). Pages are ~16 KB (kBytesPerPage); large appends are encoded page-by-page, in parallel via oneTBB, and reads decode pages in a oneTBB pipeline.Index— a map fromdataset_idto the byteoffsetof a run of pages, plus any attributes set over that span (see below).Epilogue— the final entry:global_index_offsetand the signature"End of D7 Photons Data File".
Because entries are only ever appended and never rewritten, the format is append-only / write-once: a file can be read while it is still being written, and once the epilogue is added nothing changes.
How a Data page stores its samples depends on the dataset's element type.
The relevant fields are the payload of the Data message in
seven.proto:
| Element type | Data field(s) |
Encoding |
|---|---|---|
int16/int32, uint16/uint32 |
seed + integers (repeated sint32) |
delta + zig-zag varint |
int64, uint64 |
seed + longs (repeated sint64) |
delta + zig-zag varint |
int8, uint8 |
raw_bytes (bytes) |
raw |
float, double |
floats / doubles |
raw |
For the integer types the page is stored as first-order differences — seed
holds the first sample and each subsequent element is the difference from its
predecessor
(vault_encoder.cc):
seed = x[0]
integers[i] = x[i + 1] - x[i]
Since integers/longs are protobuf sint32/sint64, they are zig-zag +
varint encoded on the wire, so the small deltas typical of photon
arrival-time and counter streams collapse to one or two bytes each. The
decoder reverses it with a running prefix sum
(vault_decoder.cc):
x[0] = seed
x[i + 1] = integers[i] + x[i]
Floating-point and byte datasets carry no diff step — they are copied
verbatim into floats / doubles / raw_bytes.
Random access needs the byte offset of every Data page, but writing one
giant index only at the very end would leave a half-written or crashed file
unreadable. The "seven" backend instead scatters partial index islands
through the stream, produced by
SevenVaultWriter::FlushIndex:
- As each page is stored, its
{dataset_id, offset}is appended to an in-memory running index and to a cumulative global index. - Every
index_stepblocks (512 blocks = 8 MB) the running index is serialised as anIndexentry, appended to the stream, and cleared. The file thus ends up dotted with small index "islands", each covering the pages written since the previous one. Attributes set viaset_attributetravel in the island for their span. - On
Closethe cumulative global index is written once (tagged with an internal marker attribute), the stream is padded to a block boundary, and theEpilogue— pointing at that global index — is written in a fresh final block.
The reader
(SevenVaultReader::InitIndex)
has two strategies:
- Global (fast path) — read the last block to get the
Epilogue, seek toglobal_index_offset, and replay theIndexentries found there. Used for cleanly closed files. - Scatter / recover — if there is no valid epilogue (the file is still
open, or was truncated by a crash) or the caller passes the
recoverbackend option, the reader hops block-to-block everyindex_stepblocks and collects the partial index islands to rebuild the page map. With thefulloption it then linearly scans anyDatapages past the last island — the tail written since the most recent flush — so even the un-indexed remainder of a live file is recoverable.
The sources here come from Photonscore's internal monorepo, zoo, which
stays their origin: the C++ library, the .NET library and their tests are
copied from it, under this repository's license header, and kept in step
by hand. VERSION.txt is zoo's version at the last sync,
or a patch above it for a release of this repository's own. What has no
counterpart there is this repository's: the build (configure.py, the
Makefile, d7.sln), the bootstrap, the Python package, the examples and
the workflows.
Pull requests are welcome. A change to a copied source is carried into zoo by hand and comes back here with the next sync.
Photonscore's own code is licensed under the Apache License 2.0 — see
LICENSE and NOTICE. Vendored third-party components
under third_party/ retain their original licenses.
D7 is developed and maintained by Photonscore GmbH, makers of time-resolved single-photon instrumentation:
- LINCam — a position-sensitive single-photon counting system.
- LINTag — an ultra-performant time-tagger.
Both produce .photons data. Learn more at
www.photonscore.de.