LaudareDataset Benchmarking Framework

This project provides a framework for benchmarking various tools and models for Handwritten Text Recognition (HTR), Handwritten Music Recognition (HMR), and Layout Recognition tasks on the Laudare Dataset. It supports experiments using 5-fold cross-validation, sequential learning, and cross-manuscript setups.

Setup

Install system dependencies:

• gcc (recommended) or clang (not tested)
• Python 3 development headers
• uv package manager
• curl and jq

1. clone all submodules:

git submodule update --init --recursive

or, if you haven't cloned the repo yet:

git clone [email protected]:LaudareProject/LaudareBenchmarks.git --recursive

2. Install python dependencies using ./setup_environments.sh. This will create 3 different venvs, in the following paths:
   • .venv used for all the other models
   • .venv-calamari used for calamari (which is incompatible with kraken), inherits packages from .venv
   • .venv-kraken used for kraken (which is incompatible with calamari), inherits packages from .venv It will also download Kraken models for OCR and Layout Analysis.

Reproducibility

1. Download data: put it in the ./data directory: Link to Zenodo
2. Run the experiments: to reproduce my experiments, just run the main script:

./run.sh --task synthesis
./experiments.sh data/I-Ct_91
./experiments.sh data/I-Fn_BR_18

You can also run more specific experiments (see the Help file) and/or add your own framework.

For --n-fold runs, run.sh passes the planned fold count to result analysis. An aggregate is complete only when every requested fold has every task metric. The JSON report records status, expected_folds, available_folds, missing_folds, missing_metrics, and metrics; statistics are included only for metrics present in every requested fold. Incomplete single-framework analysis writes that report and exits nonzero. --debug --n-fold expects only two folds and is a wiring check, not reportable results.

End-to-End Pipeline Benchmarking

Start with bounded, fold-0 debug checks for OCR and OMR:

# Cascaded layout → predicted line/staff regions → PaddleOCR-VL
uv run python -m benchmarking.run_pipeline_benchmark \
  --framework doclayout_yolo+paddleocr_vl \
  --model-name default+default \
  --data-dir data/I-Ct_91 --fold 0 --edition diplomatic --task ocr --debug
uv run python -m benchmarking.run_pipeline_benchmark \
  --framework doclayout_yolo+paddleocr_vl \
  --model-name default+default \
  --data-dir data/I-Ct_91 --fold 0 --edition diplomatic --task omr --debug

# Direct full-page PaddleOCR-VL, without an external layout detector
uv run python -m benchmarking.run_pipeline_benchmark \
  --framework paddleocr_vl_e2e --model-name default \
  --data-dir data/I-Ct_91 --fold 0 --edition diplomatic --task ocr --debug
uv run python -m benchmarking.run_pipeline_benchmark \
  --framework paddleocr_vl_e2e --model-name default \
  --data-dir data/I-Ct_91 --fold 0 --edition diplomatic --task omr --debug

Direct E2E --debug predicts only the first test page per task with at most 128 new tokens, so its metrics are a wiring check, not reportable transcription accuracy. Omit --debug for a full fold. Pipeline mode stages layout detection, converts predicted PAGE-XML OCR line/TextLine regions to category 6 and OMR staff regions to category 5 in COCO JSON, runs the corresponding line/staff crops, and evaluates against the page-level test references. Regions for another task are never used as a fallback, so missing task-specific regions yield an empty crop set. E2E mode fine-tunes and predicts from full-page images; its one .pred.txt per page is compared with the evaluator’s annotation-order concatenation of the page’s ground-truth regions. Results are stored in fold-specific result directories; pipeline predictions use a pipeline_{layout_fw}+{text_fw} prefix so they do not overwrite standalone results. Use --task layout / --task ocr / --task omr to restrict the run to one stage.

3. Review Results:
   • Results are stored below results/${LAUDARE_EXPERIMENT_ID:-default}/{dataset}/{edition}/.
   • N-fold aggregates are stored in the aggregated/ subdirectory.
   • Log files for each tool and task will also be available.

Experiment Matrix GUI

gui/index.html is a dependency-free matrix of frameworks (rows) × protocols (columns) for building a custom experiments.sh. Greyed cells are impossible combinations: synthetic pretraining has no layout data, and CATMuS pretraining is OCR-only. The Generate experiments.sh button downloads a replacement script containing only the checked cells, keeping the original scaffolding. See gui/README.md for usage and tests.

Pre-training on Synthetic Data

The framework supports pre-training models on synthetically generated data before fine-tuning on real data. This can improve performance, especially when training data is limited.

For generating the synthetic data, you need gregoriotex installed and available in your PATH. A simple way is to install the following packages with your OS package manager or using the official package (in case of troubles, just install the full Texlive distribution):

• texLive-music
• texlive-latex-recommended
• texlive-luatex
• texlive-latex-extra

Generate data with the following command:

./run.sh --task synthesis

For this task, the page budget, the font directory, and the capital letter directory can also be specified (see ./run.sh --help). The runner converts each page budget to a total line target. The --pretrain-samples option sets the OMR page budget to one twentieth of the OCR budget to keep the two generation times similar. To write to another directory with exact line targets, run:

./run.sh --task synthesis --pretrain-dir data/pretrain_data_bis \
  --ocr-lines 1000000 --omr-lines 900000 --workers 10

1. Synthetic Data Generation: Creates artificial text/music images using:
   • Historical text from Biblioteca Italiana (for OCR)
   • Gregorian chant notation from GregoBase (for OMR)
   • Medieval fonts and decorative capitals
   • Realistic background textures and distortions
2. Pre-training Phase: Models are first trained on large amounts of synthetic data to learn general patterns
3. Fine-tuning Phase: Pre-trained models are then fine-tuned on your real dataset

Generated synthetic data is cached in ./data/pretrain_data/. Music rendering batches scores into one lualatex invocation and tracks each score's PDF pages explicitly. Wrapped, blank, or edge-clipped score images are rejected rather than assigned a potentially incorrect full-line label.

Music and text compositing share one pre-paste augmentation pipeline. It varies each rendered line independently in music-only and text-only pages:

• 35% receive shared geometric deformation: broad waves, shorter ripples, slant, horizontal displacement, and 5–12% variation in staff spacing. Wave amplitudes scale with the detected red-staff spacing (23–52% for the broad wave, 5–13% for ripples), rather than fixed pixels. Notes and staff share this map; its positive vertical scale retains staff order.
• Independently, 90% receive glyph-scale elastic deformation of dark ink, with displacement bounds randomly chosen from 9–25% of staff spacing and a control grid approximately one staff spacing apart. This reshapes neume or letter contours; half of those also receive fractional stroke thickening. Strong local deformation is not a guarantee that every generated glyph remains readable.
• Independently, 30% receive patchy ink fading across both black and colored strokes. Half of those also receive partial erosion for thinner strokes; erosion retains at least 65% of the original coverage, before fading, rather than deleting fine stems or staff lines.
• Adaptive transparent padding protects combined staff and ink displacements before cropping back to content.
• Half keep their native size, shrinking only when needed to fit. The rest use 65–100% of the largest aspect-preserving scale that fits the available slot.
• Half are centered; the rest randomly divide the remaining horizontal space between left and right padding, retaining a 20-pixel page margin.

Text is first rendered as transparent black ink, including decorative capitals when requested by the single-line renderer. It uses half the font's lowercase x height (minimum four pixels) as the equivalent of staff spacing, with the same strengths, probabilities, scaling, and padding as music. The resulting alpha mask is tinted with the selected text ink color before pasting, so distortions affect the ink rather than the page texture.

Bounding boxes follow the final cropped, scaled, and positioned ink. Batched compositors return COCO xywh; single-line renderers return xyxy. These augmentations affect newly generated pages only; cached synthetic data is not rewritten.

To create or extend OMR pretraining data to 900,000 total line annotations with ten workers:

uv run python -u -m benchmarking.extend_synthetic_omr --target-lines 900000 --workers 10

The OMR extender creates the split files and the annotation directory when the data directory is empty. When the directory holds earlier output, the extender appends new pages and keeps the existing pages. The extender samples contiguous 7–15-group passages at every start offset, carrying the active clef into each excerpt. Musical-label deduplication applies across existing annotations and new excerpts; changing fonts alone does not count as new music. Source-chant identity, section, offset, and length are retained as provenance. Chants with unsupported chromatic alterations or unknown notation are skipped rather than assigned incorrect natural-pitch labels. Newly generated excerpts from the same source chant have a stable split assignment; legacy data is preserved and is not retroactively re-split. The often-quoted roughly 86k non-overlapping segments are not a capacity ceiling: overlapping offsets and different lengths supply additional distinct passages. Checkpoints count annotations actually saved, not candidates enumerated, and are written every 20 images by default (--flush-images). PAGE XML is rebuilt when the run finishes. OCR data is not changed by this command.

You can enable pre-training with the flag --enable-pretraining when running experiments on real datasets. This flag will save the pretrained model in a standardized path. If you run the script run.sh again with the same --enable-pretrain flag, it will reuse the pre-trained model (if any exists), skipping the pre-training.

TrOCR-Large uses an adaptive per-device batch size capped at 8 to avoid GPU-memory exhaustion on large line-image datasets. TrOCR-Small retains its uncapped adaptive batch size.

To create or extend a synthetic OCR corpus to a total line count, run:

uv run python -m benchmarking.extend_synthetic_ocr --target-lines 1000000

The OCR extender creates the split files and the annotation directory when the data directory is empty. When the directory holds earlier output, the extender appends to the train and validation JSON files and the image directory, skips text already in those splits (after Unicode normalization, case-folding, and whitespace normalization), and creates PageXML for the resulting OCR splits. The raw Biblioteca Italiana corpus is loaded once, using the existing HTTP cache. Legal whole-word abbreviation combinations are enumerated exhaustively, smallest expansion first; lines that admit only one spelling are skipped, and a few very long paragraphs are skipped because their combination space is exponential and they are far too wide to render as a single line. Every candidate is checked against normalized text already stored or reserved. Each page selects a font and size from the same configured ranges, then gathers only lines that fit that exact choice; too-wide candidates are discarded before image rendering, rather than wasting an image batch. This changes which unique candidates are selected, not the source corpus, abbreviation rules, fonts, font-size range, or augmentation pipeline. If the finite source is exhausted before the target, the command preserves progress and errors; another seed cannot create content beyond those legal variants. A new licensed source would be needed. The OMR extender applies the analogous uniqueness key to normalized chant notation.

Image rendering uses 10 worker processes by default; --workers accepts values from 1 through 10. JSON is checkpointed every 200 generated images (adjustable with --flush-images). Both commands accept --debug, which caps a run at 50 new lines and uses one worker. Run Python with -u to see progress immediately rather than waiting for buffered output.

Use this Dataset with your Model/Framework

Quick Start

To add your custom model:

1. Create benchmarking/train_test/train_test_yourmodel.py using the template below
2. Add model configurations to benchmarking/models.json
3. Test with: python -m benchmarking.run_single_fold_benchmark --framework yourmodel --model-name "yourmodelsize" --debug --task yourtask

Template

Have a look at train_test_yolo.py and the other existing train_test_*.py scripts for fully working examples. Below is a self-explaining template.

def train_test_yourmodel(
        args=None, # the arguments parsed from command line by run_single_fold_benchmark.py
        is_train_test_mode=None, # if True, no val_json is provided, do it yourself
        is_sequential=None, # if True, this function is running as a step in sequential learning
        output_dir=None, # where to save predictions (see format below)
        train_json=None, # the COCO-format JSON file for training data
        val_json=None, # the COCO-format JSON file for validation data
        test_json=None, # the COCO-format JSON file for test data, this is None when pre-training
        save_model_path=None, # where to save the trained model (if any)
        load_model_path=None, # path to a file containing a pretrained model (if any)
        model_to_finetune=None, # an identifier (e.g. model name or path) of a base model
        # this is the one configured in models.json and has lower priority than
        # load_model_path (i.e. if load_model_path is given, it must be
        # used, not this parameter)
):
  ##################################
  # some information that can be accessed are in args, e.g. args.data_dir, args.framework,
  # args.task, args.debug, args.model_index, args.sequential_step, args.pretrained_model,
  # args.edition
  # see `uv run -m benchmarking.run_single_fold_benchmark --help` for details
  ##################################
  # Use utils.path_json2pagexml if you need PageXML files as input
  ##################################
  if args.task == "ocr":
    # load your model, see below for recommended loading patterns
    model = load_your_model_function(model_name, base_model, load_model_path, args)

    # train your model here
    model = train_your_model(train_json, val_json, model)

    save_your_model_function(model, save_model_path)

    if test_json is not None:
      # then load the best model and run predictions on the test set
      # see below for the standardized output formats
      test_predictions = predict_your_model(test_json, model)

      # save predictions in the given directory with the proper naming scheme
      # you can use annotations.ann_handler.create_new_pagexml_file(...) to easily create
      # PageXML files
      save_your_predictions(test_predictions, output_dir)

Input Data Format

The framework provides COCO-style and PageXML annotations.

Here is an example of our COCO-style JSON format:

// Example from ocr_test.json
{
  "images": [
    {
      "id": 5,                    // Unique image identifier
      "width": 2083,              // Image dimensions
      "height": 2717,
      "file_name": "c003r.png"    // Relative path from image-root
    }
  ],
  "annotations": [
    {
      "bbox": [434, 358, 1524, 141],  // MANDATORY: [x, y, width, height]
      "image_id": 2,                   // MANDATORY: Links to images[id]
      "category_id": 6,                // MANDATORY: From categories[]
      "description": "text content",   // MANDATORY for OCR/OMR
      "id": 73,                        // Unique annotation ID
      "iscrowd": 0,                    // Optional
      "area": 214884,                  // Optional
      "segmentation": []               // Optional
    }
  ],
  "categories": [                    // MANDATORY: Category definitions
    {
      "id": 6,                       // Matches category_id in annotations
      "name": "line",                // Human-readable name
      "supercategory": "layout"      // Task grouping
    }
  ]
}

In the above, note that the only non-standard field is description, which contains the text or music content for OCR/OMR tasks.

We also provide PageXML files. Since the function you need to implement receives the paths to json files, you can find the list of PageXML files corresponding to any json file with the function benchmarking.utils.path_json2pagexml(...).

Output Requirements

OCR/OMR Tasks

• Format: Text files ({image_stem}.pred.txt) or, alternatively, PageXML files.
• Content: Concatenated text predictions for all lines in image
• Example: "predicted text for line 1 predicted text for line 2"

For PageXML files, the framework parses text/music in the TextLine > TextEquiv > Unicode tags.

Layout Detection

• Format: PageXML files ({image_stem}.xml)
• Method: Use annotations.ann_handler.create_new_pagexml_file()

Model Configuration

Add to benchmarking/models.json:

"yourmodel": {
  "ocr": {
    "small": "yourmodel-small-ocr",
    "base": "yourmodel-base-ocr",
    "large": "yourmodel-large-ocr"
  },
  "omr": {
    "default": "path/to/your/omr/model"
  },
  "layout": {
    "default": "path/to/your/layout/model"
  }
}

Testing Your Implementation

remote_test.sh runs the configured experiment in a new window of the only existing tmux session on a remote host, then streams its stdout and stderr locally. Configure the target in pyproject.toml:

[tool.remote_test]
host = "gpuseeweb"
directory = "LaudareBenchmarks"
command = "~/.local/bin/mise exec uv -- ./experiments.sh data/I-Ct_91 --device cuda:1 --debug"

The remote host must have exactly one tmux session and a checkout at directory. Start and poll an experiment with:

./remote_test.sh

Polling stops after 60 seconds by default, while the tmux job continues. Resume output-only polling without launching another job:

./remote_test.sh --poll

Set a different polling duration in seconds when needed:

REMOTE_TEST_POLL_TIMEOUT=300 ./remote_test.sh --poll

Before running a remote experiment, validate the scripts locally:

bash -n remote_test.sh remote_test.remote.sh
./remote_test.sh --poll

Credits

LAUDARE PRoject -- https://laudare.eu/ ERC Advanced Grant (follow the link for details)

Framework by Federico Simonetta -- https://federicosimonetta.eu.org