Repository navigation
Feat/out of core sources - #62
Merged
Merged
Conversation
`with sqlite3.connect(...)` ends the transaction but does not close the connection, so every catalog call left one open until the interpreter exited (ResourceWarning: unclosed database under -X dev). _get_connection is now a context manager that commits or rolls back and then closes. Tests fail on the previous code.
rioxarray.open_rasterio was never closed. The handle was finalised at interpreter exit and every `disscube run` that derived a variable ended with "Error in sys.excepthook:" on stderr (exit code 0, results fine). The raster is now closed in a finally block, once the aligned arrays are in memory. The test fails on the previous code.
The MapBiomas and PRODES adapters read the window and write the registered GeoTIFF a block of rows at a time (uint8, copy_window + register_file). derive() aligns a raster source a strip of grid rows at a time when it has more than block_pixels pixels (default 50 million; DISSCUBE_BLOCK_PIXELS or the block_pixels argument), for the operators whose cells depend only on the pixels inside them (Operator.block_safe). The aligner clips the band to the grid before reprojecting and takes the fine resolution from the whole source. The PRODES of the whole Legal Amazon (9.7 billion pixels) or MapBiomas over Brazil at 30 m is then one pipeline file; no new TOML syntax. Registered MapBiomas files are byte for byte those of 0.5.0. Strips equal the whole grid for class shares, majority and median with GDAL's exact transformer; with the default approximate one an edge cell can pick the neighbouring source pixel. mean and sum agree to ~1e-12. The BDC adapter still reads whole windows.
MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit percentage = [3, 4, 5, 6, 49] is the share of the cell in any of the listed classes. A MapBiomas level-1 class (Forest, Farming, Water…) is a group of leaf codes, and the coverage maps hold only leaves. Raster sources only; a vector source still takes a single class.
a coarser level
groups = {label = [codes]} replaces the codes of each group by its label before the reduction, so one
derivation gives the dominant MapBiomas level-1 class from the leaf-class map. Codes in no group and nodata
are ignored; a code listed in two groups is an error.
own code PRODES folded the hydrography of the edition into 'other' (3). With separate_water = true (TOML, or register_prodes_source/read_prodes/Scheme.lookup) it is class 4, so a map of the classes shows rivers apart from non-forest vegetation. Off by default: the local map is unchanged. Tested on a legend with a hydrography class, and on the real edition (Amazon near Santarem).
[source.codes] Replaces the separate_water flag (not released): [source.codes] (or codes= in register_prodes_source) gives the code of forest, deforested, non_forest and water; categories with one code are merged. Default unchanged (1, 2, 3, 3). docs/guides/landcover.md says how the legend is read (labels -> categories) and lists the legend of the current edition.
sized by the footprint in the source A strip of grid rows spans the whole width of the grid; in a projected grid over a longitude/latitude source (BDC Albers over the Legal Amazon, PRODES in EPSG:4674) its footprint is ~0.4 G source pixels whatever its height, so the block budget did not bound the memory (OOM, exit 137). Blocks are now rows x columns, halved along the larger side until the measured footprint in the source is within block_pixels.
residual class has the same year
The lookup merged {year: code} of the dYYYY and rYYYY classes in one dictionary, so the
residual class overwrote the yearly one for every year from 2010 and its pixels came out
as no data. The two kinds are now looped separately. Found by running Acre: accumulated
deforestation rose by 1 thousand km2 from 2008 to 2024 instead of 12 thousand.
GeoTIFF into a PNG One panel per band, with the scale chosen from the values (shares as 0 to 100 %, a few integer codes as classes, otherwise continuous). A quick look at a result, not a publication figure; needs the new optional extra [plot] (matplotlib).
clipping The reprojection can make a valid polygon self-intersect, and GeoDataFrame.clip then raises TopologyException. One bad feature among 10,000 (INCRA settlements, found in the open-lucc-workflow) stopped the whole pipeline. Repair only the invalid features after the reprojection and log how many.
Version 0.6.0 in pyproject.toml, CITATION.cff (date-released 2026-10-14) and the README BibTeX. The Unreleased section of the changelog becomes 0.6.0, with the two Fixed blocks merged. run.json now carries "disscube": the installed version (null when running from a source tree that is not installed), so a run is tied to a release without noting the commit by hand; test and docs updated. The README limitation on memory now says which sources are read by blocks (MapBiomas, PRODES) and which in memory (Brazil Data Cube item windows, all held before the reduction).
(mypy union-attr) open_rasterio is typed to return a DataArray, a Dataset or a list of Datasets; the finally block called .close() on the union and mypy reported union-attr at aligner.py:338. The finally now closes the single object or each element of the list, which is also correct for multi-subdataset files.
cannot use the exact transformer rasterio.warp.reproject has the tolerance parameter only from rasterio 1.5 (Python >= 3.12). On Python 3.11 (rasterio 1.4.4, GDAL 3.10) the fixture's tolerance=0 is ignored, the approximate transformer stays in use and a few cells of majority differ between blocks and the whole grid (8 of 1260 and 11 of 3422), which failed the test_strips_give_the_same_cells_as_the_whole_grid job on py3.11. The fixture now skips there; the changelog says where the equality holds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release 0.6.0. It lets MapBiomas and PRODES rasters of 30 m be read, aligned and aggregated over Brazil and the Legal Amazon on a 15 GiB machine, adds
disscube plot, fixes a PRODES bug that dropped the yearly classes, repairs invalid polygons, and records the DisSCube version inrun.json. Found while runningdisscube-grids,disscube-benchmarkandopen-lucc-workflow.Added
uint8(copy_window,register_file).derivealigns a raster source by blocks of rows and columns, sized by the footprint measured in the source (transform_bounds, densify 21), when it has more thanblock_pixelspixels (default 50 million;DISSCUBE_BLOCK_PIXELS). Operators markedblock_safe: mean, sum, std, median, min, max, majority, minority, percentage. No new TOML syntax.disscube plot: a PNG of the bands of an exported GeoTIFF, with--bands,--year,--cols,--dpi,--shared-colorbarand--crop. A quick look, not a publication figure. Needspip install 'disscube[plot]'.percentageaccepts a list of classes (class_code = [3, 4, 5, 6, 49]), raster sources only.majority/minorityacceptparams.groups: the map is read at a coarser level and the group label that covers most (least) of the cell is returned. A code in two groups is an error.[source.codes]declares the local code offorest,deforested,non_forestandwater;separate_watergives the hydrography its own code.run.jsonrecords the DisSCube version ("disscube";nullwhen running from a source tree that is not installed).Fixed
Scheme.lookupnow walks both tables. On Acre the accumulated deforestation goes from 27 thousand to 39 thousand km² between 2008 and 2024 (it was 27 to 28 thousand); the direct count on the raster is 27.7 and 40.2 thousand, the difference being the aggregation into cells.clipraisedTopologyException(one bad feature among 10,000 in the INCRA settlements layer). The aligner repairs invalid features (make_valid) after the reprojection and before the clip, and logs how many.Error in sys.excepthookat the end ofdisscube run. The aligner opened the source raster and never closed it. Nowtry/finallycloses it. The SQLite connections of the catalog are closed too (with sqlite3.connect()only ends the transaction).Behaviour to know
tolerance=0), cell by cell. With the default approximate transformer, up to about 11% of the cells differ by more than 1e-9 (border cells, maximum difference below 0.2). Bit-for-bit comparisons hold only with the exact one.[[derive]]of a block-read source re-reads and re-aligns it, one pass per variable and year (deriving all variables of a source in one pass is not done). The README limitation now says so.Tests
ruffclean.run.jsonversion,plot.Measured at scale (i7-7700T, 15 GiB, 1 core, Python 3.12.3)
After merging
Tag
v0.6.0, publish on PyPI, Zenodo DOI for the version.open-lucc-workflowthen moves its pin from 0.5.0 to 0.6.0 and checks that the input rasters are identical.