Repository navigation
Upload fails with large file, works with smaller ones (nondescriptive readerror) #47
Description
Hi there,
I am using easyDataverse and thus pydvuploader.
However, when uploading a "larger" 2.1 GB file (fastq.gz), the upload stalls and fails with a "ReadError" and nothing else.
It works for smaller files in the same directory.
It also fails when renaming the file to remove the .fastq.gz
Likely unrelated: wierd filenames like "index.html?C=N;O=D" also fail. Here, renaming helps.
Any idea how to solve this?
I attached the (very long) error log, which unfortunately does not help me at all. (to avoid confusion: streamlit is the name of my conda-env, it does not run as an actual webapp at the moment, but in an .ipynb within positron.
However, running it in a standard console throws the same error -> both example runs attached)
My code (besides the console run):
`
import dvuploader as dv
path = "testProjectX/SDFSF_3232423_24R388-1_S143_L000_R2_001.fastq.gz" (large file)
"#path = "uploadCode.tmp" WORKED (small file)"
"# path = "testProjectX/exampleHtml2" # WORKED (small file) "
"# path = "testProjectX/index.html?C=D;O=D" did NOT work (small file)"
"# path = "testProjectX/R2fastqCopied.fastq.gz" NOT working either... takes too long?" (large file)
"# path = "testProjectX/R2fastqCopied" # Nope..." (large file)
"# Add file individually"
files = [
dv.File(directory_label="fastq", filepath=path, tab_ingest = False),
]
dvuploader = dv.DVUploader(files=files)
dvuploader.upload(
api_token=dataset.API_TOKEN,
dataverse_url=dataset.DATAVERSE_URL,
persistent_id=dataset.p_id,
n_parallel_uploads=1, # Whatever your instance can handle
`
Activity
JR-1991 commentedon Sep 25, 2025
@JulianRein, thank you for submitting the issue. I was able to replicate the error even locally. Upon closer inspection of the passed file handlers, including a sanity check for size and correctness, I found that the correct file is being passed to the request library. I suspect that this is not a specific issue with
python-dvuploader, but rather with eitherhttpxor Dataverse itself.I’ll check if the issue works on previous Dataverse versions. If it does, we can exclude this as the source of the problem. It’s possible that larger files are restricted because Dataverse recommends HTTP-based uploads up to 2GB. In general, it’s advisable to use the S3 direct upload (which is also supported here) to upload larger volumes, as HTTP uploads can be unreliable with large files.
I’ll get back to you once I have more information.
JulianRein commentedon Sep 25, 2025
@JR-1991 Thanks again for the quick reaction.
As a side note: I did (try to) increase the dataverse file limit to > 5GB with below command, to no avail:
JR-1991 commentedon Sep 25, 2025
@JulianRein I have added an intermediate streaming layer to track the upload progress, because somehow with the new
httpxversion this did not work as before, and it seemed to have resolved the error (local and remote). I suspect that theReadErrorhas stemmed from too frequent readouts, causing an error, but I am not sure. I will put this more to the test and will get back to you with a PR to test on your machine.JulianRein commentedon Oct 12, 2025
@JR-1991 Thanks for working on it. Any updates on the current status?
JR-1991 commentedon Oct 22, 2025
@JulianRein apologies for my delayed response, could not get back earlier. I’ve pushed the changes that resolved the error locally to a new branch and opened a new draft pull request (#48) for testing. Can you try it again with these changes?
JulianRein commentedon Oct 24, 2025
@JR-1991 thanks for the changes.
The upload progresses further now. It uploads the 2 GB test file (slight delay at 30% completion in the progress bar, but that is a minor issue), takes about 30 sec to 100% completion (local "upload").
However, it just hangs after, waiting for something?
After several minutes, if fails. Any idea why?
Here is my test code, error log attached ( tenacity.RetryError: RetryError[<Future at 0x72aa0033b530 state=finished raised ReadError>])
from easyDataverse import Dataverse \n dataverse = Dataverse( server_url="http://localhost:8080", api_token="XXXXXXX", ) dataset = dataverse.load_dataset( pid="perma:DV/AXSME6", ) dataset.add_file(local_path="testProjectX/DX130_305732_24R388-1_S143_L000_R2_001.fastq.gz", dv_path = "fastq/", tab_ingest=False) dataset.update()Note: "streamlit" is my conda env
bugfixError.txt
JR-1991 commentedon Oct 24, 2025
@JulianRein, thanks for the feedback!
Upon reviewing the files you’ve uploaded, the TSV file might trigger a dataset lock, preventing any retries initiated by
tenacity. Although you’ve explicitly stated that you don’t want to perform tabular ingestion, there was a recent bug that caused the tab ingest to fail. This bug was fixed and merged into themainbranch in #46, but this branch hasn’t been updated yet. I’ve merged themainbranch with the fix into this PR.Could you try uploading the files with and without the TSV file to ensure that this isn’t a general issue? I’ve tested a 2GB mock file on our Stuttgart instance, and the upload works. So, I believe this might be the solution to this problem. :)
JulianRein commentedon Oct 24, 2025
@JR-1991 Thanks for the quick reply!
I deleted the .tsv and the .tab from the project, see below. (I assumed that I should just test, not update to the new ingestion-fix PR).
However, the hangup still persists with the same error. Is there any use to try it on a new project? Any other ideas?
JR-1991 commentedon Oct 24, 2025
@JulianRein, no worries. I just missed updating this PR and hoped it would resolve the error.
That’s a tough one. The
ReadErrorindicates that something on the instance side seems to be causing the issue. Since you’re hosting Dataverse locally, could you share the Dataverse container logs? Perhaps we can find something in there.JR-1991 commentedon Oct 24, 2025
I managed to reproduce the error locally, and I believe it’s primarily a local development issue. When uploading a mock 2GB file, the container collapses once the upload reaches 100%. I’ve attached a screenshot of the logs, which clearly shows that the container is restarting. Furthermore, I’ve tested this with EasyDataverse using the DVUploader from this PR, but it didn’t make a difference whether I used the EasyDataverse scaffold or the pure DVUploader.
To be certain, I’ve tested this using DVCLI, and it resulted in the same crash. The crash occurs even earlier here, likely because Rust is significantly faster than Python. To be absolutely certain, I’ve used the described bash-based upload method, and it also crashes.
I suppose this is hitting the limits of local Dataverse, which may not be able to handle the speed of the upload. I’m not sure if there’s anything else we’ve missed in the local setup, but I conclude that this error is not related to DVUploader. However, I can confirm that the 2GB upload works on a true production setup.
JulianRein commentedon Oct 24, 2025
Thanks a lot for testing this. Yes, in the last run I have also seen that the server restarted after the upload failed.
With production environment, I assume you mean the non-docker install of dataverse?
I would open a dataverse-issue then.
JR-1991 commentedon Oct 24, 2025
Yes, this could be related to the container-based approach or that the local instance is simply overwhelmed by the speed of the upload. When using the direct upload locally, there are no issues because
LocalStackandminio(which are similar to S3 storage) are designed to handle the load.However, I believe that Dataverse is not designed to handle massive and fast loads and is more likely to experience slower speeds due to typically slower, realistic networks. But yeah, this is just a guess :-D
Yes, this is a great idea!
JulianRein commentedon Nov 5, 2025
I tested some more by restricting the lo interface speed with
sudo tc qdisc add dev lo root tbf rate 50mbit burst 10mbit latency 50msThe upload is now pretty slow and even, takes a long 5 min or so. Afterwards, the bug still happens (Python task not ending, server/container restarting)
—> it does not seem to be a upload-speed issue.
I also observed the Web-GUI upload failing with crash, so this really seems to be a dataverse-docker issue :/
Let's hope they are as reactive as here.