bpo-31945: Configurable blocksize in HTTP(S)Connection (#4279)
blocksize was hardcoded to 8192, preventing efficient upload when using file-like body. Add blocksize argument to __init__, so users can configure the blocksize to fit their needs. I tested this uploading data from /dev/zero to a web server dropping the received data, to test the overhead of the HTTPConnection.send() with a file-like object. Here is an example 10g upload with the default buffer size (8192): $ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/ Uploaded 10.00g in 17.53 seconds (584.00m/s) real 0m17.574s user 0m8.887s sys 0m5.971s Same with 512k blocksize: $ time ~/src/cpython/release/python upload-httplib.py 10 https://localhost:8000/ Uploaded 10.00g in 6.60 seconds (1551.15m/s) real 0m6.641s user 0m3.426s sys 0m2.162s In real world usage the difference will be smaller, depending on the local and remote storage and the network. See https://github.com/nirs/http-bench for more info.
N
Nir Soffer committed
ad455cd9243319b896c86074ffeb3bf78a82f4ec
Parent: 30f4fa4
Committed by Victor Stinner <victor.stinner@gmail.com>
on 11/6/2017, 9:16:37 PM