Python 3.8+pickleterm

pickle.PickleBuffer

out-of-band buffers

A wrapper around a buffer (bytes, bytearray, memoryview) that lets protocol 5 transfer large data outside the main pickle stream without copying.

Syntax

pickle.dumps(pickle.PickleBuffer(buf), protocol=5, buffer_callback=callback)

A tool for authors of libraries like NumPy rather than for everyday code. It only works with protocol=5: on older protocols dumps raises PicklingError. Without buffer_callback the buffer is simply copied into the pickle; with it the buffer is handed to the callback separately, and the data can only be loaded with loads(data, buffers=...), otherwise UnpicklingError. Loading, as with all of pickle, runs code from the data — trusted sources only.

Examples

import pickle
data = bytearray(b'big payload')
side = []
blob = pickle.dumps(pickle.PickleBuffer(data), protocol=5, buffer_callback=side.append)
print(len(side))  # → 1
print(bytes(pickle.loads(blob, buffers=side)))  # → b'big payload'
try:
    pickle.loads(blob)
except pickle.UnpicklingError as e:
    print(type(e).__name__)  # → UnpicklingError

See also

Open in the full glossarydocs.python.orgReport an inaccuracy