"keras “pickle_safe”: what does it mean to be “pickle safe”, or alternatively, “non picklable” in python?" Code Answer

i have no familiarity with keras, but from a glance at the documentation, pickle_safe just means that the tuples produced by your generator must be "picklable".

pickle is a standard python module that is used to serialize and unserialize objects. the standard multiprocessing implementation uses the pickle mechanism to share objects between different processes -- since the two processes don't share the same address space, they cannot directly see the same python objects. so, to send objects from process a to process b, they're pickled in a (which produces a sequence of bytes in a specific well-known format), the pickled format is then sent via an interprocess-communication mechanism to b, and unpickled in b, producing a copy of a's original object in b's address space.

so, to discover if your objects are picklable, just invoke, say, pickle.dumps on them.

>>> import pickle
>>> class myobject:
...    def __init__(self, a, b, c):
...      self.a = a
...      self.b = b
...      self.c = c
...
>>> foo = myobject(1, 2, 3)
>>> pickle.dumps(foo)
b'x80x03c__main__nmyobjectnqx00)x81qx01}qx02(xx01x00x00x00cqx03kx03xx01x00x00x00aqx04kx01xx01x00x00x00bqx05kx02ub.'
>>>

dumps produces a byte string. we can now reconstitute the foo object from the byte string as bar using loads:

>>> foo_pick = pickle.dumps(foo)
>>> bar = pickle.loads(foo_pick)
>>> bar
<__main__.myobject object at 0x7f5e262ece48>
>>> bar.a, bar.b, bar.c
(1, 2, 3)

if something is not picklable, you'll get an exception. for example, lambdas can't be pickled:

>>> class myother:
...   def __init__(self, a, b, c):
...     self.a = a
...     self.b = b
...     self.c = c
...     self.printer = lambda: print(self.a, self.b, self.c)
...
>>> other = myother(1, 2, 3)
>>> other_pick = pickle.dumps(other)
traceback (most recent call last):
  file "<stdin>", line 1, in <module>
attributeerror: can't pickle local object 'myother.__init__.<locals>.<lambda>'

see the documentation for more info: https://docs.python.org/3.5/library/pickle.html?highlight=pickle#what-can-be-pickled-and-unpickled