Cannot read large buffer/file from engine

[frmdstryr]

I'm trying to read a file/large string buffer from an engine . But it is returning <memory at 0x036F3B70> instead of the files data.

To reproduce:

1. Start an ipcluster
2. Create a client
3. Create a file > 1MB
4. Read the file using apply function

In [2]: from ipyparallel import Client

In [3]: c = Client()

In [4]: d = c[0]

In [5]: def read(path):
   ...:     with open(path,'rb') as f:
   ...:         return f.read(1024000)
   ...:

In [6]: r = d.apply_async(read,p)

In [7]: r.get()
Out[7]: '<memory at 0x0319E5D0>'

In [8]: def read():
   ...:     return 'a'*1024000
   ...:

In [9]: r = d.apply_async(read)

In [10]: r.get()
Out[10]: '<memory at 0x0319E580>'

This worked fine on 3.1.0.

[frmdstryr]

Also happends when sending large buffers

In [9]: def write(data):
   ...:     print(data)    
   ...:     

In [10]: r = d.apply_async(write,'a'*1024000)

In [11]: r
Out[11]: <AsyncResult: finished>

In [12]: r.get()

In [13]: r.display_outputs()
<memory at 0x0303E260

[minrk]

@frmdstryr what version of IPython and Python are you using, and what platform? I'm getting the expected bytes back.

[frmdstryr]

Python 2.7.10, ipython 4.0 (just installed with pip today)

[frmdstryr]

Windows 8.1

[phrrngtn]

I have a similar problem on Windows 7 although it is happening on strings which are not all that big (1421 bytes in fact)

In [8]: print sys.version
2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]

In [9]: print ipyparallel.version
4.0.2

In [12]: print zmq.version
14.7.0

[minrk]

Thanks, appears to be a Python 2 issue. I was testing on Python 3. Fix incoming.

[phrrngtn]

I tried upping the session limit directly in the client code:
c.session.buffer_threshold = 16 * 1024 * 1042

However, my string (1421 characters long) is getting pickled into something fancy
(Pdb) p arg_bufs[0]
"\x80\x02cipykernel.pickleutil\nCannedBytes\nq\x01)\x81q\x02}q\x03U\x07buffersq\x04]q\x05T\x8d\x05\x00\x00# [-]

I tried to customize the pickling of strings
class CannedString(CannedObject):
wrap = str

ipykernel.pickleutil.can_map[str] = CannedString

and the pickler did indeed get used:
["\x80\x02c__main__\nCannedString\nq\x01)\x81q\x02}q\x03(U\x04keysq\x04]q\x05U\x04hookq\x06NU\x03objq\x07T\x8d\x05\x00\x00#

but the remote end was unable to deal with it.

Do you have any recommendations on how to disable the fancy stuff?

[phrrngtn]

When I trimmed down my file-size to 875 bytes, the s/w works just fine.

I notice that jupyter_client.session.MAX_BYTES is set to 1024 (and that this seems to be used within the pack_apply_message function) and although I tried to increase it (monkey patching), it does not seem to work.

[minrk]

I suspect it's a buffer/memoryview confusion on Python 2.

[minrk]

Should be fixed by ipython/ipykernel#44

[minrk]

If you want a mitigation while you wait for that patch to get a release, you can apply this patch at runtime:

def patch_issue_30():
    """"CannedBytes.wrap should turn buffers into bytes"""
    from ipykernel.pickleutil import CannedBytes
    from ipython_genutils.py3compat import buffer_to_bytes
    CannedBytes.wrap = staticmethod(buffer_to_bytes)

patch_issue_30() # patch locally
rc[:].apply_sync(patch_issue_30) # patch on all engines

[phrrngtn]

Confirmed working. awesome! Thanks very much!

[frmdstryr]

Works now, thanks!