Skip to content

Doesn't seem to work with h5py #82

@CMCDragonkai

Description

@CMCDragonkai

I have a file I'm opening up with fs-s3fs.

This file works since I've downloaded and written to the disk.

But if I try to open up the stream in h5py, it just fails:

with fs.open_fs('s3://...').open('...', mode="rb") as weights_file:
    with h5py.File(weights_file, mode="r") as weights_hdf5:
        print(weights_hdf5)

The error is:

Traceback (most recent call last):
  File "/nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/nix/store/cpzs1hpwzs23c41haa4dap0zjfx6xych-python3-3.7.9/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/cmcdragonkai/Projects/formbay/image-classifier/image_classifier_server/services/infer.py", line 53, in __call__
    with h5py.File(weights_file, mode="r") as weights_hdf5:
  File "/nix/store/x87nsr4hhvns002c67yg43nbaknd71pn-python3.7-h5py-2.10.0/lib/python3.7/site-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/nix/store/x87nsr4hhvns002c67yg43nbaknd71pn-python3.7-h5py-2.10.0/lib/python3.7/site-packages/h5py/_hl/files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 88, in h5py.h5f.open
  File "h5py/defs.pyx", line 666, in h5py.defs.H5Fopen
  File "h5py/h5fd.pyx", line 158, in h5py.h5fd.H5FD_fileobj_read
  File "/nix/store/jws4ma2692dadgbv52l2aa3s8l9mkp8s-python3.7-fs-2.4.11/lib/python3.7/site-packages/fs/iotools.py", line 116, in readinto
    return self._f.readinto(b)
  File "/nix/store/ldqzyr3f6l8rvxq5qprfak820kh7dm7x-python3.7-fs-s3fs-1.1.1/lib/python3.7/site-packages/fs_s3fs/_s3fs.py", line 154, in readinto
    return self._f.readinto()
TypeError: readinto() takes exactly one argument (0 given)

There's something wrong with the readinto() method that h5py appears to be using.

But if I just do weights_file.read() all is fine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions