DataFeeder(feed_list, place, program=None)
DataFeeder converts the data that returned by a reader into a data structure that can feed into Executor and ParallelExecutor. The reader usually returns a list of mini-batch data entries. Each data entry in the list is one sample. Each sample is a list or a tuple with one feature or multiple features.
The simple usage shows below:
place = fluid.CPUPlace() img = fluid.layers.data(name='image', shape=[1, 28, 28]) label = fluid.layers.data(name='label', shape=, dtype='int64') feeder = fluid.DataFeeder([img, label], fluid.CPUPlace()) result = feeder.feed([( * 784, ), ( * 784, )])
If you want to feed data into GPU side separately in advance when you use multi-GPU to train a model, you can use decorate_reader function.
place=fluid.CUDAPlace(0) feeder = fluid.DataFeeder(place=place, feed_list=[data, label]) reader = feeder.decorate_reader( paddle.batch(flowers.train(), batch_size=16))
- feed_list (list) – The Variables or Variables’name that will feed into model.
- place (Place) – place indicates feed data into CPU or GPU, if you want to feed data into GPU, please using fluid.CUDAPlace(i) (i represents the GPU id), or if you want to feed data into CPU, please using fluid.CPUPlace().
- program (Program) – The Program that will feed data into, if program is None, it will use default_main_program(). Default None.
ValueError– If some Variable is not in this Program.
# ... place = fluid.CPUPlace() feed_list = [ main_program.global_block().var(var_name) for var_name in feed_vars_name ] # feed_vars_name is a list of variables' name. feeder = fluid.DataFeeder(feed_list, place) for data in reader(): outs = exe.run(program=main_program, feed=feeder.feed(data))
According to feed_list and iterable, converters the input into a data structure that can feed into Executor and ParallelExecutor.
Parameters: iterable (list|tuple) – the input data. Returns: the result of conversion. Return type: dict
Takes multiple mini-batches. Each mini-batch will be feed on each device in advance.
- iterable (list|tuple) – the input data.
- num_places (int) – the number of devices. Default None.
the result of conversion.
The number of devices and number of mini-batches must be same.
decorate_reader(reader, multi_devices, num_places=None, drop_last=True)
Converter the input data into a data that returned by reader into multiple mini-batches. Each mini-batch will be feed on each device.
- reader (function) – the reader is the function which can generate data.
- multi_devices (bool) – whether to use multiple devices or not.
- num_places (int) – if multi_devices is True, you can specify the number of GPU to use, if multi_devices is None, the function will use all the GPU of the current machine. Default None.
- drop_last (bool) – whether to drop the last batch if the size of the last batch is less than batch_size. Default True.
the result of conversion.
ValueError– If drop_last is False and the data batch cannot fit for devices.