Skip to content

Commit 03e7334

Browse files
committed
docs: add example of loading data asynchronously to readme
1 parent 57aa746 commit 03e7334

File tree

1 file changed

+45
-0
lines changed

1 file changed

+45
-0
lines changed

README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,51 @@ await trainer.trainAndTest({
127127
});
128128
```
129129

130+
##### Loading data asynchronously
131+
132+
When working with large dataset, you might find out that the whole dataset
133+
can't fit in memory. In this situation you might want to load the data in
134+
chunks. To do this, you can define the asynchronous generators for
135+
`trainingDataset`, `validationDataset` and `testingDataset`.
136+
137+
This library provides the `makeChunkedDataset` helper to make it easier to
138+
create chunked datasets where chunks are controlled with `skip` and `take`
139+
parameters.
140+
141+
`makeChunkedDataset` helper accepts the following parameters:
142+
143+
- `loadChunk` – an asynchronous function accepting the numeric `skip` and `take`
144+
parameters and returning an array of samples.
145+
- `chunkSize` – the number of samples loaded per chunk.
146+
- `batchSize` – the number of samples in each batch.
147+
148+
```typescript
149+
const loadTrainingSamplesChunk = async (skip: number, take: number): Promise<Array<Sample>> => {
150+
// Your samples chunk loading logic goes here. For example, you may want to
151+
// load samples from database, or from a remote data source.
152+
};
153+
154+
const makeTrainingDataset = (): data.Dataset<TensorContainer> => makeChunkedDataset({
155+
loadChunk: loadTrainingSamplesChunk,
156+
chunkSize: 32,
157+
batchSize: 32
158+
});
159+
160+
// You should also define similar functions for validationDataset and
161+
// trainingDataset. We omit this for the sake of brevity.
162+
163+
const trainingDataset = makeTrainingDataset();
164+
const validationDataset = makeValidationDataset();
165+
const testingDataset = makeTestingDataset();
166+
167+
await trainer.trainAndTest({
168+
trainingDataset,
169+
validationDataset,
170+
testingDataset,
171+
printTestingResults: true
172+
});
173+
```
174+
130175
#### Saving the model
131176

132177
To save the trained model, you need to call the `save` method of the

0 commit comments

Comments
 (0)