@@ -127,6 +127,51 @@ await trainer.trainAndTest({
127127});
128128```
129129
130+ ##### Loading data asynchronously
131+
132+ When working with large dataset, you might find out that the whole dataset
133+ can't fit in memory. In this situation you might want to load the data in
134+ chunks. To do this, you can define the asynchronous generators for
135+ ` trainingDataset ` , ` validationDataset ` and ` testingDataset ` .
136+
137+ This library provides the ` makeChunkedDataset ` helper to make it easier to
138+ create chunked datasets where chunks are controlled with ` skip ` and ` take `
139+ parameters.
140+
141+ ` makeChunkedDataset ` helper accepts the following parameters:
142+
143+ - ` loadChunk ` – an asynchronous function accepting the numeric ` skip ` and ` take `
144+ parameters and returning an array of samples.
145+ - ` chunkSize ` – the number of samples loaded per chunk.
146+ - ` batchSize ` – the number of samples in each batch.
147+
148+ ``` typescript
149+ const loadTrainingSamplesChunk = async (skip : number , take : number ): Promise <Array <Sample >> => {
150+ // Your samples chunk loading logic goes here. For example, you may want to
151+ // load samples from database, or from a remote data source.
152+ };
153+
154+ const makeTrainingDataset = (): data .Dataset <TensorContainer > => makeChunkedDataset ({
155+ loadChunk: loadTrainingSamplesChunk ,
156+ chunkSize: 32 ,
157+ batchSize: 32
158+ });
159+
160+ // You should also define similar functions for validationDataset and
161+ // trainingDataset. We omit this for the sake of brevity.
162+
163+ const trainingDataset = makeTrainingDataset ();
164+ const validationDataset = makeValidationDataset ();
165+ const testingDataset = makeTestingDataset ();
166+
167+ await trainer .trainAndTest ({
168+ trainingDataset ,
169+ validationDataset ,
170+ testingDataset ,
171+ printTestingResults: true
172+ });
173+ ```
174+
130175#### Saving the model
131176
132177To save the trained model, you need to call the ` save ` method of the
0 commit comments