chunkFind
Use for await to iterate over chunks (pages) of results from a find method.
This function is useful for processing large datasets in batches without loading everything into memory at once. It uses pagination to fetch results in chunks, yielding each page's data array.
import { chunkFind } from 'feathers-utils/utils';Example
import { chunkFind } from 'feathers-utils/utils'
const app = feathers()
// Assuming 'users' service has many records
for await (const users of chunkFind(app, 'users', {
params: { query: { active: true }, // Custom query parameters
} })) {
console.log(users) // Process each chunk of user records
}Options
Type declaration
Show Type Declarations
type PaginateOption = {
default?: number
max?: number
}
export type ChunkFindOptions<P extends Params = Params> = {
/**
* The params of every `find` call. `query.$limit` is the page size, capped by
* `paginate.max`; `paginate` defaults to `{ default: 10, max: 100 }`.
*/
params?: P & {
paginate?: PaginateOption
}
/**
* Page by the id instead of by `$skip`: every page asks for the items after
* the last id of the page before, sorted by id.
*
* Use it when the loop changes items so that they leave the query, or when
* others write to the service meanwhile - with `$skip` the pages shift, so
* items are skipped or seen twice. It also stays linear on large tables,
* where the database reads and discards every skipped row per page, and it
* needs no `total`, so the `find` calls run with `paginate: false`.
*
* The items come sorted by id, so a `query.$sort` throws. The id is added to
* a `query.$select` that leaves it out.
*
* @default false
*/
keyset?: boolean
}
/**
* Use `for await` to iterate over chunks (pages) of results from a `find` method.
*
* This function is useful for processing large datasets in batches without loading everything into memory at once.
* It uses pagination to fetch results in chunks, yielding each page's data array.
*
* @example
* ```ts
*
*
* const app = feathers()
*
* // Assuming 'users' service has many records
* for await (const users of chunkFind(app, 'users', {
* params: { query: { active: true }, // Custom query parameters
* } })) {
* console.log(users) // Process each chunk of user records
* }
* ```
*
* @see https://utils.feathersjs.com/utils/chunk-find.html
*/
export declare function chunkFind<
Services,
Path extends KeyOf<Services>,
Service extends Services[Path] = Services[Path],
P extends Params = InferFindParams<Service>,
Item = InferFindResultSingle<Service>,
>(
app: Application<Services>,
servicePath: Path,
options?: ChunkFindOptions<P>,
): AsyncGenerator<Item[], void, unknown>| Argument | Type | Description |
|---|---|---|
| app | Application<Services> | |
| servicePath | Path | |
| options | ChunkFindOptions<P> |
Changing Items While Iterating
By default, chunkFind pages with $skip. When the loop changes items so that they no longer match the query, or others write to the service meanwhile, the pages shift: items are skipped or seen twice, without an error.
for await (const jobs of chunkFind(app, 'jobs', {
params: { query: { status: 'pending' } },
// pages by id instead of `$skip`, so every job comes exactly once
keyset: true,
})) {
const ids = jobs.map((job) => job.id)
await app
.service('jobs')
.patch(null, { status: 'done' }, { query: { id: { $in: ids } } })
}Without keyset, this loop leaves jobs pending: after the first page, those jobs no longer match, so $skip jumps over as many unprocessed ones.
With keyset: true, every page asks for the items after the last id of the page before. That also keeps large tables cheap, since the database no longer reads and discards every skipped row per page. The items come sorted by id, so query.$sort is not supported.