iterateFind
Use for await to iterate over the results of a find method.
This function is useful for iterating over large datasets without loading everything into memory at once. It uses pagination to fetch results in chunks, allowing you to process each item as it is retrieved.
import { iterateFind } from 'feathers-utils/utils';Example
import { iterateFind } from 'feathers-utils/utils'
const app = feathers()
// Assuming 'users' service has many records
for await (const user of iterateFind(app, 'users', {
params: { query: { active: true }, // Custom query parameters
} })) {
console.log(user) // Process each user record
}Options
Type declaration
Show Type Declarations
type PaginateOption = {
default?: number
max?: number
}
export type IterateFindOptions<P extends Params = Params> = {
/**
* The params of every `find` call. `query.$limit` is the page size, capped by
* `paginate.max`; `paginate` defaults to `{ default: 10, max: 100 }`.
*/
params?: P & {
paginate?: PaginateOption
}
/**
* Page by the id instead of by `$skip`: every page asks for the items after
* the last id of the page before, sorted by id.
*
* Use it when the loop changes items so that they leave the query, or when
* others write to the service meanwhile - with `$skip` the pages shift, so
* items are skipped or seen twice. It also stays linear on large tables,
* where the database reads and discards every skipped row per page, and it
* needs no `total`, so the `find` calls run with `paginate: false`.
*
* The items come sorted by id, so a `query.$sort` throws. The id is added to
* a `query.$select` that leaves it out.
*
* @default false
*/
keyset?: boolean
}
/**
* Use `for await` to iterate over the results of a `find` method.
*
* This function is useful for iterating over large datasets without loading everything into memory at once.
* It uses pagination to fetch results in chunks, allowing you to process each item as it is retrieved.
*
* @example
* ```ts
*
*
* const app = feathers()
*
* // Assuming 'users' service has many records
* for await (const user of iterateFind(app, 'users', {
* params: { query: { active: true }, // Custom query parameters
* } })) {
* console.log(user) // Process each user record
* }
* ```
*
* @see https://utils.feathersjs.com/utils/iterate-find.html
*/
export declare function iterateFind<
Services,
Path extends KeyOf<Services>,
Service extends Services[Path] = Services[Path],
P extends Params = InferFindParams<Service>,
Item = InferFindResultSingle<Service>,
>(
app: Application<Services>,
servicePath: Path,
options?: IterateFindOptions<P>,
): AsyncGenerator<Item, void, unknown>| Argument | Type | Description |
|---|---|---|
| app | Application<Services> | |
| servicePath | Path | |
| options | IterateFindOptions<P> |
Changing Items While Iterating
By default, iterateFind pages with $skip. When the loop changes items so that they no longer match the query, or others write to the service meanwhile, the pages shift: items are skipped or seen twice, without an error.
for await (const job of iterateFind(app, 'jobs', {
params: { query: { status: 'pending' } },
// pages by id instead of `$skip`, so every job comes exactly once
keyset: true,
})) {
await app.service('jobs').patch(job.id, { status: 'done' })
}Without keyset, this loop leaves jobs pending: after the first page, those jobs no longer match, so $skip jumps over as many unprocessed ones.
With keyset: true, every page asks for the items after the last id of the page before. That also keeps large tables cheap, since the database no longer reads and discards every skipped row per page. The items come sorted by id, so query.$sort is not supported.