Options
All
  • Public
  • Public/Protected
  • All
Menu

Interface CrawfishCloudReturnNoProto

Hierarchy

  • CrawfishCloudReturnNoProto

Index

Methods

Methods

all

  • all<T>(inp: { body: boolean; using: UsingFunc<T> }, ...filters: string[]): Promise<T[]>

  • Get a Promised Array the matched files from S3
    description

    where you can override the prior values of body and filters

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
const arr = await crab.all({body:true, using: asVfile})

    Type parameters

    • T

    Parameters

    • inp: { body: boolean; using: UsingFunc<T> }
    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Promise<T[]>

iter

  • iter<T>(inp: { NextContinuationToken?: string; body: boolean; using: UsingFunc<T> }, ...filters: string[]): AsyncGenerator<T, void, undefined>

  • Get the Async Iterator
    description

    where you can override the prior values of body and filters

    todo

    Decide how to handle overlapping paths from varrying buckets

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
for await (const vf of crab.iter({body:true, using: asVfile}) ){
console.log(vf)
}

    Type parameters

    • T

    Parameters

    • inp: { NextContinuationToken?: string; body: boolean; using: UsingFunc<T> }
      • Optional NextContinuationToken?: string
      • body: boolean
      • using: UsingFunc<T>
    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns AsyncGenerator<T, void, undefined>

reduce

  • reduce<T, U>(init: T, mapper: UsingFunc<U>, reducer: (prior: T, current: U, i: number) => T, ...filters: string[]): Promise<T>

  • Reduce the files in the S3 Globs to a result.
    description

    Reduce the files represented in the glob into a new Type. The process will batch sets of 1000 elements into memory and reduce them from there. Larger memory trades for speed since you spend less time spent setting up network connections and buffering network responses

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
const count = await crab.reduce(0, using: asS3, reducer:(p)=>p+1)

    Type parameters

    • T

    • U

    Parameters

    • init: T

      starting value for reducer

    • mapper: UsingFunc<U>
    • reducer: (prior: T, current: U, i: number) => T

      folds the set of S3Items into a new Type

        • (prior: T, current: U, i: number): T

        • Parameters

          • prior: T
          • current: U
          • i: number

          Returns T

    • Rest ...filters: string[]

      S3 url globs

    Returns Promise<T>

s3Array

  • s3Array(...filters: string[]): Promise<S3Item[]>

  • Promised S3Items Array
    description

    load the whole glob set into memory and resolve

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
const arr = await crab.s3Array()

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Promise<S3Item[]>

s3Iter

  • s3Iter(...filters: string[]): AsyncGenerator<S3Item, void, undefined>

  • Async Iterator with S3 Items - since the data starts out that way...
    description

    Asyn Generator of S3Items

    todo

    Decide how to handle overlapping paths from varrying buckets

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
for await (const s3i of crab.s3Iter('s3://ericdmoore.com-images/*.jpg') ){
console.log(s3i)
}

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns AsyncGenerator<S3Item, void, undefined>

s3Stream

  • s3Stream(...filters: string[]): Readable

  • S3 Stream
    description

    where you can override the prior values of body and filters

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
crab.s3Stream('s3://ericdmoore.com-images/*.jpg')
.pipe(S3ImageOptim())
.pipe(destinationFolder())

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Readable

stream

  • stream<T>(inp: { body: boolean; using: UsingFunc<T> }, ...filters: string[]): Readable

  • Get a Stream
    description

    where you can override the prior values of body and filters

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
crab.stream({body:true, using: asVfile})
.pipe(rehypePipe())
.pipe(destinationFolder())

    Type parameters

    • T

    Parameters

    • inp: { body: boolean; using: UsingFunc<T> }
    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Readable

vfileArray

  • vfileArray(...filters: string[]): Promise<VFile[]>

  • Promised Vfile Array
    description

    load the whole glob set into memory and resolve

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
const vfArr = await crab.vfileArray()

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Promise<VFile[]>

vfileIter

  • vfileIter(...filters: string[]): AsyncGenerator<VFile, void, undefined>

  • Async Iterator with vfiles

    description

    where you can override the prior values of body and filters

    todo

    Decide how to handle overlapping paths from varrying buckets

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
for await (const vf of crab.vfileIter('s3://ericdmoore.com-images/*.jpg') ){
console.log(vf)
}

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns AsyncGenerator<VFile, void, undefined>

vfileStream

  • vfileStream(...filters: string[]): Readable

  • Vfile Stream
    description

    within the unifiedjs ecosystem its useful to get a stream of vfiles - this method is setup for that use case.

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
crab.vfileStream('s3://ericdmoore.com-images/*.jpg')
.pipe(jpgOptim())
.pipe(destinationFolder())

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Readable

vinylArray

  • vinylArray(...filters: string[]): Promise<File[]>

  • Promised Vinyls Array
    description

    load the whole glob set into memory and resolve

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c}, 's3://ericdmoore.com-images/*.jpg')
const vArr = await crab.vinylArray()

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Promise<File[]>

vinylIter

  • vinylIter(...filters: string[]): AsyncGenerator<File, void, undefined>

  • Async Iterator with vinyls

    description

    where you can override the prior values of body and filters

    todo

    Decide how to handle overlapping paths from varrying buckets

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
for await (const v of crab.vinylIter('s3://ericdmoore.com-images/*.jpg') ){
console.log(vf)
}

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns AsyncGenerator<File, void, undefined>

vinylStream

  • vinylStream(...filters: string[]): Readable

  • Vinyl Stream
    description

    within the gulp ecosystem its useful to get a stream of vinbyls - this method is setup for that use case.

    example
    import {crawler, asVfile} from 'crawfishcloud'
import {S3, SharedIniFileCredentials} from 'aws-sdk'
const credentials = new SharedIniFileCredentials({profile:'default'})
const s3c = new S3({credentials, region:'us-west-2'})

const crab = crawler({s3c})
crab.vinylStream('s3://ericdmoore.com-images/*.jpg')
.pipe(jpgOptim())
.pipe(destinationFolder())

    Parameters

    • Rest ...filters: string[]

      Set of S3 URL glob filters

    Returns Readable

Generated using TypeDoc