feat: make impit the default HTTP client

[barjin]

Makes @crawlee/impit-client the default HTTP client for @crawlee/basic's BasicCrawler.

[janbuchar]

This makes @crawlee/basic depend on the native extensions, which restricts the supported architectures. Is that OK? I remember we discussed making impit an optional dependency and falling back to native fetch + a warning.

[barjin]

I remember we discussed making impit an optional dependency and falling back to native fetch + a warning.

Huh, I don't recall that, but it doesn't sound bad. Do you mean something like a dynamic impit import in try { } and FetchHttpClient being the catch {} branch?

[janbuchar]

I remember we discussed making impit an optional dependency and falling back to native fetch + a warning.

Huh, I don't recall that, but it doesn't sound bad. Do you mean something like a dynamic impit import in try { } and FetchHttpClient being the catch {} branch?

Yes, that's the general idea

[B4nan]

I don't remember that either, but it makes sense for sure. I would include the impit client in crawlee package, so it just works, but if you want to go with the crawler packages, it makes sense to install it explicitly. I can imagine it won't be often necessary for browser crawlers too.

[janbuchar]

So the general consensus seems to be that we should gracefully handle the case when impit is not installed. Please re-request me once that's done 🙂

[barjin]

This charade is necessary because of AdaptivePlaywrightCrawler, as it doesn't call the async init() on the HTTP-based crawler.

This way, we defer the async import of impit until the actual usage (sendRequest() callsite).

[janbuchar]

Oomph. Wouldn't calling init in AdaptivePlaywrightCrawler be less annoying? That being said, the rest of the PR is clean, and this isn't too bad, either. So treat this comment as non-blocking.

[barjin]

Agreed, but I won't deal w/ AdaptivePlaywrightCrawler internals if I can choose not to 😄

[janbuchar]

Oomph. Wouldn't calling init in AdaptivePlaywrightCrawler be less annoying? That being said, the rest of the PR is clean, and this isn't too bad, either. So treat this comment as non-blocking.