Skip to content

feat: fall back to bundled data.json when remote URL returns non-200#2994

Open
Aryan-Shastri wants to merge 1 commit into
sherlock-project:masterfrom
Aryan-Shastri:fix/fallback-local-data-on-remote-failure
Open

feat: fall back to bundled data.json when remote URL returns non-200#2994
Aryan-Shastri wants to merge 1 commit into
sherlock-project:masterfrom
Aryan-Shastri:fix/fallback-local-data-on-remote-failure

Conversation

@Aryan-Shastri

@Aryan-Shastri Aryan-Shastri commented Jun 10, 2026

Copy link
Copy Markdown

Problem

Users on older installations (e.g. Ubuntu 24.04 apt package shipping sherlock 0.14.3) hit a cryptic crash when the remote MANIFEST_URL returns a non-200 response:

Bad response while accessing data file URL 'https://...'

No fallback, no actionable guidance — just a hard failure.

Fix

When the remote data URL returns a non-200 response, fall back to the bundled sherlock_project/resources/data.json that ships with the package, and print a warning so the user knows what happened.

If the local fallback also doesn't exist, raise a FileNotFoundError with an actionable message (pip install -U sherlock-project) instead of a bare crash.

Behaviour after this change

  • Remote fetch succeeds → no change, works as before
  • Remote fetch returns non-200 → falls back to local data.json, prints warning
  • Local fallback missing too → clear error with upgrade instructions

Testing

# Simulate the broken old URL from #2924
from sherlock_project.sites import SitesInformation
s = SitesInformation('https://raw.githubusercontent.com/sherlock-project/sherlock/master/sherlock/resources/data.json')
# Output: Warning: Remote data unavailable (HTTP 404). Falling back to bundled local data file.
# Output: Loaded 440 sites from fallback
Closes #2924



---

When the remote MANIFEST_URL returns a non-200 response (e.g. 404 on
older installations pointing to a stale URL), try loading the bundled
sherlock_project/resources/data.json instead of crashing. Prints a
warning so users know the remote fetch failed. If the local fallback
also doesn't exist, raises a FileNotFoundError with an actionable
message suggesting `pip install -U sherlock-project`.

Fixes sherlock-project#2924
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant