DN42Atlas

Why I built it

DN42 has a lot of small websites, tools, looking glasses, personal pages, and strange little services scattered around it. There are existing discovery tools, but I kept running into two problems: one of the better-known options was unavailable often enough to be frustrating, and its index still missed plenty of things.

I wanted something I could run myself that started from the DN42 registry and built a fresh picture of what was actually reachable. The goal was not to make a scanner for scanner's sake. I wanted to end up with something I could casually browse and use to discover corners of the network I would otherwise never see.

Some other discovery projects go further and render or screenshot the pages they find, which is undeniably cooler. DN42Atlas takes a more boring route: collect enough metadata to make discovery useful, keep the probing bounded, and make the result simple to publish.

From registry data to something browseable

Atlas starts with registered .dn42 names, resolves them through a DN42-aware resolver, and probes a curated set of HTTP and HTTPS endpoints. For reachable services it can record basic metadata such as page titles, redirects, content types, discovered links, and referenced .dn42 hostnames.

The scan output is written as JSON. Publication then filters that raw result and turns it into a self-contained HTML viewer. That separation keeps network activity out of the publication step: generating or regenerating the public atlas does not need to resolve names or probe services again.

The static site was an accident, then a feature

I did not start the project with "make the whole thing static" as a requirement. That fell out of the architecture naturally.

Once the crawler was producing a complete scan snapshot, there was no good reason for the public viewer to query a live database. The report step could take the saved JSON, filter it, and generate everything the browser needed ahead of time.

That makes the public listing cheap and boring to host. There is no database in the crawler-to-publication path, no query service required just to browse the results, and no reason for a page view to trigger more DN42 traffic.

The separate operator/opt-out service keeps its own private state, but the discovery and public browsing pipeline does not depend on a live database.

Polite discovery, not a vulnerability scanner

Atlas deliberately does less than a general-purpose crawler or scanner. It checks a curated set of web endpoints, requests /robots.txt, and fetches the homepage only when policy allows it. It does not recursively follow links, brute-force paths, attempt authentication, or run vulnerability checks.

Concurrency, timeouts, response sizes, and link counts are bounded. If robots rules cannot be evaluated safely, the homepage is not fetched. Discovered links are useful metadata for the atlas, not instructions to immediately crawl deeper.

That keeps the project aligned with what I actually wanted: discover public DN42 services without behaving like I am trying to audit them.

DNS is input, not authority

A scanner that accepts hostnames from outside itself needs to be careful about where those names can send it. DN42Atlas therefore does not simply resolve a hostname and hand it to a normal HTTP client.

Before probing, the saved addresses must fall inside expected DN42 address space. A hostname with mixed DN42 and external results is rejected. The HTTP connection is then pinned to the validated address set while still using the hostname for the HTTP Host header and TLS SNI.

That means a later DNS change cannot silently redirect an existing scan job somewhere outside the intended network. A stale resolution file is allowed to fail rather than quietly becoming permission to probe a new destination.

Opt-out became its own engineering problem

The exclusion system started as simple hostname and CIDR files, but it grew into one of the more interesting technical parts of the project. Exclusions need to apply before probing, during publication, and when old raw scans are regenerated into new public output.

Manual host and prefix rules fail closed: malformed or missing required policy stops network work rather than silently continuing without exclusions. Public output also removes excluded references and strips private destination provenance before publication.

The self-service path adds Auth42 identity and registry ownership checks so a maintainer can request exclusion for resources they actually control. Approved changes update runtime policy and can regenerate the current public artifacts from the recorded raw scan without requiring a fresh crawl.

That is more machinery than a tiny hobby-network directory strictly needs, but it created a useful design constraint: discovery policy has to remain enforceable even after collection has already happened.

A few decisions that ended up mattering

Technology

Current status

The crawler, publication pipeline, exclusion handling, self-service operator flow, and static viewer are implemented and being prepared for public launch. The remaining work is mostly final validation and the practical question of what turns up once real people start browsing the atlas and asking it to include or exclude things I did not anticipate.