Methodology
How we check websites and determine their status.
This page explains exactly how nere.nu monitors websites, detects outages, and distinguishes between genuine downtime and local connectivity issues. If you're a technical user who wants to trust our data, this is for you.
Overview
We use a multi-layered approach combining server-side checks, browser-side checks, and crowd data. No single data source is perfect, so we combine them for the most accurate picture.
1. Server-side Checks
How we connect
Our monitoring servers send HTTP GET requests to the target domain using cURL with these settings:
- Protocol: HTTPS first, with HTTP fallback if HTTPS fails
- Timeout: 8 seconds total, 5 seconds to establish connection
- Follow redirects: Up to 5 redirects
- SSL verification: Enabled (we verify certificates)
- HTTP version: HTTP/2 preferred, HTTP/1.1 fallback
- TLS version: TLS 1.2 or higher
What user-agent we use
We rotate between realistic browser user-agents (Chrome and Safari on modern Windows/macOS) rather than a bot identifier. This is because many sites return different responses to identified bots, which wouldn't reflect what real users see.
How we determine "up" or "down"
| HTTP Status | Classification | Rationale |
|---|---|---|
| 200-399 | ✅ Up | Server is responding normally |
| 400-499 | ✅ Up | Server works, just refuses/blocks our request (403, 404, etc.) |
| 500-599 | ❌ Down | Server error — the site itself is broken |
| 503 with CDN challenge | ✅ Up | Cloudflare/DDoS-Guard challenge = site is up, just verifying visitors |
| SSL handshake error (cURL errno 35) | ✅ Up | Server is reachable, just has SSL issues |
| TCP connect + no HTTP response | ❌ Down | Server crashed mid-request |
| Connection timeout | ❌ Down | Nothing is answering at the address |
| DNS lookup failure | ❌ Down | Domain doesn't resolve |
HTTP fallback
Some domains (notably older ones like elasticbeanstalk.com) only respond on HTTP and redirect to HTTPS elsewhere. If our HTTPS check fails with a timeout, we try HTTP as a fallback before declaring the site down.
2. Browser-side Checks
When you visit a status page, we also run a check from your browser. This catches cases where:
- Your ISP has routing issues to that specific server
- Your DNS is returning stale records
- Your firewall/antivirus is blocking the site
- The site is geo-blocked in your region
We use a <img> tag pointing to the site's favicon as a non-blocking probe. If it loads, the site is reachable from your browser. This doesn't work for every site (some block hotlinking), so we also try a no-cors fetch request.
3. Crowd Reports
When multiple visitors check the same site at roughly the same time, we aggregate their (anonymous) reports:
- Widespread outage: most recent visitors can't reach the site → likely genuine downtime
- Isolated problem: only a few visitors can't reach the site → likely local issues for those users
- Mixed: some can, some can't → possibly regional outage, DNS propagation, or gradual recovery
4. Check Frequency
Different sites are checked at different intervals:
| Category | Interval |
|---|---|
| Site is DOWN | Every 2 minutes |
| Popular sites (top 1000) | Every 5-15 minutes |
| Regularly searched sites | Every 30 minutes |
| Rarely checked sites | Every few hours to daily |
| When someone visits a status page | Immediately, if cached data is older than 15 minutes |
5. Incident Detection
An "incident" is detected when we observe a status transition. We record the exact timestamp and duration. You can see incidents in the "Recent Incidents" section on each status page.
We use hourly buckets for the 7-day uptime timeline: if any check within an hour showed the site as down, that hour is marked down.
6. Response Time Measurement
Response time is measured from the start of our cURL request until the full response is received. This includes:
- DNS lookup
- TCP connection
- TLS handshake (for HTTPS)
- HTTP request + response transmission
- Any redirects followed
Because this is measured from our server's geographic location (EU), sites hosted far away will naturally appear slower. A site showing 500ms from our servers might be 100ms from a closer location.
7. Infrastructure Detection
We identify the CDN or hosting provider via HTTP response headers:
cf-ray:header → Cloudflarex-amz-cf-*orserver: AmazonS3→ AWS CloudFrontserver: gwsorserver: gfe→ Google Frontendx-served-by: cache-* fastly→ Fastlyx-akamai-*orserver: AkamaiGHost→ Akamaix-vercel-*→ Vercelvia: 1.1 vegur→ Heroku
If headers don't give a clear answer, we fall back to CNAME records (useful for detecting hosting providers even when they don't set custom headers).
8. What We Don't Do
- We don't try to log into sites or access private content
- We don't submit forms or interact with the site beyond a GET request
- We don't check private IPs, internal hostnames, or reserved TLDs
- We don't run checks more than once per 15 minutes from visitors (to avoid abuse)
- We don't report on content quality, SEO, or performance (just uptime)
9. Limitations
We're upfront about what our checks can't do:
- Geographic limitation: We monitor from European data centers. A site that works in Asia might appear down to us.
- Timing: A transient 1-second outage between checks won't be captured.
- Functional testing: A site that loads the homepage but has a broken checkout flow will appear "up" to us.
- IP blocking: Some sites block datacenter IPs, making them appear down to us but up to real users.
- DNS propagation: When a site changes DNS, our view might differ from yours for up to 48 hours.
For mission-critical monitoring, combine nere.nu with a dedicated commercial service that monitors from multiple regions.
10. Data Sources & Third Parties
We use:
- Tranco list for popular site rankings (academic, community-maintained)
- DNS over HTTPS queries to Cloudflare (1.1.1.1) for consistent DNS results
- Our own monitoring infrastructure for actual checks
We don't buy data from commercial monitoring services or resell their data.
Questions?
If you have questions about our methodology, email us at [email protected]. We're happy to explain specific edge cases or discuss improvements.