Checking the Checks

PROBE now makes a correct found-or-not-found decision 98.69% of the time it gives a clear answer, up from 96.38% in our September update. Recall for known accounts rose from 88.70% to 95.87%, and the number of missed accounts fell from 26 to 9. None of the 618 missing-account controls was reported as found.

Those numbers deserve an explanation, because part of the improvement came from fixing PROBE and part came from fixing the test.

When a miss is not really a miss

Our measurement campaign checks each automatic source against usernames we expect to exist and usernames we expect not to exist. In September, 26 known accounts were reported as not found. A tempting response would be to loosen the rules until those 26 turned green. We did the opposite: we opened each one in a real browser, next to a randomly generated username that should not exist, and compared what each page actually showed.

The 26 misses fell into clearly different groups:

• The account was gone. Five labelled accounts no longer existed. The site showed the same not-found page for them as for the random control, while still showing a separately verified live profile correctly. PROBE was right; our answer key was out of date.
• The site had moved. Five sources had changed where profiles live or how they respond. JSFiddle moved public profiles to a new path, Steemit's old availability endpoint now returns the same error for every username, and Bitbucket now exposes accounts through its public repositories API. Each fix was accepted only after a real account and two random usernames produced a repeatable difference.
• The page was behind a login. Several sites now send every visitor to the same sign-in or sign-up page, whether the account exists or not. No public response can separate those cases, so those sources became manual-only links.
• The site had closed. Ubisoft's forums now redirect to a closure notice, and EyeEm profile links now land on a different stock-photo service. Those sources were retired.

Eight misses remain unresolved. On those sources we could not confirm that any live account is still detectable, so we did not invent a fix or quietly relabel the test. They stay counted as misses until the evidence says otherwise.

A label is only as good as the page that produced it

Halfway through this round, the campaign reported two false matches. That is the result our release gate exists to stop, so the work paused until we understood it.

Both cases were usernames that an earlier automated review had labelled as missing. That review had compared each username against a random control and seen the same 404 for both, which looked like solid evidence. The problem was the endpoint: it had stopped working and returned 404 for every username. Once PROBE checked the correct page, both accounts were plainly real. The false matches were not PROBE claiming an account that did not exist; they were the answer key claiming that real accounts did not exist.

We corrected both labels with the new evidence and recorded why they changed. We also removed other automatically reviewed labels on still-unresolved sources from calibration, since they may share the same weakness. The rerun reported zero false matches.

Smaller details that changed outcomes

Some fixes were about how pages are read rather than what they say. PROBE reads only the start of each page to keep searches fast. On Cracked, the text that identifies a member profile appears well beyond that limit, so real profiles looked empty. The fix uses a profile marker that appears near the top of the page instead.

Other sites now answer automated requests with an AWS bot-protection challenge. Those responses are now reported as blocked instead of being interpreted as a profile page or a missing account. That costs some coverage on a few sources, but a challenge page says nothing about whether an account exists.

An honest denominator

The catalog changed during this work. PROBE now has 366 functional sources: 309 automatic checks and 57 manual-only links, compared with 373, 324, and 49 in September. Retired sources and login-walled sources no longer count toward the automatic score.

That matters when reading the numbers. Removing sources that could never give a clear answer makes coverage look better without making any remaining check smarter. So the headline figure we focus on is accuracy among clear decisions, along with the concrete changes behind it: 205 of 309 automatic sources are now fully verified, compared with 196 in September; 10 source rules were rewritten from paired evidence, 8 sources moved to manual-only, and 7 were retired; and five stale labels were replaced with live evidence. Coverage moved only slightly, from 75.92% to 76.36%, and we do not treat that as a headline gain.

We also added a test that compares every published source count on the site with the live catalog. If the catalog changes and a page does not, the build fails.

What it still does not mean

Zero observed false matches is a property of this controlled run, not a guarantee about every future search. Sites change, accounts are deleted, and protections are added without notice. Some correct results in September became blocked or timed out in this run because of rate limits and bot checks; those are counted honestly as unclear answers rather than retried until they look better.

A found result is still a lead, not proof that two accounts belong to the same person.

What comes next

With the source checks in a stable, measurable state, the next milestone is collecting public profile details such as display names, profile pictures, and short biographies. Those details will only be attached to results PROBE has already confirmed as found, so extra information never becomes a shortcut around the evidence standard described here.