Why visa data is harder than you think

If you have never tried to build a visa database, the obvious approach is a spreadsheet. Two hundred passports across the top, one hundred and ninety-nine destinations down the side, each cell holds a string like "30 days visa-free" or "e-visa required". Forty thousand cells. A scraper to update them weekly. A static JSON dump to a CDN. Done by Friday.

We tried that. It does not work. This is a field report from eighteen months of maintaining what is, as far as we can tell, the most comprehensive and most current visa database in the world — and the patient, unglamorous work that goes into keeping it that way.

The spreadsheet fantasy

The spreadsheet fantasy assumes that every passport-destination pair maps to exactly one value. It does not. Almost every cell is conditional on something: how long you are staying, what you are doing, whether you have residency somewhere else, whether your passport has six months of validity, whether you have an onward ticket, whether you previously overstayed in a different country, whether you are vaccinated against yellow fever, whether you are arriving by air or land or sea, whether your previous visit was more than six months ago.

A cell is not a string. It is a small program. "30 days visa-free" might mean "30 days for tourism, 14 for business, not eligible if you entered overland from Cambodia in the last 180 days, requires proof of accommodation, fee waived if you are over 60 or under 12." If we collapse all of that into one string we lose the ability to answer real questions. If we expand it into a structured object, we have a small JSON document per cell, with seven to fifteen fields, all of which can change independently.

What is actually a cell

Our internal schema for a single passport-destination requirement has thirty-two fields. They group into six categories:

• Eligibility: passport country, residency overrides, age and vaccination conditions, dual-nationality handling.
• Stay: max days, multiple entry, extensions allowed, recovery windows after exit.
• Purpose: tourism, business, transit, study, work, family visit — each one a separate requirement profile.
• Documents: passport validity, blank pages, onward ticket, proof of funds, accommodation, invitation letters.
• Process: visa-free, visa on arrival, e-visa, embassy visa, transit visa — plus the application URL and an expected processing time.
• Fees: tourism vs business pricing, fast-track surcharges, currency, last-confirmed value with a stale-after timestamp.

When we report "45 days visa-free" in the widget, the answer is computed at request time from these fields against the traveler's actual situation. The string is the leaf, not the cell. This is the design decision that most distinguishes Orizn from the JSON-blob services that came before us.

The IATA problem

IATA's Timatic is the database airline check-in agents use. It is widely considered the source of truth for whether you will be boarded onto a flight. It is also a flawed and incomplete dataset for what travelers actually need to know.

Timatic answers a narrow question: will this passenger be denied boarding by the airline? It does not encode the entirety of consular policy. It does not always know about residency-based exemptions. It is updated on roughly a monthly cadence, which means a country that announces a policy on the first of the month might not be reflected in Timatic until the fifth of the next month. We have seen real cases of policies changing on a Tuesday, taking effect on Saturday, and the corresponding Timatic row not updating for six more weeks.

Building on Timatic is therefore necessary but not sufficient. We ingest it. We cross-reference it. We never trust it alone.

How we scrape (and why it fails)

We watch about four hundred and twenty sources. Consular websites, ministries of foreign affairs, e-visa portals, official press release feeds, parliamentary gazettes for the few countries that publish visa rules through formal legal channels. Roughly half are HTML pages. A third are PDFs. The rest are a mix of structured APIs, RSS feeds, and — embarrassingly often — official social media accounts.

Each source has a fingerprint: a hash of its structural skeleton (DOM positions for HTML, layout boxes for PDFs), independent of the words inside. When the fingerprint changes, we know the page was edited even if we have not yet figured out what changed. When the words change inside an unchanged fingerprint, we treat that as a high-confidence content edit and queue it for review. When both change, we treat it as a redesign and the source goes back to the human team to be re-modeled from scratch.

Scraping fails in three predictable ways. Bot detection — increasingly aggressive on government sites, mostly fixable with residential proxies and human-mimicking request cadence. Layout changes — fixable in software but never automatically; every redesign needs a human pass. Quiet content changes — the worst category, because the page looks the same, the fingerprint barely moves, but a single number changed. We catch most of those with diff alerts on a curated set of high-value sections. We miss some. We are honest about that and we publish our miss rate every quarter.

The human in the loop

Two researchers, between them eleven languages, sit in a Slack room with a queue of flagged changes. The internal tool shows them the old text, the new text, the source URL, a machine-generated summary of what it thinks changed, and a one-click "confirm and propagate" button. A confirmed change updates the database in seconds and ships to every downstream surface — API, widget, extension, mobile app — within ninety seconds.

A median item in the queue takes forty-two seconds. A hard item (PDF, unusual language, ambiguous wording) takes ten or fifteen minutes. We average two hundred and twelve items per day. Roughly five percent of items get escalated to an outside immigration lawyer for legal interpretation. We pay them by the question.

// Roughly what a queue item looks like internally.
type FlaggedChange = {
  source: string;          // canonical URL
  fingerprint: { old: string; new: string };
  passport: ISO3 | "*";    // affected passport, or wildcard
  destination: ISO3;       // affected destination
  field: keyof Requirement;
  oldValue: unknown;
  newValue: unknown;
  llmSummary: string;      // human-readable diff narration
  confidence: number;      // 0..1, from the layout differ
  flaggedAt: ISO8601;
};

What we publish

Every requirement in our database carries a sourceUrl and a verifiedAt timestamp. Both are surfaced on the API response, the widget, and the extension. If a piece of information is more than thirty days old without a re-verification, it is marked stale in the UI. If it is more than ninety days old, the answer is downgraded to "unconfirmed — check the official source" and the corresponding alert is fired internally to get the source re-scraped.

This is the freshness contract. We are not in the business of selling certainty. We are in the business of being honest about what we know, when we last confirmed it, and where we got it from. That is the part of the product the data team is most proud of, and the part our biggest customers (airlines, OTAs, governments) audit before they buy.

Five lessons

1. A cell is a small program. Treat it like one. Storing strings will trap you eighteen months in.
2. Trust no single source. Cross-reference IATA against consular pages, e-visa portals, and (sometimes) news reports.
3. Fingerprint structurally, not lexically. A redesign with the same words still needs human review.
4. Surface freshness everywhere. If a number is forty days old, say so. Customers reward honesty over false confidence.
5. Build the review tool first. A scraper that can flag ten thousand things a day is useless without a path to confirm them.

Visa data is harder than it looks, and that is the moat. Anyone can write a scraper. Almost nobody is willing to do the unglamorous, unending work of keeping forty thousand cells fresh, sourced, and honest. That is the work. We will be doing it for the next twenty years.