FORSMILE
JA
AI2026/11/21

Airbnb's Text De-identification for AI Training: Presidio as the Base, Realistic Fake Values, and Per-Language Testing (Its Chart Flags Japanese Names and Places as Weak)

Airbnb's blog (Nov 17, 2026) explains how it removes personal data from text before AI training: Presidio, realistic fake values, per-language tests. Its chart flags Japanese names and places as weak.

← Back to Blog

On November 17, 2026, Airbnb's engineering blog published a post on how it removes personal data from text before using it to train AI models. The subtitle says it covers how Airbnb detects and replaces personal data in messages, reviews and support chats before the data reaches a model. The notable choice is that it does not just delete: it swaps names and addresses for realistic fake values, so the text stays useful for training. The base is Presidio, an open-source library that Microsoft originally developed and that recently moved to community governance. Airbnb says it added its own recognizers and replacement rules, a shared component that teams can reuse, and a benchmark to measure quality.

First, a note on the numbers. The post gives no numbers for detection accuracy, volume or cost. Accuracy appears only in a grid figure by language and entity type, shown by the size and color of each circle. So below I focus on the architecture and design decisions the post confirms, and I separate what the figure shows from what it does not (Airbnb's statements).

What it protects, and what it replaces

  • ▸Scope: transaction data used to train machine learning models. Airbnb says it now uses transaction information to train models, and that it removes personally identifiable information (PII) first to minimize the number of copies and to prevent leakage into and through its models. The subtitle names messages, reviews and support chats
  • ▸What counts as PII: Airbnb says it keeps a shared taxonomy, from commonly used fields such as names and card numbers to less common ones such as precise location coordinates. This post is about text de-identification
  • ▸Policy first, then code: privacy decisions start as policy, set and maintained by Airbnb's Privacy Architecture team with Legal, Policy and Engineering. Engineering then implements the decisions as code
  • ▸Why replace instead of delete: Airbnb removes PII and swaps in simulated data that is as realistic as possible. Its reason is that a placeholder tag signals that something was removed, and that signal can itself become a pattern a model learns, which leads to mis-training

The architecture the post confirms (the base and how it is distributed)

  • ▸The base is Presidio. The post lists six requirements: (1) on-premises hosting, so data stays on Airbnb's infrastructure; (2) general-purpose and modular; (3) multilingual support; (4) detection and obfuscation kept separate but inside one framework; (5) support for redaction, masking and realistic replacement; (6) consistent fake values within a run, so the same name always maps to the same fake
  • ▸What it passed over: managed cloud services, because they are not hosted on-premises, and standalone named entity recognition (NER) models, which find names and places in text, because they solve only part of the problem
  • ▸How it is distributed: Presidio ships as a Python package, and each pipeline imports it in-process rather than calling a service Airbnb operates. Raw personal data is de-identified inside the calling process, so no separate service holds it. The post says this also removes a central bottleneck, and that the decision is reversible, since the same library can be wrapped into a service later
  • ▸The tradeoff, and the fix: a library only protects data if every pipeline calls it, on the current version, with a compliant configuration. Airbnb's models train through a shared ML workflow orchestrator, so the library ships there as a reusable pipeline step. A team supplies only its data schema, marking which columns carry personal data. Airbnb owns the recognizers, operators, language handling and runtime environment
  • ▸A tag that gates deployment: the resulting dataset is tagged as de-identified, the tag follows the data into the model registry, and it is designed to gate deployment to production. Each run records the library version and configuration, so registered models can be traced back to the datasets that produced them. The post's figure shows opt-out filtering and de-identification before training, and a final check that lets a model through only if the registry confirms the tag
Airbnb's de-identification: the flow of detect, replace and tag, the kinds of recognizers, and how quality is measured

Detection (the Analyzer)

  • ▸Two stages: Presidio runs text through two stages. The Analyzer finds spans that look like personal data and labels each with an entity type (such as a name or a phone number) and a confidence score. The Anonymizer decides what replaces each span. Everything Airbnb built sits inside one of the two
  • ▸Three kinds of recognizers: the post says it uses three. One is pattern-based (decided by shape and nearby words) and one is NER-based (decided by a model). The figure also shows a third, a remote kind that calls external services (RemoteRecognizer), but the text does not explain it. Recognizers run independently, so two can claim overlapping text, for example a listing ID inside an email address. Overlaps pass through the Analyzer untouched and are resolved in the Anonymizer
  • ▸Language detection comes first: recognizers and models are language-specific. Names and addresses are short and carry few language cues, so Airbnb detects language on the whole record, or takes it from the record's locale metadata when it exists. If neither gives a confident answer, it uses a more conservative configuration rather than risk a model for the wrong language
  • ▸Pattern-based tricks: Airbnb-specific types (listing IDs, gift cards) are unknown to general models, so they are pattern recognizers. A run of digits cannot be settled by shape alone: a gift card number, a credit card number that passes the Luhn check, and a listing ID all look alike. Words near the number decide it, such as “listing” a few tokens before. But stock context matching is single-token, so “gift card” never fires and “card” alone fires next to a credit card too. Airbnb added a phrase pass, an ordered multi-word match, scored above a single-word hit
  • ▸Swapping the NER backend: names and street addresses have no structural pattern, so a model is needed. Presidio's default engine, spaCy, is fast, mature, cheap at scale and needs no training. But it tags any location, including cities and landmarks, while the detail that identifies a person is a street address with a house number. On Airbnb's benchmark those broad tags never resolved to the full address span. So it rebuilt the PERSON and LOCATION recognizers on GLiNER, a zero-shot NER model that takes an entity description at inference time. That lets it ask for a street address rather than any place name, and precision improved for both types. GLiNER2 is wired into the harness as a candidate and will be adopted only when the benchmark shows a gain
  • ▸Detection is configurable: score thresholds and entity scope are set per recognizer and per consumer. A pipeline feeding raw text into training uses a low threshold and accepts false positives. On already-reviewed data, a consumer can trade recall for fewer unnecessary redactions

Replacement (the Anonymizer)

  • ▸The defaults fall short: Presidio's default operators leave a document full of placeholder tokens or blank gaps. The post says both tell a downstream model far less than realistic-looking text does, and most of Airbnb's Anonymizer work went into closing that gap
  • ▸Operators: the figure lists five: replace (with a tag), redact, faker (a synthetic value), mask and hash. For most types Airbnb uses the faker library to generate realistic fake values, so a real name becomes another plausible name. Which operator applies to which type is configurable per model and use case: training text wants natural-reading fakes, while a human reviewing flagged output might prefer a mask so the type of value is still recognizable
  • ▸Custom fakers: (1) some matches should not be faked. A listing's own URL is structural data that consumers need intact, and a generic URL faker cannot tell it from any other site. In the figure, a URL on Airbnb's own domain (airbnb.com) is left alone and an external domain is replaced. (2) The library's assumptions do not always hold. Some locales' card-network tables are too narrow to fake a card number in the text's own language, so card faking is pinned to a locale with full coverage. Keeping a real country and area code while faking the rest of a phone number needs a library that parses phone structure. And where an identifier format has a reserved or never-issued range, the fakes are drawn from it, so they cannot collide with real values
  • ▸Consistent, and unlinkable: two properties pull against each other. Across runs, the same input must produce different fakes, so the fakes do not become a fingerprint tying two outputs together. Within a run, the same real value must map to the same fake, or the data stops being coherent: the same person mentioned twice in one record needs the same fake name both times. A globally shared random state cannot do both, so each fake is seeded from the real value plus a secret unique to the run, and Airbnb does not store a real-to-fake mapping
  • ▸Structured columns use the same engine: identifiers are hashed with a caller-supplied salt (a secret mixed into each value before hashing). Coordinates are truncated to a precision set by Privacy. 64-bit internal IDs, the numeric keys for listings and reservations, go through format-preserving encryption, so the output is another valid ID of the same width and range and a de-identified table passes the same schema checks as the original

How quality is measured

  • ▸Three axes: recall (how much real personal data the system finds), precision (how much of what it flags really is personal data) and throughput (detection speed at pipeline scale). The post says optimizing any one produces a system that fails in a predictable way. A miss is a leak, so when recall and precision must be traded, Airbnb weighs recall
  • ▸Broken out by language and entity type: an aggregate score is where the failure that matters most hides, so every metric is reported per entity type and per language, with the number of labeled instances behind it, plus a table of which entity type claimed spans that belonged to another. One identifier swallowing another's spans needs a different fix from a recognizer that simply misses
  • ▸Score the product, not an intermediate: the benchmark scores the overlap-resolved spans the library actually emits, not the Analyzer's raw candidates. A phone-shaped ZIP code inside a correctly detected address never reaches the output, so it does not count against phone-number precision
  • ▸Same yardstick for swaps and releases: the harness takes the NER backend as a parameter, so spaCy, GLiNER and GLiNER2 are scored side by side on the same golden sets, which is why GLiNER owns PERSON and LOCATION today. A new language ships only when a labeled dataset in that language exists to score it against. Gaps are folded into the benchmark so it keeps getting harder, and before any release the deployed version and the candidate run side by side. A version ships only when no language or entity type that worked before has regressed

What the figure shows, and what it does not (our reading)

  • ▸What is in it: the evaluation section has a grid of six languages by six entity types. The languages are English, Spanish, French, German, Japanese and Polish. The entity types are PERSON, LOCATION, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD and LISTING_ID. It is titled “F1 · entity × language”. F1 is a common metric that combines missed detections and false alarms into one number
  • ▸Only relative sizes can be read: email address, credit card and listing ID circles look full in all six languages. PERSON and LOCATION are green and nearly full for English, Spanish, French and German, but red, with a large gap in the circle, for Japanese and Polish. For phone numbers, only Japanese has a small gap
  • ▸What cannot be read: the circles have no numeric scale, so exact rates are unknown. The bottom left carries a label reading “Overall 9 / 10 looks healthy”, but the text never explains what the 9 out of 10 counts. I also could not tell from the text whether the grid is Airbnb's measured result or an illustration. So I cannot say that Japanese name and place detection is actually weak, only that the figure shows it that way
  • ▸How it fits the design (our reading): the pattern in the figure, strong for types decided by shape and weaker for names and places that depend on context and models, is consistent with the design in the text: patterns where the shape is fixed, models where it is not, and new languages only once labeled data exists. The text, however, says nothing specific about Japanese

What we could not confirm

  • ▸Concrete detection accuracy or false-positive rates. The figure has no scale, and the text does not say whether it is measured or illustrative, or what “9 / 10” means
  • ▸The amount of data, which models are trained, or which teams train what. The post stays at “we now train on transaction information”
  • ▸Throughput and cost numbers. The post says throughput is measured, but gives no values
  • ▸How opt-out works. The figure says “opt-out filtering”, and the text does not describe it
  • ▸Plans to improve weaker languages, including Japanese. The text only says new languages can be added, that new LLM-based recognizers could be included, and that work on media other than text has started
  • ▸Whether the third recognizer kind (remote) is actually used. It appears only by name in the figure
  • ▸How I read it. Medium's page refused automated access, so I used the full text of the same post from its RSS feed (dated 18:01 GMT on November 17, 2026) and the figure image. I have not checked whether the post was updated after publication

What to take from this case (our view)

  • ▸Deleting and replacing with realistic values are different designs. For training text, Airbnb's reason is that even a placeholder can affect the model. Once you replace, you need values that stay consistent within a run and cannot be linked across runs. The post's answer is a scheme with no mapping table: seed each fake from the real value plus a per-run secret
  • ▸Distribution matters as much as detector accuracy. A shared pipeline step, a de-identified tag, a deployment gate that needs the tag, and a record of version and configuration per run are written up as one design. The premise is that installing a library does not mean every pipeline uses it
  • ▸Do not rely on an aggregate score. Measuring per language and per type, and shipping only when nothing regressed, keeps weak languages from hiding behind a good average. If you handle Japanese text, measure on your own text (our view, not an Airbnb figure)
  • ▸If you use a service that may train on your posts or messages, check its terms and settings for whether such removal happens and whether you can opt out. This post is Airbnb describing its own system, and it does not show that other services do the same

Sources

Related articles