Federated learning arrived with a compelling promise: train machine learning models across distributed devices without ever centralizing raw data. Google popularized the architecture in 2017 with keyboard prediction on Android. Since then, the research community has treated FL as a near-default solution for privacy-preserving ML. The term gets dropped in regulatory filings, product white papers and privacy impact assessments as if it were a certification rather than a design pattern.
It is not a certification. Federated learning is a training coordination mechanism, and like every coordination mechanism, it inherits the trust assumptions of the parties doing the coordinating. The focus keyword here is federated learning privacy, and the central claim of this article is direct: federated learning does not automatically preserve privacy. Understanding why requires examining the honest-but-curious server model, gradient inversion attacks and the specific protocol choices that can close the gap between marketing claim and cryptographic reality.
Why Federated Learning Is Not Automatically Private
The standard FL architecture involves a central aggregation server that coordinates training rounds. Participating clients download a global model, compute local gradient updates on their private data and send those updates back to the server. The server aggregates the updates, typically by averaging, and pushes a new global model. Raw training data never leaves the client device. This is the privacy narrative.
The problem is that gradients are not raw data, but they are derived directly from raw data. A gradient update computed on a single image encodes information about that image. A gradient update computed on a small local dataset encodes information about every record in that dataset. The channel between client and server ships compressed representations of private data every single training round.
This would matter less if the server were fully trusted. In a standard distributed training setup, the server operator is the model owner and full data trust is assumed by design. Federated learning is architecturally motivated by the opposite situation: clients do not want to trust the server with their data. The moment you accept that framing, the server becomes a potential adversary, and every gradient update is a potential leakage vector.
The W3C's Data Privacy Vocabularies and Controls Community Group (DPVCG) has begun formalizing the concept of purpose limitation and data minimization in ML pipelines, but none of those vocabulary terms close the mathematical gap between "gradients instead of data" and "actual privacy." Federated learning privacy cannot be asserted by architecture alone. It must be proven by threat model analysis.
The Honest-But-Curious Server Threat Model
Cryptographic protocol design uses a taxonomy of adversary types. The strongest adversary is malicious: they deviate from the protocol, lie about inputs and actively corrupt outputs. The weakest relevant adversary is honest-but-curious, sometimes called semi-honest: they follow the protocol correctly but attempt to learn as much as possible from the data they legitimately observe.
Federated learning research frequently defaults to the honest-but-curious server assumption. This means the aggregation server runs the protocol honestly, it does not inject poisoned model weights, it does not drop clients strategically and it does not deviate from the averaging step. But it does inspect every gradient update it receives. It stores them. It runs inference against them. It correlates them across rounds and clients.
This is not a paranoid scenario. It is the contractually accurate description of any commercial FL deployment where the model owner operates the aggregation server and the clients are users who simply want a useful model without giving up their data. The server operator has legitimate access to the gradient channel. The honest-but-curious model is not an edge case. It is the baseline threat in nearly every real deployment.
What makes this threat model particularly sharp is that the server does not need to be adversarial by intent. A data breach, a subpoena, a rogue employee or a misconfigured logging pipeline can expose aggregated gradient histories to parties who were never supposed to see them. Federated learning privacy evaluated only at the moment of aggregation ignores the full data lifecycle. The NIST Privacy Framework (version 1.0, published as NIST IR 8062) identifies this exact gap under the GOVERN function: privacy risk must be assessed across the entire data lifecycle, not just the point of collection.
Gradient Leakage: What Attackers Can Reconstruct
The theoretical concern about gradient leakage became concrete with the Deep Leakage from Gradients (DLG) attack, published by Zhu et al. (arXiv:1906.08935). The attack demonstrated that an adversary who observes gradient updates can reconstruct the original training images and labels with high fidelity by solving an optimization problem: find input data that would produce the observed gradients. For small batch sizes the reconstruction is near-perfect. For batch sizes used in mobile FL deployments the attack degrades but does not disappear.
Subsequent work sharpened the attack. R-GAP (arXiv:2004.00053) showed that for fully connected layers the reconstruction problem has a closed-form solution, requiring no optimization at all. iDLG (arXiv:2001.02610) extended the original attack to recover ground-truth labels analytically. Each paper in this line of work narrows the gap between "we only sent gradients" and "they can read our data."
The attacks are not purely academic. They require the adversary to observe individual client gradients before aggregation. This is exactly what the aggregation server sees. A server running gradient inversion against individual client updates, before averaging them into the global model, has access to the precise threat surface these attacks exploit.
The practical severity depends on several factors: model architecture (convolutional vs. fully connected), batch size, data modality (images are more recoverable than tabular records), and whether any obfuscation has been applied. But the core lesson is durable: gradient updates are not a privacy-safe representation of training data. They are a lossy but often reversible encoding of it.
Secure Aggregation Protocols and Their Limits
The cryptographic response to gradient leakage from the server is secure aggregation. The goal is to allow the server to compute the aggregate of client updates without ever seeing any individual update in plaintext. Google's Practical Secure Aggregation protocol (Bonawitz et al., CCS 2017) accomplishes this using pairwise masking: each pair of clients agrees on a random mask. Each client adds its masks to outgoing updates. The server sums everything and the masks cancel out, leaving only the aggregate.
Secure aggregation is a meaningful advance. It directly addresses the honest-but-curious server threat for individual gradient inspection. But it introduces its own trust and performance assumptions that practitioners need to understand.
First, secure aggregation protects against single-round gradient inspection but does not protect against membership inference from the aggregate itself. An aggregate over a small cohort still leaks information about participating clients. Research on membership inference attacks against FL models (Nasr et al., IEEE S&. P 2019) shows that even aggregated model updates can reveal whether a specific individual participated in a training round.
Second, the masking protocol requires clients to be online simultaneously and to perform pairwise key agreement. In large-scale mobile deployments with high dropout rates, the protocol has significant complexity overhead. The practical implementations approximate the full security model with engineering trade-offs.
Third, secure aggregation does not protect against a malicious server that injects adversarial model updates between rounds to probe client data. The honest-but-curious model is the scope of the protocol. A malicious server is a different threat requiring different defenses, including Byzantine-robust aggregation rules from the distributed ML literature.
Differential Privacy as a Formal Guarantee
The strongest formal privacy guarantee available in federated learning pipelines is differential privacy (DP). In the FL context, DP is applied at the client level: each client clips its gradient update to a bounded norm and adds calibrated Gaussian or Laplace noise before sending it to the server. The resulting mechanism satisfies (epsilon, delta)-DP with respect to any single training example in any client's dataset.
The NIST definition of differential privacy (NIST SP 800-226, draft guidance) frames DP as a mathematically provable bound on how much any single individual's data can influence a published output. Applied to federated learning, this means that even if the server inspects individual updates, the noise prevents reconstruction attacks from succeeding beyond the DP privacy budget.
The honest accounting of DP in FL requires acknowledging the epsilon-utility trade-off. Tight privacy budgets require large noise magnitudes, which degrade model accuracy. The literature on FL with DP shows that useful models at reasonable epsilon values typically require large cohorts. Small-cohort FL with tight DP guarantees produces models with limited utility. This is not a flaw in the mathematics. It is the correct signal. Privacy has a cost, and that cost must be surfaced in design documentation, not buried in footnotes.
DP also does not address all threat surfaces in the FL pipeline. It protects gradient updates but says nothing about the privacy of the global model itself, which can be attacked with model inversion (Fredrikson et al., CCS 2015) and membership inference techniques. A complete privacy engineering analysis must treat gradient privacy and model privacy as separate problems requiring separate mitigations.
When Federated Learning Actually Protects Data
Federated learning provides genuine privacy protection when four conditions are met together. Practitioners should treat this as a checklist, not a sliding scale.
- Secure aggregation is implemented at the protocol level so the server never observes individual client gradients in plaintext. This closes the direct gradient inversion threat against the aggregation server.
- Differential privacy is applied at the client with a documented and auditable epsilon budget. The privacy budget should be disclosed in any privacy impact assessment filed under GDPR Article 35 or CCPA compliance documentation.
- Cohort sizes are large enough that aggregate updates are not trivially invertible. Research guidance varies by modality, but single-digit cohort sizes undermine both DP guarantees and secure aggregation anonymity sets.
- The threat model is documented explicitly, naming the honest-but-curious assumption, the malicious server boundary and the post-training model privacy risks. Undocumented threat models are not threat models.
When these conditions are met, federated learning is a legitimate privacy-preserving architecture. When they are not, it is distributed model training with a privacy narrative that does not survive scrutiny.
The gap between the marketing claim and the engineering reality matters because organizations are making regulatory and contractual representations based on FL deployment. A GDPR data protection impact assessment that cites federated learning as a privacy-preserving measure without specifying whether secure aggregation and DP are implemented may not satisfy the accountability principle under GDPR Article 5(2). The ICO's guidance on AI and data protection (updated as of 2026) explicitly requires that privacy-preserving claims in ML systems be substantiated by technical evidence, not architectural description alone.
PDAOS and the Data Sovereignty Framing
The Personal Data Asset Origination System (PDAOS), developed through Own Your Data Inc., approaches federated learning from a data sovereignty perspective that the standard FL literature rarely engages. The central claim in PDAOS is that personal data is a provenance-bearing asset: its origin, derivation chain and consent context travel with it through any pipeline that touches it.
Federated learning, in the PDAOS framing, does not eliminate the provenance problem. It relocates it. When a client device computes gradient updates from personal health records, conversational text or location traces, those gradients are derived assets. They carry provenance from the underlying personal data even if that personal data never left the device. Any federated learning system that claims to respect data sovereignty must account for this derived-asset chain.
This is not merely philosophical. W3C's PROV Data Model (PROV-DM, published as a W3C Recommendation) provides a formal vocabulary for expressing provenance chains in data systems. A PDAOS-compliant FL implementation would encode the consent basis, the data asset identifiers and the derivation relationship between training data and gradient updates in a machine-readable provenance record attached to each training round. This enables audit, enables revocation and enables the kind of transparency that transforms federated learning from a marketing claim into a verifiable privacy commitment.
Volume 6 of The Invisible Series, The Invisible Data, develops this argument at length: the invisibility of data in modern pipelines is not accidental. It is architectural. Federated learning made the training data invisible to the server. That is a real and meaningful step. But invisibility is not the same as sovereignty. Data subjects who cannot inspect the provenance of derived artifacts computed from their data have not gained sovereignty. They have gained plausible deniability for the system operator.
The path from federated learning as a coordination mechanism to federated learning as a genuine privacy architecture runs through the same disciplines that define privacy engineering broadly: formal threat modeling, cryptographic protocol selection, differential privacy accounting and provenance-aware consent management. None of those disciplines can be skipped. Each one is load-bearing.
For practitioners building FL systems in 2026, the starting resources are the original Bonawitz et al. secure aggregation paper (CCS 2017), the McMahan et al. DP-FL work (arXiv:1710.06963), the NIST Privacy Framework and the W3C PROV-DM specification. The MyDataKey platform and the provenance architecture in PDAOS offer an implementation reference for attaching consent and derivation records to FL training rounds. The Invisible Series provides the conceptual framing for why these technical choices are inseparable from questions of data rights and individual sovereignty.